Channels charging a monthly fee on YouTube will end in December, after the idea failed to take off.