I have some seat times on various 911-based race cars, generations of RSRs, GT3 Rs, Clubsports (mostly 10-100 hours, but 100+ hours on Clubsports). They all have one thing in common that is significantly harder than the base road car (3RS/2RS). Getting in and out of the car. Due to the roll cage and full bucket seat, it's significantly harder to do so compare to the stock road cars. If you're a big dude, it's impossible to getting in and out without removing the steering wheel, and even after the removal of steering wheel, you're probably going to need someone to help you to get in or out of the car. Tough to get in and out with your dignity in tact.
But you can also make the road car harder to get in or out. Slap a full roll cage and bucket seat on smaller cars (like S2K or Elise), then it would be even more difficult than many race cars. Even the base Elise/Exige (with the roof and steering wheel) is only slightly easier than the race cars. So after the roll cage...
Other than that, they aren't significantly harder to drive, as long as you're driving it in Winter/Autumn/Fall or driving one with air conditioning (recent models have one as an option, but if you drive one without AC in Summer, then it's very difficult to do it for a long session). When you drive LM GT3 or GTE cars for the first time, you need some time to adjust your braking points, because the correct braking points for those cars definitely feel very wrong for anyone who're mostly familiar with road cars on the same tracks. And that's about it.
Of course driving it to the limit is somewhat harder than the road car variants, but it's because they're faster cars, not because they're race cars. Driving the faster road car to the limit is also harder than driving the slow road car to the limit. That's the reason why many people start with slower FWD cars, and graduate to RWD cars and then gradually faster RWD cars. If you're already familiar with GT3 RS, enough to produce a good time on GT3 RS (within 0.5% of the best lap of the same setup/tire car, which was set by one of the good pro drivers not a bad one), it doesn't take too long to bring up the pace to the acceptable level.
But it's true that if you're only familiar with FWD cars or 300hp RWD cars enough to produce a good lap, then you have long way up ahead (at least hundreds of hours). If you can't even produce a good lap with slower cars, or don't have any experience on track, expect a thousand hours of seat time ahead at the very minimum (this figure assumes you're fit to do that, and if you don't expect an order of magnitude higher seat time). The amount of time required itself is not a big problem, but many people can't do that because they're very costly. For a cheap slow FWD car, it's roughly $300-500/hour, and it shoot up to $3-5k/hour for faster road cars, and you need at least a thousand hours from various speed category.