Funny you should say this. Someone posted this video on here a few weeks ago. It's very, very informative about how game designers and other designers treat this concept of "fun" as this kind of generic thing, where actually there are definitely some specific things you can do to make your games more fun.
I forget who posted it. Props to you, whoever you were.
While I agree to some extent that "fun" is kind of nebulous, at some point you have to draw the line between a good game and a bad one. Pretty much every game every created has at least one person that enjoys it, so simply having that isn't a good measure.
The easiest way is the same ways we judge art. We can judge it technically on execution and construction, the fundamentals of how it's made. And we can judge it artistically, on how well it achieves the goals it aims for. Does this game evoke the emotions and experiences that it tries to?
I don't think games are as personal as you make out, just as other media like movies, TV and books aren't that personal. Certainly, Breaking Bad may resonate more strongly with some people than others. But great art speaks to fundamental things about humans. Great games (generally) speak to the fundamental desires of play.
Shooters let you play at being a soldier. Uncharted lets you play at being Indiana Jones. A great car game lets you feel like you felt when you were a kid playing with Matchbox toys, or whatever your equivalent is. It lets you feel that you're taking part in something, it lets you feel things that you otherwise wouldn't feel or think. It lets you use your imagination within that world to take part in it, whether that be by embracing the story that the developer has laid out for you or by creating your own story.
GT does these things, but to a limited extent. As far as it's a sandbox for playing with cars, it's quite limited. You have little control over the visual aspect of the cars. You have little control over the scenarios that they're placed in. The scenario that the developer gives you is entirely bland and uninspiring. And you have a bunch of design choices making it difficult for you to just PLAY, and I use the word play there as a child would play. Doing things because you have a crazy idea and it's enjoyable.
This is why I criticise GT5. The things it aims for, it doesn't reach. It tries to provide the experience of working your way up from a slow car to a fast one, but the dodgy economics and leveling makes the experience jarring and difficult. It tries to provide the experience of racing, but the dodgy AI make the experience less than pleasant. It tries to provide the experience of driving on a track, but the inconsistent and unstable graphics give lots of opportunity for immersion to be broken. The severe limitations on being able to create your own events and customising your own cars means that creating your own "story" is very difficult unless your story involves hot lapping.
GT5 doesn't get to the things that it reaches for. GT6 appears to be better, but I still think they're failing by boosting the "stats" of the game without addressing fundamentals.
If you liken it to building a sports car, horsepower sells. Raising the horsepower makes the car at least sound more desirable. But without the other components throughout the car to support that power (chassis, suspension, internal luxuries, whatever) it's not going to be that good to drive. Some people will ignore the shortfalls to get that power, but most people would probably be more pleased if they'd spent some time making a more rounded vehicle.
I notice that while modern American muscle cars still retain a lot of the style and characteristics that made people love them, they now have taken a lot of positive handling and useability aspects from other types of cars. They're more generally appealing.
GT5 is a 60's muscle car. Hugely stylish, and pretty good at it's one thing. But only enthusiasts would own one as a daily driver. GT1 was built in the game equivalent of that time, when all games were like that. That's not the case any more.
Making great gameplay is hard. If it wasn't, we'd have a lot more great games. It doesn't mean it can't be done, or that we should pretend that OK games are anything more than OK.