Everyone likes and dislikes a part of GT7 in their own way, as mentioned, the word 'controversial' doesn't quite fit the situation.
When you look deep down into it, the driving physics are satisfying to use, the graphics actually contain more little details than you'd thing, the models are held to a high quality standard and the sound is massively improved over older games in the series.
(a great example of 'little details' PD put in is when you fully lightweight mod a road car, the car has a tinny 'hollow' noise filter on the interior view, denoting that the sound deadening has been removed to make the car lighter.)
As a core driving sandbox it's actually their best one yet! The main difference is that they've nailed what other companies struggle to do at its core but PD have messed up the 'game' part of GT7, which is one of the things that bothers me. Another are design choices that don't make sense or they fail to put in features that made the older games great.
It's not so much wanting Kaz to step down, his personal history is one of constant renewal and fresh starts, which is why one of the biggest parts to let down the 'campaign' is for example how linear it is, and how every GT game feels quite different. You start with a choice of only 3 cars, then you collect the rest in the following two races, then you get another car and then have to win the other two. Older and more favoured GT's had a wider selection of cars available in the used car market that you can choose yourself. So it's rarer for people to play through the game exactly the same way. Almost all of the races in GT7 are the 'catch the rabbit' formula of starting half a lap down behind the leader and you have only a few laps to catch up, no qualifying, rare grid starts. There are a couple of races that are ok but they weren't apart of the main 'campaign' and are hidden away, plus there's a custom race mode but setting it all up yourself is tedious (plus no qualifying anyway).
My point is, Kaz stepping down wont solve things overnight, but either himself or someone needs to put some effort and direction to make a game like GT8 actually good, to focus on a campaign with a structure that makes sense, have harder events locked behind licences thus not making licences useless for example.
If they keep making dumb design choices in the future, then the game itself wouldn't exactly be controversial, but maybe keeping the ideologues in charge could be seen as another story, especially if the series starts to struggle.