It's a bit unreasonable to say the wait was 5 years.
Not really. That's how long it took them to make GT5 after they finished GT4.
How come nobody complains about the 4 year gap between GT3 & 4?
I guess you weren't around at the time, huh?
GT4 was scheduled to come out Christmas 2003. It was delayed, and so GT4P was put out as a consolation to stop the fans from screaming quite so loudly. A lot of people were still pretty pissed off.
So yes, people complained about delays for GT4, pretty extensively. Fortunately, PD used that time to make what is arguably the best game of the series, and so in hindsight the delay seems to be justified.
That same argument was used while we were waiting for GT5. At the time Polyphony had a record of taking their time, but delivering spectacular results in return. The assumption was that GT5 would be a product that surpassed even GT4, a game truly worthy of 5 years of development. It wasn't, and that's why GT5 gets nailed for long dev times and GT4 doesn't.
You don't see people complaining about GTA V taking too long, because even though it took five years as well and some people were getting pretty salty about it near the end, the actual release turned out to be more than worth it.
If Polyphony starts putting out some games that actually seem like they've used the development time between games wisely, then maybe they'll stop catching flak for it. Neither GT5 or GT6 look like products that 120+ people worked on for 5 and 3 years respectively.