Nope. It's people with poor connections or from very far away joining your lobbies. Get 15 guys from near you with connections the same as yours and you'll have a 16 man lobby without a problem - remember, we had lagless gameplay (not replays though, oddly) in an 11 man lobby because we were physically in the same room and running through the same switch, with an utterly stable and near-zero ping system.
It's the nature of networks. It takes time to transfer the information from your device, through the network to the other devices - and with a 60fps game involving vehicles moving at a typical average speed of 120mph, anything over 17ms will be visible and manifest as a significant change of position. With a dedicated server, it might even be slower as you all have to send your information to it and receive everyone else's information from it and it is geographically fixed, meaning your online experience will always be crap if it's crap from the get-go - peer-to-peer is used to minimise lag, normalise the connections and to provide a relatively consistent online experience for everyone. If GT5 had fixed servers in Japan, you'd have a terrible game - you'd need to send your data along the California coast to San Francisco, across the pacific (there's a couple of cables on this route, but they're still several thousand miles) to Tokyo and then on to wherever the data centre is and then they'd have to send the data back to you - probably a round-trip of 0.1s. But then so would everyone else, even if they were in your street. Everyone gets 0.1s of lag, whereas on P2P the data would go no further than your local routing centre or exchange (probably 10 miles or less) and you'd experience no more than 0.02s of lag - possibly much, much less...
The net result, however, is that people with crap connections ruin it for everyone else. You get 14 guys from your street playing and you get no more than 0.02s of lag. Guy 15 joins and he's from Indonesia. Suddenly you're all having to send data to him as well - 15 lots of 0.1s round trips - and your lobby goes to hell.
If you only run private lobbies with people you know with decent connections from no more than 200 miles (crow flight) away, you'll be fine. Push it up to 500 miles and you'll still have a good online experience. Go further or have unknowns joining and you're banjoed. Matchmaking software for public lobbies should try and pair you with people with similar connections and geographical locations - I don't play public so I can't make any claims about the efficiency of GT5's matchmaking software, but in public lobbies in games like MW2 that are also P2P I always end up playing with other Brits, French, Irish, Dutch, Spanish and German players....