It was in fact quite a bit slower, and the further you "progressed" in the game the worse it became at basic functionality (which likely has to do with why GT6 had such big changes to the garage system). GT4 had its own problems and GT5 was infinitely better laid out (though slightly worse in in-race menus, and much worse in a couple of cases), but GT4 never randomly decided it was going to take 15 seconds to allow you to make a selection on the main menu after it loaded, or stop responding to controls on the garage screen if you scroll too fast until it could catch up, or cache all of its user data in a file that started growing so large it eventually slowed the whole thing down.
And that's just the stuff it does now, after the GT5 servers were scuttled. Because PD tied so much of the UI into multi-player functions, if PSN servers were being flaky it could just hang permanently until you logged out and sometimes not even without restarting the game. There were so many of what I'm assuming are optimization problems when you were connected to PSN that at times the UI would function about as well as that of a contemporary Kinect game.