I've been testing the game for the past week, and I can notice a difference in handling. In some cars more than others. It's not very noticeable in cars that have a very small offset(stagger, whatever you call it), like the BMW 135i, Hyundai Genesis, Nissan Fairlady Z, Mustang Boss 302 and you can even get away with it in the BMW M3 E92 and the M5. They will loose the rear end easier but it's avoidable even on Sport Hard tires. They are not undrivable and the skinny tire is barely noticeable.
Cars like the Dodge Viper SRT10/GTS/ACR, Cizeta Moroder V16T, modern Corvettes and any other powerful high performance sports cars, not only look awful with the rear skinny tires, but they become almost undrivable unless you use Sport Soft tires (not really a solution). It's even worst in powerful MR cars since they'll loose the rear even under braking in straight line.
As for the wheel size, I don't know if it affects the handling. I usually keep the stock size in all cases. I might increase the size 1 inch, but it's rare.