GT5's Game-Breaking Online Flaw (OP Updated: 11 Feb)

[Infinital-NG]

Biggest thing that would help in this, would be to know the ping/lag times for racers in the lobby.

As said in this post, in a P2P environment one bad seed will hurt everyone. Out of 16 racers, the majority of those racers may only need 15-35 server hops to get to the room host, while a select few might need 65 server hops. There's the problem.

Without having some sort of latency meter for everyone in the room to see, it would be hard to figure out this issue.

 

[NCRthree]

Thanks for changing the OP title.

We had to ditch our Laguna race in AIWJT GT500/300 because of this and postpone another trying to figure out a solution. Lag issues and time inconsistencies have dogged our series all season. Everyone in our group is NAT2 with good ping data. We're trying Fixed Host, no mics, visible damage off, 12 drivers for our final race. Sucks to practice all week alone and w/other racers then have the race turn to poo.

This situation has GOT to be addressed w/ PD and/or Kaz. If the future of GT5 is to be online racing (as Kaz stated recently) then something has to be changed. I know I'm losing patience bigtime.

 

[t_w_a_h]

I have witnessed this "slowing down" experience on my side for many times, but only when I run lobbies with my Croatian friends, in rooms with 12+ drivers and with Premium cars. We were very intrigued by it backthan and later we even discovered how running the Shuffle races - where majority of cars are Standard - hardly show this issue.

I can attest to this, I predominantly shuffle race and most of the time with the same group of EU guys. Whilst shuffling I have rarely (if ever) had this issue, recently though - due to the shuffle vote bug - we have been running more and more races with just one or two options from the recommended car list. This naturally leads to races with just premium models and suddenly we are experiencing this problem enough so that re-starting rooms and trying new host combinations is becoming the norm rather than an exception.

For information, the first instance of this was at Tokyo with a nearly full room of premium cars. About 6 of the 14 in the lobby were suddenly running significantly slower laptimes than during the 15 minute quali session (4 - 8 seconds).

 

[MGR]

My thoughts on a couple of points raised in recent posts;

I'm of the opinion that network speeds or connectivity are not related to this bug.

Amar's explanation of server 'suffocation' doesn't gell with me at all. This bug occurs in rooms without a 'Fixed Host' (like Private Lounges) which use a P2P type system where every PS3 is communicating with every other PS3. The only thing the host does in these cases is provide the matchmaking links to get each PS3 talking to each other.

I think the finger should be pointed at GT5's game code and the PS3 hardware itself.

Difference between free-run and race mode
Network Traffic? - Nope. Still same number of simultaneous cars on circuit. Same amount of data.
Recording replays? - Yep. The PS3 is not recording replay data during free-run.

Number of cars
I think this is where 'network traffic' gets dragged into the argument. But of course, the more cars racing the more that's asked of the PS3 hardware too.

Premium cars
The issue is far more likely to occur with highly detailed cars. Premiums don't need more network traffic, but they do ask more from the hardware.

FFB Wheel
FFB wheel effects would surely use more processing resources than dual shock rumble?
-

The only thing I know for sure is that it's INEXCUSABLE, whatever the reason.

 

[jon-]

My thoughts on a couple of points raised in recent posts;

I'm of the opinion that network speeds or connectivity are not related to this bug.

Amar's explanation of server 'suffocation' doesn't gell with me at all. This bug occurs in rooms without a 'Fixed Host' (like Private Lounges) which use a P2P type system where every PS3 is communicating with every other PS3. The only thing the host does in these cases is provide the matchmaking links to get each PS3 talking to each other.

I think the finger should be pointed at GT5's game code and the PS3 hardware itself.

Difference between free-run and race mode
Network Traffic? - Nope. Still same number of simultaneous cars on circuit. Same amount of data.
Recording replays? - Yep. The PS3 is not recording replay data during free-run.

Number of cars
I think this is where 'network traffic' gets dragged into the argument. But of course, the more cars racing the more that's asked of the PS3 hardware too.

Premium cars
The issue is far more likely to occur with highly detailed cars. Premiums don't need more network traffic, but they do ask more from the hardware.

FFB Wheel
FFB wheel effects would surely use more processing resources than dual shock rumble?
-

The only thing I know for sure is that it's INEXCUSABLE, whatever the reason.

Also worth adding the issue is more obvious on high draw (detailed) race circuits, where again more is asked of the cells.

We're thinking along the same lines, though of course we're just guessing at this stage.

It's almost like as you get slight screen slow down due to FPS drops the internal timers slow down too. As others have pointed out, this is a very dangerous and stupid behaviour for any games programmer!

 

[dsgerbc]

^ Don't think framerate and the glitch are too related. Had many instances of slow framerate with no noticeable change in speeds.

 

[Speedy6543]

I just tweeted Kaz on this matter.
Hopefully he takes his time to look at this thread and get's to the bottom of this issue...

 

[jon-]

Perhaps not, but thinking about it, the times when I've had the random huge screen slowdown mid corner for a second or so it's caused the car to push off line, so perhaps a much smaller screen slow down over the course of the lap is enough to add up to these times we're seeing...

 

[amar212]

Well MGR, you have just raised my eyebrow with mention of the Replay and I will go that far and say how Replay *could* be the main reason for this glitch.

It fits perfectly into picture, because from what we may presume, Replay file have to be *recorded* somewhere, and that somewhere is probably on the console/player with the highest bandwidth.

Because of the P2P nature of the connection, poor Host probably hqve to take all the distribution over himself AND record a replay. And since there is +12 cars of Premium quality - notice difference in damage modelling as a most important difference factor between P&S cars for real-time data, everytbing else could go prebaked - it could easily be the main reason for the choke.

Also, I have to say how issue was probably less noticeable once when "Visual Damage OFF" was introduced, but I can't be sure on this one.

Replay saving could easily be the main reason why the host is bottlenecked by bandwidth. And it could also explain why I can play my replays perfectly, while others have a slowdowns and twitches with the same file that was recored after the race under the same conditions. Replay could very easy be the key to this issue.

 

[BWX]

Timing is inextricably linked to any physics calculations in any real-time simulation, any programmer who has attempted a real-time simulation of anything can tell you this.

Yes, exactly. I don't know why anyone would think otherwise. Please someone smarter than me tell me how this is wrong.

I guess the cause of the problem doesn't matter anyway. We need to prove the problem exists, let PD know, and they'll figure it out.

 

[dsgerbc]

Replay saving could easily be the main reason why the host is bottlenecked by bandwidth.

Doubt it. Every PS3 records it's own replay. It does it every time, in good races and in glitched races.

 

[MGR]

^ Don't think framerate and the glitch are too related. Had many instances of slow framerate with no noticeable change in speeds.

Yep, I've had that too.

But when the bug occurs the frame rate just seems a little 'off' for basically the whole race. Not a massive slow down.

The replay issue really bugs me. It's like the PS3 has missed frames from being recorded. As if the game (when you were playing it) wasn't able to process the physics for that frame so it gets dropped all together.

It may only be a few frames per second, but it's like the game just fills these frames when playing with 'dead frames'. The dead frames do not have any physics applied to them and are not recorded in the replay. That's why the car won't corner, brake or accelerate properly.

 

[nasanu]

Perhaps not, but thinking about it, the times when I've had the random huge screen slowdown mid corner for a second or so it's caused the car to push off line, so perhaps a much smaller screen slow down over the course of the lap is enough to add up to these times we're seeing...

The frame rate is not linked to the physics engine. The very fact that you get dropped frames (and screen tear) hints at this. When the PS3 can not cope with drawing the current frame it just drops its work load and starts on the next frame. The physics engine runs much faster than the graphics engine (PD has stated how many loops it does per second but I can't remember other than to say its well above the 60fps graphics target) and logically the physics need priority over the graphics. The physics engine runs alone but the graphics engine relies on inputs from the physics engine, logic says that there will always be data for the graphics engine and the physics engine will never have to wait for anything.

But as has been said nobody outside of PD really knows. There are too many theories atm, bandwidth, graphics, controllers, replays... Eventually you guys will come around to what I was saying much earlier. You need to believe nothing and test everything. Everyone coming in here so sure its one thing or another has not got anyone anywhere.

Doubt it. Every PS3 records it's own replay. It does it every time, in good races and in glitched races.

^This.



Personally I'd like to see a person record (off the screen) a live race that is effected and then also save the replay. Should not be too hard if this happens all the time as I am led to believe.

 

[jon-]

But as has been said nobody outside of PD really knows. There are too many theories atm, bandwidth, graphics, controllers... Eventually you guys will come around to what I was saying much earlier. You need to believe nothing and test everything. Everyone coming in here so sure its one thing or another has not got anyone anywhere.

Finally, something we agree on

 

[will-i-am]

Everyone coming in here so sure its one thing or another has not got anyone anywhere.

Bro, nobody said its definitely this or that for sure. Everyone is gathering info on what SEEMS to be related or just going on. Nothing you have said can prove nothing is going on and nothing anybody else said can PROVE it is. This is why this is being done. Of course there will be things said that wont apply but those will be weeded out hopefully. Why are you so inclined to shut everyone down? Please help out by proving it doesnt exist or does. With what you say you seem to know it shouldnt take you long unless you dont have an idea yourself and just blowin hot air!

Is it safe to say that everything youve said cant be added cause you have no proof?

No we listen to what you have to say as well because you may be right on some things without proof, just like what other have said may be true.

Again NO ONE has said anything was the absolute reason.

 

[Dave_Panic]

I'd like to say one more thing about timing/frame rate. I'm speculating here but it's likely that there's an internal fixed physics frame rate that's supposed to remain constant. The graphics frame rate then varies independently of this. This is quite a common way of preventing wacky results in simulations. I would guess that that's fixed at 1000th of a second due to the timing resolution of the game, although they could be interpolating to achieve that. Regardless, if something is upsetting this the simulation will be affected.

I only mention this because while there shouldn't be a link between the graphics and physics frame rates it stands to reason that a fully bogged down system will not be able to keep pace. But, as said many times, this is just speculation.

 

[cicua]

The frame rate is not linked to the physics engine.

Really? What is it you know that we do not?

The very fact that you get dropped frames (and screen tear) hints at this.

In what way exactly? What makes you think it does not also drop the physics calculations when it drops a gfx frame? This was the plausible theory of BWX which you have never commented on in other ways than bragging "impossible, because that is not how it is done!".

Again, bold statements from you without any knowledge to back it up.

Eventually you guys will come around to what I was saying much earlier.

I do not like that attitude.

You need to believe nothing and test everything.

Never disagreed on this actually.

Everyone coming in here so sure its one thing or another has not got anyone anywhere.

Yes it has. Before testing it is helpful to gather observations and make theories from those, so you have an idea what to go looking for. It has so far led MGR to revise the OP to become a better summary.

But in this whole process I have seen you adding exactly zero constructive input. I suggest you leave this thread or start behaving.

 

[rsergio007]

This article is interesting because it documents the way round-trip latencies seem to affect players, and indicates there's always one player acting as a "master" (the article doesn't explain it but from their comments I assume they created a non-fixed ownership room, I could be wrong):
http://www.eurogamer.net/articles/digitalfoundry-the-lag-effect-psn-xbox-live-analysis

Also, the very GT5 built-in manual sheds some light into the architecture saying, on one hand, that the fixed ownership type (hosted by someone with a good connection) is better for people with bad connections to join since each player only talks to the host and, on the other hand, that "there is less possibility of lag occurring in a mesh type model". This seems to indicate that in a mesh/p2p room each player as access to communicating with all participants but probably ends up exchanging messages with whomever grants a slower round-trip latency, which could be in many cases better than always talking to a fixed room host. Again, only PD might clarify.

Anyway, all this is consistent with two ideas conveyed on this thread: that people with much faster connections (on small rooms) tend to be faster on track; and that from a certain room size, and especially after 2.02 the person with that fast connection becomes overburden with traffic and gets his performance hit quite badly.

Regarding replays, and knowing none of us seems to work for PD and actually know how it works, the very time discrepancy on replays could be the exact representation of the timing/performance issue during the race. I'm reaching here but, replays probably get built with the same exact coordinates exchanged between players during the race and actually used by the game engine to draw the cars and events. And the information in it is not that light. You get positioning, brake and acceleration %, gear, tyre status, fuel level, all this displayed in a very smooth way. Of course could be clever code doing all the smoothing but certainly tons of coordinates are passed around. And yes, to a point where whom/whatever is being used as master clock gets way out of whack.

 

[nasanu]

I'd like to say one more thing about timing/frame rate. I'm speculating here but it's likely that there's an internal fixed physics frame rate that's supposed to remain constant. The graphics frame rate then varies independently of this. This is quite a common way of preventing wacky results in simulations. I would guess that that's fixed at 1000th of a second due to the timing resolution of the game, although they could be interpolating to achieve that. Regardless, if something is upsetting this the simulation will be affected.

I only mention this because while there shouldn't be a link between the graphics and physics frame rates it stands to reason that a fully bogged down system will not be able to keep pace. But, as said many times, this is just speculation.

But the game gets arguably more bogged down offline than online. Don't forget that offline 15 AI drivers need to be processed. Not only that but their cars also need to be processed physics wise. Online (as a SCEA physics programer I am in contact with trying to get information out of confirms) games normally will not run a full physics model for opponent cars. They just take data as it comes in and run a light model on it correcting it as new data flows in. Ever seen cars driving sideways or floating up into the air online? The full physics engine would not allow that so it is pretty safe to assume PD does not run a full sim on opponent cars online.


A lighter physics model for 15 cars plus no AI calculations for 15 drivers == Large cpu savings.

 

[Dave_Panic]

But the game gets arguably more bogged down offline than online. Don't forget that offline 15 AI drivers need to be processed. Not only that but their cars also need to be processed physics wise. Online (as a SCEA physics programer I am in contact with trying to get information out of confirms) games normally will not run a full physics model for opponent cars. They just take data as it comes in and run a light model on it correcting it as new data flows in. Ever seen cars driving sideways or floating up into the air online? The full physics engine would not allow that so it is pretty safe to assume PD does not run a full sim on opponent cars online.


A lighter physics model for 15 cars plus no AI calculations for 15 drivers == Large cpu savings.

I agree with this. In fact I do not think it is to do with a lack of processing power, it's most likely something entirely different. I only mentioned this because you said that physics/graphics frame rates are not linked when in fact they are, even if you try to separate them this way.

 

[nasanu]

Regarding replays, and knowing none of us seems to work for PD and actually know how it works, the very time discrepancy on replays could be the exact representation of the timing/performance issue during the race. I'm reaching here but, replays probably get built with the same exact coordinates exchanged between players during the race and actually used by the game engine to draw the cars and events. And the information in it is not that light. You get positioning, brake and acceleration %, gear, tyre status, fuel level, all this displayed in a very smooth way. Of course could be clever code doing all the smoothing but certainly tons of coordinates are passed around. And yes, to a point where whom/whatever is being used as master clock gets way out of whack.

I am sure you are right about the replays, they are huge in size. Not sure where you are going with it though?..

As for ping times, if ping times were a cause I think I would certainly be effected more than most, but I've never experienced it. I frequently play with US and Japanese players, right on the edge of the playable ping range.

Oh and amar said he gets it with local players.

 

[Speedy6543]

Ok guys, this is some breaking news IMO
I'm quoting myself to keep all of the info that matters in this particular issue together.

Here's a copy (PAL) of the replay of this race.

To be found here on the forum.

I watched the replay again and timed the laps with my watch's stopwatch.
My conclusion here was that the racetimer in GT5 is incorrect.
Here are some compared laptimes all timed on lap 2 of the race.

Myself, GTP_Speedy6543 (DFGT)

• Racetimer: 1'53
• My stopwatch: 1'39

GTP_gooners (DS3)

• Racetimer: 1'37
• My stopwatch: 1'25

GTP_Hydro (DFGT)

• Racetimer: 1'51
• My stopwatch: 1'37

I now devided all times in GT5 by the real times to see how long one second is in this GT5 online race.
GTP_Speedy: 113sec/99sec = 1.141414sec
GTP_Gooners: 97sec/85sec = 1.141176sec
GTP_hydro: 111sec/97sec = 1.144329sec

Conclusion: GT5's in race seconds last for ~1.14 real life seconds.

So the timer is equally different for everyone.
However, when you will watch the PAL replay you will notice that the first driver, GTP_Gooners (DS3), is able to maintain higher speeds through the turns than anyone else on track. I think all other drivers where using a wheel. EDIT: GTP_MadMax86 is on DS3 too

So there must be something going on with the grip here.

Here are the results I got from my replay of the race,

Lap 2 of the race:

Speedy
GT5 timer: 1'53
My stopwatch: 1'53

Gooners
GT5 timer: 1'37
My stopwatch: 1'37

Hydro (host)
GT5 timer: 1'51
My stopwatch: 1'51

Deviding the all GT5 times seperatly by the stopwatch times gave me 1.0...
This means, in my replay, each real time second took 1.0 seconds in GT5.

Did you feel the lack of grip in the second lap? Though the lap 2 time is terrible, I can definitely see that you're trying to drive like its on ice.

Thoughts on this fellas?
Is this the new way of GT5 handeling lag???

 

[cicua]

This article is interesting because it documents the way round-trip latencies seem to affect players, and indicates there's always one player acting as a "master" (the article doesn't explain it but from their comments I assume they created a non-fixed ownership room, I could be wrong):
http://www.eurogamer.net/articles/digitalfoundry-the-lag-effect-psn-xbox-live-analysis

Interesting article indeed, I like when someone actually get their hands dirty. But it sounds exactly like a private lobby as the manual describes it, too bad they do not state it.

 

[Speedy6543]

^
^
BTW Gooners (no issues) is from Canada and so is Hydro (who appeared to have some issues too).

 

[nasanu]

Ok guys, this is some breaking news IMO
I'm quoting myself to keep all of the info that matters in this particular issue together.




Thoughts on this fellas?
Is this the new way of GT5 handeling lag???

No idea what to make of that. Perhaps everything was recording a bit slower for you guys and the replay plays at normal speed making it all run too fast? Still that would only explain a trivial detail.

The big question is what makes your replay get recorded slowly? When you were in the race did it all seem to run at normal speed?

 

[Speedy6543]

When you were in the race did it all seem to run at normal speed?

Yes, everything seemed to be normal untill I got into the first turn. I had no grip compared to the grip I had in the qualifying just before the race.
And Gooners could drive away from all of us in a miraculous way...

 

[BWX]

Offline Ai probably run a 'light' physics model too, probably. Not that it matters because even if it gets fixed in the next patch we'll probably never get answers so no one will ever be proven right or wrong. I sure hope they give us a hint to the problem because it will be very interesting to find out.

 

[Speedy6543]

Did you feel the lack of grip in the second lap? Though the lap 2 time is terrible, I can definitely see that you're trying to drive like its on ice.

And what do you make of this, nasanu???
You did check that replay did you?
Don't tell me there isn't something fishy going on here....

 

[cicua]

Ok guys, this is some breaking news IMO
I'm quoting myself to keep all of the info that matters in this particular issue together.
...
Thoughts on this fellas?
Is this the new way of GT5 handeling lag???

Good research 👍

I think the recorded lap times must be correct; you actually did a 1.53, in wall-clock time, and lost 16 real-life seconds to Gooners. After all you were in the same race physically, and you got far apart.

Therefore the replay must be wrong. Maybe it is playing too fast (could have corrupted timestamp markers inside it). Or maybe it simply lacks a lot of frames (have you compared file sizes?).

So I see two different issues: 1) Loss of grip. 2) Replay gets borked.

They might be separate issues, but are maybe symptoms of each other or otherwise connected.

It is interesting that Gooners' replay had the same lap times, but plays correctly, but still shows you on ice.

It looks like if you can show a replay that plays too fast it might be proof that you were disadvantaged. Not that it will help make the race better after the fact

-- update

Here is what I think: When the PS3 skips frames they are not recorded, and at the same time, it skips all physics calculations. So what to do in next physics update? It knows the speed of your car, so it just interpolates that, and applies some extra travel distance to make up for the lost frame. But it totally forgets to apply tyre calculations. So for 1/60 sec you get zero brake, accel or lateral force, and the car just travels free in the direction and speed it already had.

This would explain the reported lack of grip.

And why would it forget those calcs? Plain bug. Probably introduced or made worse as a side effect of the new tyre stuff in 2.02.

Not sure if this was already said, but at least now I understand it in my own words

 

[CSLACR]

Ok guys, this is some breaking news IMO
I'm quoting myself to keep all of the info that matters in this particular issue together.




Thoughts on this fellas?
Is this the new way of GT5 handeling lag???

It could be server lag, or system lag.
What was damage set to? Visual damage on/off?

Here is what I think: When the PS3 skips frames they are not recorded, and at the same time, it skips all physics calculations. So what to do in next physics update? It knows the speed of your car, so it just interpolates that, and applies some extra travel distance to make up for the lost frame. But it totally forgets to apply tyre calculations. So for 1/60 sec you get zero brake, accel or lateral force, and the car just travels free in the direction and speed it already had.

This would explain the reported lack of grip.

And why would it forget those calcs? Plain bug. Probably introduced or made worse as a side effect of the new tyre stuff in 2.02.

Not sure if this was already said, but at least now I understand it in my own words

That's kind of what I was thinking, though I think damage might compound the issue.