Polyphony Digital has unveiled its new artificially intelligent driving system named “Gran Turismo Sophy”, developed in collaboration with Sony AI. It will be included in Gran Turismo 7 via a future update.
Sophy was developed using the latest in reinforcement learning techniques, which train an AI-agent by “rewarding” or “penalizing” it for specific actions it takes inside an environment. In Sophy’s case, Sony AI researchers used a training algorithm called “Quantile-Regression Soft Actor-Critic (QR-SAC)” to encode the rules of racing in a way that it could understand what it was supposed to do.
More specifically, Sophy was taught to master three specific skills in GT Sport: car control, racing tactics, and racing etiquette.
Car Control
First, Sophy had to learn how to control a car to drive quickly around a track on its own. It was trained on the possible outcomes of “high-speed actions”, learning how to take corners at their limit. It was penalized for negative actions — like hitting walls, cutting corners, or leaving track boundaries — and rewarded for clean, ever-faster laps.
Sophy was trained in GT Sport’s environment, running on over 1,000 virtualized PlayStation 4 consoles in Sony Interactive Entertainment’s specialized cloud gaming platform. This allowed Sony AI researchers to run hundreds of experiments simultaneously so Sophy could learn as quickly as possible.
“It takes about a day to have an agent go from knowing nothing to being able to drive around a specific track,” said Michael Spranger, COO of Sony AI. “Then, it takes about two days to get into the top five percent of human drivers, then another 10-12 days to really push it to the limit and equal the best drivers out there. Within that time frame, Sophy experiences around 300,000km of driving.”
“I am a racer and learned techniques on how to drive fast, like slow-in-fast-out,” Gran Turismo creator Kazunori Yamauchi said. “Gran Turismo Sophy does not learn that way. I think that after Sophy launches into the world, the textbooks about driving will have to be changed. For example, when Sophy goes into a curve it actually turns and brakes. Usually, when you go into a curve, the load is only on the two front tires, but Sophy has the load on three tires: two in the front and one in the rear as well. It allows the car to break as it is turning and is not something human beings would be able to do, conventionally. What ultimately happens is that it is driving fast-in, fast-out.”
Yamauchi-san also noted that Sophy had learned specialized driving techniques Polyphony Digital had only seen used by elite drivers — such Formula One World Champions Sebastian Vettel and Lewis Hamilton — in Gran Turismo Sport.
Racing Tactics
Racing tactics, which involve other cars, was a more complex process. Sophy was trained using “mixed-scenario” training that involved hand-crafted race situations that were likely to be seen on each track with specialized opponents. This exposed Sophy to typical race experiences and taught it how to deal with things like crowded race starts, how to use the slipstream to pass opponents, and how to defend against passing attempts.
Racing Etiquette
Perhaps the most difficult challenge — for both an AI system and human drivers — is racing etiquette, or what is and is not appropriate on-track behavior. Sony AI’s researchers found ways to “encode both the written and unwritten rules of racing” into Sophy’s reward functions. The team also had to carefully balance the opponents that Sophy faced to ensure it did not become too aggressive or too passive.
To demonstrate Sophy’s abilities against human players, Polyphony Digital hosted some of the best Gran Turismo World Tour drivers at two special “Race Together” events in its Tokyo offices. Human players defeated Sophy at the first event in July 2021, but the Sony AI team emerged victorious at the second event a few months later in October.
Polyphony Digital has published videos from each event — complete with commentary from the “voice of Gran Turismo” himself, Tom Brooks — to demonstrate Sophy’s driving techniques and how it has evolved.
Player Reactions
In addition to the head-to-head “Race Together” events, more Gran Turismo players, including Igor Fraga, Valerio Gallo, and Emily Jones, have had a chance to drive with Sophy and share their impressions.
Nature Magazine Article
Gran Turismo Sophy has been featured on the cover of the February 2022 issue of Nature, revealed today. The full research paper, titled “Outracing champion Gran Turismo drivers with deep reinforcement learning”, is available for Nature subscribers, though the abstract is available publicly on the academic journal’s website.
The abstract notes that pseudocode detailing the training process is available as a supplement to the article, and mentions that “the agent interface in GT is not enabled in commercial versions of the game, however, Polyphony Digital has provided a small number of universities and research facilities outside Sony access to the API and is considering working with other groups.”
Videos of Sophy’s learning process in GT Sport have also been posted on Github and Sophy’s new project website.
Sophy in Gran Turismo 7
Although Sophy was developed using GT Sport’s environment, Kazunori Yamauchi confirmed its technology will be coming to Gran Turismo 7 in a “future update” for the game. It is not yet clear exactly how Sophy will be integrated into the game, though we do know Polyphony sees Sophy as not just a virtual competitor, but a teaching tool.
“Ultimately, we want to be able to provide joy and fun as Sophy races with people,” Yamauchi-san explained. “The objective is not to just beat human beings, but to create a friend, a comrade, or a buddy. We want players to learn together and grow together with Sophy. […] We want to create something that has a positive effect on society, that promotes sympathy, kindness, and is creative and ethical.”
Stay tuned — GTPlanet will have more exclusive information about Sophy in the next few days…
See more articles on Gran Turismo Sophy, Igor Fraga, Kazunori Yamauchi, and Valerio Gallo.
