2024 Fall Robotics Colloquium: Ishan Durugkar (Research Scientist at Sony AI)

In this talk

1. 0:38

   The racing challenge

   Durugkar focuses on GT Sophy, a reinforcement learning project that aimed to beat elite Gran Turismo drivers and later become part of the game. The team had to solve close vehicle control, race strategy, and sportsmanship in a simulator whose physics and tracks are carefully modeled.

2. 8:01

   Training a fast driver

   Instead of screen images, the agents received game data such as speed, tire slip, track progress, and nearby cars, then chose continuous steering and acceleration or braking actions. The team trained with QR-SAC on data from PlayStations, and found that distributional critics and multi-step returns improved lap times. Sophy learned to beat top human lap times, especially through chicanes where it braked less.

3. 29:10

   Balancing aggression and etiquette

   Agents used nearby opponents’ relative positions, speeds, and accelerations to make race decisions. Training only against the built-in AI made Sophy too forceful, while self-play alone made it too cautious. The team mixed opponents and used separate replay buffers for events such as slipstream passing and chicanes to preserve a range of racing skills.

4. 36:39

   Choosing policies and winning races

   Because the team had hundreds of saved policies, it filtered them for safe and fast driving, evaluated specific skills, held policy races, and had people assess the finalists. In October 2021, GT Sophy beat four elite human drivers, improving the team score from 70–86 in the earlier event to 104–52.

5. 42:52

   From research to the game

   The team adapted Sophy for enjoyable play, first testing it in a limited beta and then releasing it as a permanent Gran Turismo feature. By the November 2023 release, one policy could drive 340 cars across nine tracks and nine tire compounds; later updates expanded support to more than 400 cars. Durugkar says future work includes fuel and tire wear strategy, driving style, and applying the methods to other Sony games.