MIT Robotics – Vladlen Koltun – A Quiet Revolution in Robotics Continued

In this talk

1. 1:51

   Why simulation belongs in robotics

   Koltun argues that simulation has become a practical tool for building and deploying robot policies, and that results in central robotics problems make blanket skepticism harder to justify. He frames this talk as a companion to a separate talk on legged locomotion, focusing here on agile flight.

2. 6:26

   Early steps in autonomous drone racing

   The team began with a vision-based drone that used only onboard sensing and computation, with no motion capture or prebuilt map. An early gate detector predicted waypoints, and later versions trained the vision system entirely in simulation using varied textures, lighting, and camera conditions. The drone could adapt when people moved the gates.

3. 15:24

   Designing policies that transfer

   For acrobatic flight, the team optimized trajectories in simulation using privileged state information, then trained a small policy to track them on the drone. Feature tracks and temporal models compressed visual input and helped bridge differences between simulated and real images. For cluttered environments, the team used simulated stereo depth with realistic artifacts, then trained a planner and controller in simulation.

4. 28:44

   Swift beats human champions

   Six years into the project, the Swift drone raced autonomously against FPV world champions using onboard sensing and a Jetson TX2. Simulation training alone did not transfer in this high-performance setting, so the team measured perception and dynamics errors on real flights, modeled those residuals, and fine-tuned the policy in simulation. Swift achieved the fastest run and won the most head-to-head races, while relying on a gate detector trained with real data.

5. 42:48

   Strengths, limits, and lessons

   Koltun discusses tradeoffs in the comparison, including the human pilots' video latency, Swift's inertial sensing, and the pilots' ability to adapt after damage. Swift could not handle unfamiliar tracks or gate designs well, and it ignored its opponent when choosing how aggressively to race. In separate experiments, reinforcement learning outperformed optimal control, which Koltun attributed to its ability to optimize objectives without requiring an intermediate path or trajectory.