Robotics' End Game: Nvidia's Jim Fan

In this talk

1. 1:37

   Applying the language model playbook

   Fan maps language model progress from GPT-3 pre-training through instruction tuning, reinforcement learning, and automated research. He proposes a similar path for robotics: predict future physical states, tune models toward useful robot actions, then use reinforcement learning to improve performance.

2. 3:31

   From video prediction to robot action

   Fan argues that vision-language-action models devote too much capacity to language and are better at representing objects than physical behavior. He points to video models that appear to learn properties such as gravity and buoyancy, then introduces Dreamer, which predicts future scenes alongside motor commands. He calls this broader approach world action models, designed for open-ended instructions.

3. 7:44

   Scaling robot data with human activity

   Fan says teleoperation is constrained by robot availability and estimates that practical collection often amounts to about three hours per robot each day. He describes UMI and Dex-UMI interfaces that let people collect manipulation data directly, then presents Ego-Exo, pre-trained on 21,000 hours of first-person human video with no robot data. Its action tuning uses 50 hours of motion-capture data and four hours of teleoperation; Fan reports a scaling relationship between video hours and dexterity performance.

4. 13:59

   Expanding reinforcement learning environments

   Fan proposes combining a small number of real robot stations with scanned objects in physics simulators and learned video-based simulators. DreamDojo takes robot actions and generates visual frames and sensor states, without relying on hand-coded physics equations or graphics engines. Fan’s proposed system uses compute across these environments to increase the data available for robot learning.

5. 17:00

   Three milestones toward automated industry

   Fan names three goals: robots that match human performance across varied tasks, fleets controllable through software interfaces, and robots that can research and build improved successors. He imagines applications including lights-out factories and automated wet labs. He predicts the broader robotics end game could arrive by 2040.