Sergey Levine (UC Berkeley): Robot Foundation Models

In this talk

1. 13:52

   Why learn robot control end to end

   Levine contrasts modular systems, which separate perception, planning, and control, with models trained directly from observations to actions. In a shape-sorting task, end-to-end learning handled a roughly one-millimeter insertion tolerance better than a pose detector with 1.3-centimeter error, partly by sliding the object until it fit.

2. 23:34

   Building vision-language-action models

   Vision-language-action models combine image input and language understanding with robot actions. Levine describes π₀’s continuous action expert, which uses flow matching to generate 50-step chunks for high-frequency control. In laundry-folding demonstrations, the robot could recover when extra objects disrupted the task, though its longer sequences still included mistakes.

3. 29:27

   Reasoning, coaching, and generalization

   Levine describes adding intermediate semantic steps to robot training, with labels generated using tools including Gemini and SAM. In π₀.₅, verbal coaching can guide robots through unfamiliar homes and supply useful training examples; training on these steps also improved performance, even when the steps were not used during execution.

4. 36:43

   More data improves general-purpose skills

   In the RT-X project, 33 labs contributed data across 22 robot types, and the resulting generalist model performed 50% better on average than the individual labs’ models. π₀ used about 10,000 hours of pretraining data, followed by smaller amounts of curated demonstrations. In household tests, training across 60 to 100 locations brought performance in unseen homes close to performance in a home included during training.

5. 41:48

   Reinforcement learning boosts performance

   Levine presents reinforcement learning as a way to make pretrained robot policies more reliable and faster. In deployments for tasks including espresso preparation and chocolate-box assembly, online RL improved throughput by about 2× over a demonstration fine-tuning baseline on three of four tasks. He says open challenges include broader generalization, better reasoning, and faster inference.