U of T Robotics Institute Seminar: Sergey Levine (UC Berkeley)

In this talk

1. 1:23

   Cross-embodiment training

   Levine contrasts task-specific robot models with generalist models trained across robots and tasks. In the RT-X project, 34 labs contributed data from 22 robot types; the resulting model averaged about 50% better performance than domain-specific models in the reported comparisons. He also describes how vision-language models can contribute language understanding to robot control.

2. 8:55

   Building and adapting π₀

   Levine presents π₀, a model trained on data from seven robot types, web sources, and Open X-Embodiment. It uses a PaliGemma vision-language model alongside an action expert that generates continuous action sequences. The training recipe pairs roughly 10,000 hours of broad pre-training data with smaller, carefully curated post-training sets, enabling tasks such as folding laundry and assembling boxes.

3. 22:30

   Adding sequential reasoning

   Levine describes giving robot policies intermediate reasoning steps, such as locating objects and breaking a task into subtasks. In experiments with OpenVLA, embodied chain-of-thought improved performance by about 50% over the base model. A separate high-level policy paired with π₀ also improved instruction following and handled spoken changes to a sandwich-making task.

4. 31:11

   Using reinforcement learning

   Levine describes online reinforcement learning for tasks including chip insertion, cable routing, and bimanual assembly, with rewards from an image classifier and corrections from human interventions. In one chip-insertion experiment, the robot reached full success after about 30 minutes; RL policies also completed tasks two to three times faster than the imitation-learning comparison. He also describes distilling specialist RL experience into generalist models and using self-generated goals to gather training data.

5. 40:31

   Open questions and future directions

   Levine compares robot-model training with language-model recipes: broad pre-training followed by focused post-training, potentially including reasoning and reinforcement learning. He says key open questions include data composition, model architecture, and how to add high-level reasoning. In the Q&A, he notes that adapting to new embodiments can take from single-digit to triple-digit hours, depending on the task, and that action chunks may limit responsiveness in dynamic settings.