π0: A Foundation Model for Robotics with Sergey Levine - 719

In this talk

1. 1:32

   Why build a general robotics model

   Levine says Physical Intelligence aims to develop one model that can be adapted to many robotic tasks, reducing the effort required to build a separate system for each application. He sees this as a long-term effort enabled by progress in transferable models, vision-language systems, and reinforcement learning.

2. 8:35

   How π0 turns perception into movement

   π0 combines a vision-language model with a smaller action component that uses diffusion to produce continuous robot movements in chunks. Levine says the training approach matters as much as the architecture, and describes the action component as a specialized partner to the larger model. The original model predicts about 50 steps at 50 hertz, while later work explored representing actions with fewer tokens.

3. 15:44

   Pretraining broadly, then refining carefully

   The team trained on roughly 10,000 hours of demonstrations from people operating different robots, then used about two to 20 hours of selected data to adapt the model to particular tasks. Levine says varied, imperfect demonstrations help robots recover from mistakes, while carefully chosen examples teach reliable task strategies. The laundry demo used the same robot software as other tasks; the preparation went into collecting useful demonstrations.

4. 40:21

   Open release and next research questions

   The released package includes fine-tuning code, base model checkpoints, and example models for DROID and ALOHA; users are expected to adapt the model with their own robot data. Levine also describes π0-FAST, which uses a different action tokenizer and trained about four times faster than a naive tokenization approach. Future work includes following detailed instructions and improving transfer across tasks, objects, settings, and robot types.