Keynote: Robot Foundation Models - Sergey Levine

In this talk

1. 0:01

   Foundation models meet robotics

   Levine frames robotic learning as a shift toward models that can transfer broad knowledge to new robots and tasks. He describes RT-2-X following a web-delivered instruction to move an eggplant into a pot, even though the model’s trainer had not worked with that robot.

2. 1:45

   Why diverse robot data matters

   Early vision-language-action models adapted multimodal language models by training them to output robot actions from images and demonstrations. Open X-Embodiment combined data from 34 labs and about 22 robot types. Levine says the resulting general model scored about 50% better on average than specialist methods across participating labs, while diverse training data improved the hardest language-following results threefold for one robot.

3. 6:27

   A motor-control component for dexterity

   Levine contrasts early models that represented actions as coarse text tokens with newer designs that add a dedicated component for smooth, continuous control. He describes π₀ as a model pre-trained on roughly 10,000 hours of data across robot platforms, then adapted to tasks such as folding laundry. Diverse pre-training helped it recover from mistakes and handle interruptions, including extra laundry or objects placed in its way.

4. 11:17

   Longer tasks and language supervision

   π₀.₅ adds a language-based reasoning step for tasks with multiple stages, such as cleaning a bedroom in a home it had not seen during training. Levine says people can also improve the model by coaching it in words, without providing new robot actions, when it already has the motor skills needed for the task. In their mobile-robot experiments, only 3% of the training data came from mobile robots.

5. 14:12

   Progress and open challenges

   Training across about 100 homes brought performance on new homes close to performance when trained directly in the test home, though it remained imperfect. Levine says current models still rely on imitation and need better task-success optimization, robustness, speed, planning, and generalization. He notes that PyTorch ports of π₀ and π₀.₅ are available to try.