RoboBoom
AI Agent Frontier· March 29, 2026· 1:00:13

Prof. Sergey Levine: Robotic Foundation Models

Professor Sergey Levine of UC Berkeley and Physical Intelligence speaks with the AI Agent Frontier Seminar about robotic foundation models and how robots can learn to act reliably in the physical world.

In this talk

  1. 0:35

    Why physical interaction matters

    Levine contrasts a language model that can explain how to do household chores with a robot that can actually perform them. Everyday tasks such as folding laundry and cooking require planning across semantic, visual, spatial, and motor levels. He argues that learning through real-world interaction can teach AI about physical phenomena beyond human-made environments such as the web.

  2. 6:34

    From language models to dexterous action

    Planning through code can handle logical tasks, but many hand movements rely on skills people cannot easily describe step by step. First-generation vision-language-action models such as RT-2 represented actions as text, while newer designs such as π₀ use a dedicated action expert to produce continuous movements. Levine shows π₀ folding laundry and recovering from some mistakes.

  3. 21:52

    Reasoning and teaching robots

    Embodied chain of thought lets a vision-language-action model identify objects and break instructions into subtasks. Levine says adding this reasoning improved OpenVLA's performance, even compared with Google's much larger RT-2-X. In π₀.₅, people can teach tasks through verbal instructions, and the model can use those lessons to act autonomously in unfamiliar homes.

  4. 32:17

    Pooling data for generalization

    The RT-X project pooled robot data from more than 30 research labs; its generalist model averaged 50% better performance than the labs' task-specific models in their own settings. Levine describes a training recipe that combines broad, diverse pretraining with smaller, high-quality post-training data. Pretraining also helps robots recover from errors, while data from other robot types can benefit platforms that contributed little data themselves.

  5. 39:51

    Reinforcement learning for reliability and speed

    Levine presents reinforcement learning as a way to improve the robustness and throughput of generalist robot policies. Physical Intelligence's π-Star 0.6 uses a value function to judge whether actions move a task toward success, then uses new autonomous experience to refine that judgment. Examples include making espresso for about 13 hours and assembling chocolate boxes, with throughput roughly doubling on several tasks compared with supervised fine-tuning.

Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.

More talks