Unlocking Scalable Robot Learning in the Real World–Karl Pertsch (UC Berkeley and Stanford)

In this talk

1. 1:00

   Why robot learning needs to scale

   Pertsch contrasts general-purpose AI models with robot policies that can act in the physical world. Existing robot policies can perform complex tasks, but often need task-specific data and are brittle to changes such as lighting or camera position. He proposes combining diverse robot data with high-capacity models to build policies that can handle more tasks and environments.

2. 7:07

   Building shared robot datasets

   Pertsch describes DROID, a community effort that collected about 70,000 teleoperated episodes across roughly 560 scenes and 80 tasks, including in people’s homes. He also co-led Open X-Embodiment, which pooled more than a million episodes across 22 robot embodiments. Sharing and reusing data lets researchers build on existing collections instead of starting from scratch for each project.

3. 13:45

   Designing policies for dexterous control

   Pertsch traces work from modular policies such as Octo to OpenVLA, an open-source vision-language-action model trained on pooled robot data. Standard action tokenization struggled with fast, dexterous tasks because it produced long sequences of repetitive, low-information tokens. FAST uses discrete cosine transform compression to shorten action sequences; Pertsch reports over tenfold compression on some tasks and faster training for π₀-FAST than the original π₀. A fine-tuned policy also made reasonable attempts at new tabletop tasks on DROID robots without further in-lab training.

4. 34:13

   Making robot evaluation more reproducible

   Real-world robot evaluations are slow and difficult to reproduce, so Pertsch and colleagues explored simulation as a scalable alternative. Across eight environments, simulated success rates reflected real-world performance better than validation loss did. More realistic visuals improved evaluation reliability, while changing physics parameters or simulators made little difference on the static pick-and-place tasks they studied. The environments still require manual work and do not cover every task.

5. 42:51

   Limits and next research directions

   Pertsch says current policies still struggle with unfamiliar objects, dexterous tasks, reliability, and safety. He identifies opportunities to combine robot data with internet-trained models, learn dexterous skills from human video or simulation, and make policies easier to teach through natural-language or visual feedback. In the Q&A, he notes that zero-shot success rates around 50% to 80% fall short of the reliability needed for real applications, and suggests reinforcement learning could help policies improve on the job.