Karol Hausman, co-founder and CEO, and Kevin Black, a researcher at Physical Intelligence, discuss robot foundation models and the path toward models that can control many kinds of robots with Going Direct’s host.
Kevin describes a three-stage progression: language models gain image input to become vision-language models, then gain robot action output to become vision-language-action models. The model takes images and instructions as input and produces actions such as joint positions or velocities. Karol frames these systems as pattern-recognition models that learn from data across language, images, and robot behavior.
Karol recalls the RT-2 demo in which a robot placed a Coke can on Taylor Swift’s picture, despite having no robot training data about her. Kevin says internet pretraining supplied the concept, while robot training connected that knowledge to actions. To support dexterous control at about 50 Hz, Physical Intelligence added an action expert; it also developed FAST, an action representation that is faster to train but slower at inference.
Kevin says Physical Intelligence open-sourced π0 to reduce the scientific risk that general robot intelligence may take decades to solve. OpenPI lets developers try the model and report uses and limitations, including applications the team had not anticipated. Karol says the company is building some hardware in-house while aiming to support other platforms, and argues that better intelligence could make capable robots less dependent on costly precision.
The team says π0.5 was designed to generalize to unfamiliar environments, using diverse data and a model that can both propose a subtask and carry it out. Karol reports that training across about 100 homes matched performance from training in the target home; Kevin says the paper’s results reach that level around 80 to 100 homes. They also discuss limits of imitation training: matching demonstrated actions may not optimize the outcome people actually care about.
Kevin explains that slow inference can hurt a robot because the physical world keeps changing; real-time action chunking lets the robot move while planning its next actions. Karol says longer video history could help a model adjust after failed attempts, but adding history can also reduce performance. The guests identify diverse data, better outcome-based objectives, reliable hardware, and real-world deployments as areas for further work.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Karol Hausman and Tobi Springenberg of Physical Intelligence discuss how robotic foundation models can learn tasks, generalize across settings, and improve through real-world experience.
