Fall 2025 GRASP SFI - Suraj Nair, Physical Intelligence

In this talk

1. 0:13

   Why robots need a generalist foundation

   Nair describes the goal as robots that can follow language instructions and handle unfamiliar homes, objects, and tasks. He compares this flexibility to people, who can do many kinds of work and then specialize. He argues that foundation models offer a useful recipe: broad training first, followed by task-specific adaptation.

2. 6:39

   Building and training Pi 0

   Pi 0 combines a pretrained PaliGemma vision-language model with a flow-matching action component and data from multiple robots and tasks. Nair says the team had collected more than 10,000 hours across dozens of environments, hundreds of tasks, and over a dozen robots. The model handled several tasks on different robot types without fine-tuning, while a smaller model trained without VLM pretraining performed worse.

3. 13:09

   Compressing actions for faster training

   Nair explains that adding a flow-matching action head made training slow and could interfere with knowledge from VLM pretraining. The FAST approach applies a discrete cosine transform to action chunks, then quantizes and compresses them into tokens. He reports that this reached Pi 0’s performance in just over 160,000 steps, compared with more than a million for the original approach; knowledge installation combines discrete-action training with a flow-matching component for faster inference.

4. 23:57

   Taking Pi 0.5 into unfamiliar homes

   For Pi 0.5, Physical Intelligence added data from mobile robots operating in many homes, along with web data and the earlier cross-robot dataset. The model uses a high-level policy to propose subtasks, such as picking up a pillow, and a low-level policy to control the robot. In four tested tasks, training across about 100 environments produced performance in unseen scenes similar to training in those scenes; removing general pretraining sharply reduced generalization.

5. 42:43

   Lessons and open problems

   Nair emphasizes broad evaluation across multiple unseen settings, since a single test environment gave noisy results. He says task-progress measures can reveal differences more clearly than binary success rates, and that high-quality post-training data and consistent camera setups matter. Open questions include scaling useful data collection, improving performance beyond current success rates, and finding action representations that preserve both dexterity and pretrained knowledge.