Ep#6 with Yang Gao: FP3: A 3D Foundation Policy for Robotic Manipulation

In this talk

1. 2:11

   A point cloud policy for manipulation

   Gao presents FP3 as a 1.3-billion-parameter policy that takes point clouds instead of images. He says it handled unfamiliar settings with about 80% success, while the compared image-based models rarely succeeded there.

2. 5:44

   Pretraining and task adaptation

   FP3 is pretrained on DROID, which provides around 60,000 trajectories across 86 tasks, then adapted to new tasks with roughly 80 demonstrations per task using LoRA. Its 3D encoder, Uni3D, is pretrained on synthetic data and fine-tuned with the policy. Gao says DROID’s depth information is central to the method.

3. 16:20

   Generalization and evaluation

   Gao shows examples including towel folding and moving trash into a bin, with tests involving changed object positions, backgrounds, and distractors. Across four tasks, FP3 averaged about 95% success in-domain and 82% in the wild; the hosts note the results are based on around 20 trials per reported figure. He attributes remaining failures to factors such as sensor noise, occlusion, and gaps in training coverage.

4. 23:48

   Practical limits and next steps

   The robot moves slowly partly because demonstrations use slow VR teleoperation and inference runs separately from execution. Gao says more task demonstrations would be his first choice for improving a specific task, while broader pretraining could help across tasks. He also points to combining 2D and 3D inputs, and says larger models benefited from pretraining in his experiments.

5. 45:52

   Hosts assess the results

   After the interview, the hosts praise the generalization results and discuss the tradeoff between a depth sensor and a standard camera. They suggest 3D perception can simplify spatial actions such as moving an arm to a target, while noting sensor and compute costs depend on the robot setup.