Spatial Intelligence as Structured Representation for Robotics - Wenlong Huang 2.20.2026

In this talk

1. 2:32

   Why robotics needs better transfer

   Huang contrasts the promise of imitation-trained policies and vision-language-action models with robotics’ limited supply of training data. He describes transfer efficiency as a way to measure how much pretraining helps a model learn a downstream task, and cites an analysis suggesting robotics transfer was roughly 100 times less efficient than language modeling under the compared setups. He argues that world modeling could help by learning from varied interaction data without requiring task-specific demonstrations.

2. 19:52

   Representing robot actions in 3D

   Huang introduces PointWorld, which takes RGB-D observations, a robot description, and robot actions, then predicts how points across the scene will move. Robot actions are encoded as 3D point flows on the grippers, derived from the robot’s geometry and kinematics. This representation supports different robot shapes and configurations, while the model predicts effects such as cloth deformation, object articulation, and contacts under gravity.

3. 29:43

   Data and scaling results

   Recent 3D vision models helped the team annotate more than 60% of the DROID dataset. Combined with simulated interactions from BEHAVIOR, filtering yielded about 500 hours of 3D interaction data. Huang reports that larger datasets and models reduced prediction error; training across Franka and Galaxea embodiments worked best when actions were represented as gripper point flows. In one evaluation, pretraining let a model approach lab-specific results without fine-tuning, while using 5% of fine-tuning data beat a from-scratch baseline.

4. 46:21

   Planning with the world model

   The team pairs PointWorld with model-based planning to generate behaviors at test time, without demonstrations or fine-tuning. Examples include pushing, deformable-object manipulation, articulated objects, and tool use. Huang notes that the model predicts point motion rather than appearance, can mistake correlation for causation, and assumes commanded actions can be carried out.

5. 50:39

   Toward a complete robotics system

   Huang frames the world model as a source of physical intuition that links robot actions to changes in the scene. A separate goal specification would tell the system what it should change, and he suggests spatially capable vision-language models could help provide it. A complete system would combine goal reasoning, physical predictions, and learned motor skills grounded in 3D.