Fall 2026 GRASP SFI - Hanxiao Jiang, Columbia University

In this talk

1. 0:19

   Where physics fits in robotics

   Jiang asks how physical knowledge can support different robotics approaches, including programmed control, learning from demonstrations, reinforcement learning, planning, and agentic systems. He argues that physics can inform all of them, with explicit dynamics models offering a way to predict the results of actions.

2. 13:24

   From images to simulation

   Jiang describes Fidgets, a CVPR 2025 project that builds a 3D scene from an image, simulates object motion, and renders the result as video. The approach lets users change material properties or initial velocity, but its estimates may be plausible without matching the specific object, and errors in geometry or contact can affect the result.

3. 23:01

   Building and testing physical digital twins

   Phase Twin uses videos of real interactions to estimate the properties of deformable objects such as ropes and clothing. Jiang describes using the reconstructed model to simulate new actions, plan robot motions, and compare robot policy performance in simulation and reality; across three tasks, simulated and real success rates showed a strong positive correlation.

4. 36:02

   Scaling simulation for robotics

   Deform360 collects about 2,000 interaction sequences across 17 categories, using 41 cameras and synchronized tactile data. Boba improves batched Gaussian simulation throughput from about 30 to 3,000 FPS on an RTX 4090, which Jiang says enabled training a vision-based reinforcement learning policy for rope manipulation.

5. 43:43

   Lessons for useful physics models

   Jiang highlights physics as a useful prior when data is limited and describes applications spanning dynamics modeling, policy evaluation, and reinforcement learning. He says a simulator need not be maximally detailed: its accuracy, speed, and scale should suit the robotics problem and the interactions the robot needs to predict.