Spring 2026 GRASP SFI - Chuning Zhu, University of Washington

In this talk

1. 0:22

   Why robot learning needs more data efficiency

   Zhu opens with a rice-pouring demonstration that takes 20 seconds but requires 40 seconds of cleanup after an operator error. He argues that imitation learning depends on costly, high-quality robot demonstrations and often ignores the temporal information in videos. Passive videos of people and their environments could provide additional data about interactions and physical behavior.

2. 4:52

   One model for actions and video

   Unified World models action and observation diffusion in one transformer, using separate noise schedules to represent policies, forward and inverse dynamics, and video prediction. The approach can train on robot trajectories and on videos without action labels. On DROID tasks, Zhu reports about 50% success for the baseline, 78% after pretraining, and higher performance when videos are included in co-training.

3. 18:55

   Scaling joint video and action generation

   Zhu describes GR00T-Dreams, which starts from the pretrained Wan 2.1 14B video model and generates multistep visual rollouts alongside actions. He notes that larger models can take longer to train and run; engineering optimizations reduced reported inference time from about five seconds to 300 milliseconds. At this scale, co-training with human or cross-embodiment videos improved performance, and the model showed some generalization to unseen verbs.

4. 25:27

   Predicting task-relevant semantics

   Zhu argues that pixel prediction can overlook small but crucial details, such as whether a screw stays in a gripper, while spending compute on irrelevant parts of a scene. Semantic World Models instead predict answers to task questions about future states, such as whether rice is in a pot or a block touches another block. A planner can refine candidate actions using the model's answer likelihood; experiments showed strong results on manipulation tasks and compositional generalization across block colors and shapes.

5. 41:49

   Limits and open research directions

   Semantic models require future question-and-answer training data and task criteria, which Zhu says were manually designed in the reported experiments. He sees a tradeoff between pixel detail and language's compositional structure, with longer descriptions potentially becoming a bottleneck. He identifies variable-resolution video, reasoning traces, tactile signals, long-term memory, and model self-improvement as areas for further work.