Ep#101: Coupled Local and Global World Models for Efficient First Order RL

In this talk

1. 2:09

   Why use a learned world model

   Joseph explains that manipulation is harder to simulate than locomotion because objects and environments change during interaction. Their approach learns a simulator from real robot data, avoiding the expense of collecting large numbers of real-world training rollouts.

2. 4:24

   Coupling global and local models

   The method uses a large image-based world model to generate trajectories and a smaller latent-space model to compute policy gradients. The large model stays frozen during training, while the smaller DreamerV3 model is updated around the current policy so backpropagation remains practical.

3. 7:32

   Training data, rewards, and model errors

   The team trains on unstructured teleoperation data, collecting four hours for Push-T, 12 for Go To, and 20 for the G1 task. For some tasks, they label whether frames match demonstrations to create a binary reward. They also improve the global model by deploying policies that exploit its errors, observing the failures on the robot, and training on those examples.

4. 17:09

   Results on three robot tasks

   The experiments cover pushing a T-shaped object, guiding a quadruped to push a cube into a goal, and lifting a box by its handle with a G1 humanoid. The researchers report higher sample and time efficiency than their PPO and DreamerV3 comparisons, with zero-shot success rates of 9/10, 9/10, and 8/10 respectively. They also show search behavior that behavioral cloning did not learn from the limited demonstrations.

5. 26:04

   Limitations and future directions

   The models and policies were trained from scratch, and reward labeling still takes human effort on some tasks. The researchers hope to start from pretrained world models and policies, automate reward labeling with vision-language models, and use action-aware models to improve exploration. Ruhollah cautions that transferring video-model priors into robot models remains difficult and that real robot data is still needed to capture each embodiment's abilities.