Ep#86: RISE: Self-Improving Robot Policy with Compositional World Model

In this talk

1. 2:00

   A world model for policy improvement

   RISE uses a dynamics model to predict future observations and a value model to score them. Together, they act as a simulated environment where a robot policy can generate training data and improve through an online reinforcement learning loop. The world model is separate from the policy and can be discarded at inference, avoiding added inference latency.

2. 7:33

   Learning from successes and failures

   The policy trains on both teleoperated offline data and rollouts from the world model, because model-generated data can be biased. The dynamics model predicts images from multiple camera views, while a value model based on Pi 0.5 uses progress estimation and temporal-difference learning to score outcomes, including failures. Yang says the value model and dynamics model both need failure examples to learn meaningful distinctions.

3. 23:46

   Adapting the models and preparing the policy

   The dynamics model is pretrained on AI2-THOR and the Bridge Open Dataset, then fine-tuned for the target Piper arms. Yang says about 500 trajectories can make predictions look good, though failure cases are harder to learn because public robot datasets contain few of them. Before online improvement, the policy is fine-tuned with demonstrations, policy rollouts, and DAgger recovery data, including trajectories teleoperators create from constructed failure states.

4. 33:58

   Online improvement and its limits

   RISE uses advantage conditioning so the policy can learn from trajectories of different quality with a supervised flow-matching loss. During online improvement, the current policy generates fresh data that the world model scores; batches use six offline samples and four online samples, a ratio Yang says may need task-specific tuning. Yang acknowledges that an inaccurate world model can mislead training, and says offline data helps anchor the policy.

5. 13:49

   Dynamic tasks and future directions

   The discussion covers sorting small, moving LEGO bricks on a conveyor, a task that supervised fine-tuning alone did not solve. The policies run at 30 Hz, while RISE runs at 7 Hz, which limits performance on fast-moving objects. Yang says pixel-level world models may struggle with contact-rich tasks such as flipping a block, and points to simulation distillation and modular reward models as directions worth exploring.