Ep#79: Rhoda AI - Causal Video Models Are Data-Efficient Robot Policy Learners

In this talk

1. 1:38

   Using video to learn robot policies

   Tong Zhou frames general-purpose robot control as a data problem: robotics has far less interaction data than language and vision. Rhoda AI uses web video to train a causal video model that predicts what happens next, then a smaller inverse dynamics model translates those predicted frames into robot actions. The system runs in a streaming, closed loop.

2. 5:57

   Why predict video before actions

   The researchers distinguish their method from DreamGen, which uses generated video to create training data, and DreamZero, which predicts video and actions jointly. Rhoda AI predicts video first, then maps it to actions, keeping the video objective consistent from pretraining through post-training. They describe the inverse dynamics step as constrained because it tracks motion shown in the video rather than deciding what to do.

3. 18:53

   Pretraining, memory, and demonstrations

   Rhoda AI says it trained its causal video model from scratch and used hundreds of years of web video, including general internet footage as well as egocentric video. The team reports that examples in its blog used fewer than 20 hours of robot data; industrial scanning and bread-cutting tasks used 10 to 20 hours and ran continuously at customer sites. Long visual context helps with multi-step tasks such as unfolding, inspecting, and refolding a T-shirt, and the researchers also show robots following new tasks from human video demonstrations.

4. 42:14

   Real-time control and open challenges

   To keep the robot moving during inference, Rhoda AI overlaps video prediction with execution of action chunks. It feeds the current action back into the system to reduce oscillation when consecutive predictions select different objects. The researchers say prediction horizon should depend on how dynamic and precise a task is, and that causal attention enables KV-cache reuse for faster generation. They identify more capable pretraining, longer context, dexterous control, multimodal reasoning, and better evaluation as priorities.