Ep#36: Whole-Body Conditioned Egocentric Video Prediction

In this talk

1. 0:15

   Why egocentric human data matters

   The discussion describes a dataset with about 2,000 hours across 50 scenes, capturing varied activities such as cooking, walking, and handling objects. Uton argues that egocentric footage and diverse human actions make the model learn links between movement and visual change, while third-person video can leave most of the background unchanged.

2. 5:02

   How the model predicts future video

   The approach uses an autoregressive diffusion model with video tokens, action tokens, and causal attention to predict future frames from past ones. Action is supplied through adaptive layer normalization, and random time skips mix shorter and longer motion scales, including examples at 0.5, 1, 2, and 4 FPS.

3. 10:35

   What action control can represent

   Rollouts show navigation and hand movements, including moving forward, turning, and raising a hand. The same semantic task, such as opening a drawer or door, can involve different poses and object interactions depending on the scene. Uton frames the model as learning probable outcomes from action and context, while noting that closed-loop observations can help correct uncertain predictions.

4. 21:49

   Evaluating control and using predictions for planning

   Adding noise to the conditioning actions worsened visual prediction, and a text-conditioned model struggled with fine-grained hand manipulation. The team also evaluated camera and hand poses using VGGT, beyond pixel similarity. For planning, the model can simulate candidate actions toward a visual goal and score the resulting predictions; training used about 32 H100 GPUs for a 1 billion parameter model over a couple of days.

5. 35:41

   Open questions about world models

   Uton raises the need to test whether realistic video reflects coherent understanding, such as whether a model predicts two apple halves after an apple is cut. The conversation also considers curiosity-driven learning and whether interactive exploration could be more efficient than relying on large datasets and compute. Uton says the right exploration objective may be difficult to define with a conventional preset reward.