Ep#21 TesserAct: Learning 4D Embodied World Models

In this talk

1. 2:27

   Why build a 4D world model

   Hu Jun describes a world model as a system that takes an instruction or robot action and predicts future video states. TesserAct adds time to 3D scene information so it can model changing environments. He says video models may generalize better to unfamiliar scenes than 3D language models.

2. 8:47

   Architecture and training data

   TesserAct uses the CogVideoX video diffusion backbone with added depth and normal projectors, and predicts RGB, depth, and normals. Training uses supervised fine-tuning on simulated RLBench data and real-world datasets including RT-1 and Bridge, plus Something-Something V2. The team trained on 16 A100 GPUs for three weeks; Hu Jun says pretraining used roughly 100,000 videos.

3. 12:26

   Capabilities and evaluation

   The model generates action videos from images and prompts, including picking up different objects and using a sponge to clean a table. Prompts can specify robot embodiments such as Franka or Google robots. The team evaluates video, depth, normals, point-cloud quality, simulated action prediction, and novel-view synthesis.

4. 27:24

   Trade-offs and limitations

   Hu Jun says TesserAct generates higher-resolution video than the Meta model discussed, but generating a video can take about 200 seconds. It predicts from a single view, can struggle with unfamiliar objects, and may add a new robot arm instead of moving one already in the image. The team hopes to improve control using camera positions and numerical robot actions.

5. 31:25

   World models as data generators

   The group discusses combining real robot data with challenging simulator data, then using a world model to generate more training videos across scenes and domains. Hu Jun says this could help address limits on real-world data collection. He also notes that stronger models will still require substantial data and compute.