Robots that learn from machine dreams

In this talk

1. 0:04

   Generative data for robot learning

   Robot training needs varied examples that reflect physical interactions with the world. LucidSim uses generative models to create many different images on demand, giving robots a flexible source of visual training data.

2. 0:39

   Building simulated visual experiences

   LucidSim is a simulated environment for teaching robots visual parkour. It renders scenes with generated images so a robot can encounter different ways the world may appear, supporting mobility in human environments.

3. 1:15

   Adding physics and motion to images

   The team uses a physics engine to guide image generation and ControlNet to shape image composition using depth or semantic information. Dreams in Motion extends a generated image into later frames, warping pixels to reflect changes in the robot’s camera view.

4. 2:05

   Learning from the robot’s own actions

   Training on data from a different robot may not work well. The research finds that data produced by the learning robot’s own actions, called on-policy data, accounts for most of its eventual performance.

5. 2:28

   From individual skills to longer tasks

   The program aims to develop capable individual policies, build a broad set of skills, and help robots reason and explore across tasks with many steps. The longer-term goal is for robots to handle open-ended activities without learning every skill from scratch in the real world.