Actuate 26 | What Does the GPT-3 Moment for Robotics Look Like? by Pete Florence

In this talk

1. 0:11

   Learning a task from one example

   Florence describes a goal: a robot watches one demonstration, infers the task, then performs it amid real-world variation. He points to children imitating actions as evidence that people can learn this way without verbal instructions.

2. 2:38

   A long history of demonstration learning

   Florence traces the idea from early industrial robots such as Unimate, which recorded and replayed motions, to MIT’s 1970 Copy Demo, which stacked blocks from an image. He also highlights research from about a decade ago that explored using attention to condition a model on a single example.

3. 6:45

   Gen 1.5 learns from prompts

   Florence recounts Generalist’s progression from copying LEGO towers to Gen 0 and Gen 1, then introduces Gen 1.5. He shows tasks prompted with one video, including pouring a cup, opening a pouch, and putting a marker in a cup; after a mistake, the robot corrects itself. He also describes combining prompts into a longer task and transferring a simulator demonstration to a real robot.

4. 14:44

   Fast adaptation and physical improvisation

   Florence says the model can adapt to a new task in one to ten gradient steps using one to five minutes of data; across ten short tasks, ten steps yielded about 80% average success. In other demonstrations, it repurposes a banana as a brush, uses a dustpan to collect a block, switches hands, clears paper from a bowl, and removes a LEGO brick stuck on a finger.

5. 18:04

   What the results suggest

   Florence summarizes Gen 1.5’s one-shot in-context learning, few-step adaptation, and zero-shot physical generalization. He credits Generalist’s team and says the model had been training for eight months.