2023 Fall Robotics Colloquium: Coline Devin (Google Deepmind)

In this talk

1. 3:50

   RoboCat’s goal and design

   Devin describes the aim of lowering the cost of teaching robots new skills with a model trained across tasks and robot embodiments. RoboCat turns images, robot state, goals, and actions into tokens, allowing it to handle robots with different inputs and action dimensions. A goal image specifies the desired result, such as stacking colored objects, without requiring an exact match to the pictured arrangement.

2. 9:43

   Training and self-improvement cycle

   The first model used mostly simulated examples, plus earlier real-world demonstrations from a Sawyer arm. Devin’s team fine-tuned it on new tasks using 500 demonstrations, then gathered additional real-world attempts and labeled likely successes and failures. They added this experience to the training pool; the second iteration covered four embodiments and 240 tasks.

3. 18:33

   Results and limits of transfer

   Performance depended on the task and the amount of data: the inverted pyramid reached 17% success with 500 demonstrations, while gear insertion benefited from more teleoperation data. RoboCat was not trained with sim-to-real alignment techniques such as visual randomization or system identification. Devin also reports that robotics experience helped RoboCat learn new tasks from limited demonstrations better than the tested pretrained vision models in most comparisons.

4. 27:03

   What more data changed

   RoboCat did not handle a held-out blue-on-green stacking setup reliably without examples, but learned it quickly after 100 demonstrations in one test. Adding self-generated attempts improved the generalist model over adding demonstrations alone on the evaluated tasks. The second iteration also performed better on tasks from the first iteration, and Devin pointed to language goals, reinforcement learning, and longer context as possible next steps.

5. 39:42

   Open robotics data and discussion

   Devin briefly introduces Open X-Embodiment, a collection of robotics datasets from 21 institutions, annotated with language instructions. Unlike RoboCat’s varied action spaces, the discussed RT-1 and RT-2 training used aligned action dimensions across datasets. In questions, Devin notes that the broader approach depends on data, and that tactile sensing remained an open challenge for the team.