Fall 2023 GRASP SFI Andy Zeng, Google DeepMind, “From words to actions”

In this talk

1. 0:36

   Robotics as a test of general intelligence

   Zeng sees robotics as a way to study agents that make decisions in physical environments and improve through experience. Manipulation tasks expose challenges such as contact physics, fast feedback, and learning from limited data. In his team’s pushing work, small slips require the robot to adjust its motion to keep an object on course.

2. 6:46

   Why language models could help robots plan

   Zeng explains that language models can combine patterns learned from broad text collections to produce plausible plans, such as steps for a mobile robot to retrieve a drink. In the Socratic Models and PaLM-SayCan work, a language model breaks a request into steps, while language-conditioned policies such as CLIPort translate steps into pick-and-place actions. Inner Monologue adds visual checks after actions so the system can retry or replan when something fails.

3. 18:17

   Using code to connect instructions to actions

   Language can lose spatial and numerical detail, so Zeng explores models that translate instructions into robot code. Examples include changing an orange detector to an apple detector, interpreting “faster” by adjusting a velocity, and composing APIs with if and while logic. Code can also express different action spaces for different robot bodies and connect language goals to maps, optimization, and control.

4. 33:58

   Scaling data and adapting foundation models

   Zeng contrasts the vast text collections used to train language models with the limited robot data available. His rough estimate suggests that collecting a comparable robot dataset could take about 100 robots working for 24 years, motivating a staged approach: start with off-the-shelf models, then fine-tune as more robot data accumulates, and eventually train new models. PaLM-E combines image and language inputs and is fine-tuned on robot data, though its spatial reasoning remains limited by its training data.

5. 49:44

   Where language falls short

   In the Q&A, Zeng says language-based planning is less suited to precise manipulation, where contact and object dynamics matter. He points to video prediction as a possible complement: a planner could propose actions while a model imagines how the scene might change. He also stresses that robots must adapt to specific environments, since general assumptions, such as where a refrigerator is, may be wrong.