Laure Soulier: How to Leverage Language models for Robotics: a Literature Review

In this talk

1. 1:31

   Language models and their capabilities

   Soulier explains that language models predict likely tokens and use Transformer self-attention to interpret context. She highlights prompting, few-shot learning, and scaling as useful properties, while noting that larger models can handle more complex tasks such as reasoning and planning.

2. 10:26

   Grounding goals in robot actions

   A robot must connect a natural-language goal and its scene to executable commands. Language models can break goals into subtasks, ask people to resolve ambiguity, and use environmental feedback to refine plans. Soulier points to the ALFRED dataset as an example of mapping goals through subgoals to actions.

3. 13:52

   Combining language, vision, and action

   Soulier reviews approaches ranging from text-to-sequence models such as SL3 and SayCan to multimodal systems including CLASP, Voltron, and PaLM-E. These combine language with images, object properties, or scene state to select actions or generate plans. She notes that CLASP's training on the relatively simple Language Table dataset limits how much it can demonstrate broad robotics knowledge.

4. 21:33

   Interactive planning and learning from experience

   Some systems use scene descriptions, success predictions, and robot feedback to revise plans; others produce logical commands directly, with examples in the prompt helping the model use those commands. Soulier also describes tool-calling systems that query target and landmark finders, and training methods that use simulated trajectories, including failed attempts, to create data for fine-tuning.

5. 34:48

   Open questions and practical limits

   Soulier raises questions about whether plans should use natural language or directly produce logical actions, how much scene grounding requires real execution, and whether models generalize across tasks. She also asks whether prompting can support lifelong learning and notes that fine-tuning may be needed for complex, specialized work, though in-context examples could sometimes suffice.