RI Seminar: Negar Mehr : How Can Robots Learn to Interact and Coordinate with Others?

In this talk

1. 1:39

   Why robot coordination is difficult

   Mehr’s ICON Lab studies safe, intelligent interaction among robots and people. She points to Waymo cars stuck in traffic and warehouse robots stalled in a face-off as examples of coordination failures. In a two-robot object-transport task, multi-agent reinforcement learning struggled even after extensive reward engineering.

2. 7:02

   Coaching robots during training

   Mehr proposes using language and vision-language models as training coaches: they can break tasks into subtasks, define success criteria, and help diagnose failures. With this approach, robots learned to transport objects together, yield at a gate, and stabilize a seesaw for a partner. She also describes generating high-level policy code for humanoids, which completed tasks such as passing through a narrow gate and pushing a box without demonstrations or task-specific training.

3. 17:54

   Learning coordination from demonstrations

   Mehr explains that multi-agent imitation learning can fail when one robot deviates from a demonstration and its partner cannot adapt. Diffusion policies can capture different ways to coordinate, but collecting demonstrations from multiple robots is costly. Her lab instead freezes a pretrained single-agent policy and learns a coordination correction from a small set of multi-agent demonstrations, preserving skills such as picking up an object while enabling a partner to clean or remove a lid.

4. 26:46

   Using consensus to reduce forgetting

   Mehr applies a multi-agent idea, consensus, to continual learning in a single robot. The method estimates which policy parameters matter for earlier tasks, then constrains updates to preserve those parameters while learning new tasks. In sequential task experiments, it retained earlier abilities where standard fine-tuning forgot them; in mapping tests, it adapted to moved objects without keeping duplicate or outdated locations.

5. 42:17

   Questions on scale and when to use multi-agent models

   In the discussion, Mehr says her continual-learning method was tested on five sequential tasks, and she has not yet tested beyond that. She suggests treating components as separate agents when they make independent decisions, especially when humans or separate robots are involved. A coach provides feedback during training; the robots execute their learned policies without the language model.