Stanford University Professor Shuran Song explains how to teach robots manipulation skills through natural human demonstrations at an IEEE Robotics and Automation Society talk.
Song uses dishwashing to show the range of challenges: handling deformable objects and liquids, judging what “clean” means, and knowing when to stop. She argues that these decisions depend on experience, context, and physical intuition that is difficult to specify as explicit rules.
Song says demonstrations should be accessible to people without robotics expertise while preserving the detail of real skills. Teleoperation can feel unnatural, limits contact feedback, and requires robot hardware in the demonstration loop. Videos of people alone are easy to collect but leave a large gap between human and robot bodies.
With Universal Manipulation Interface (UMI), people use handheld grippers with cameras to record demonstrations, haptic feedback, and end-effector trajectories. Song says the shared interface can work across different robot hardware, and demonstrations can train diffusion policies for tasks such as dishwashing, cloth folding, and tossing. In one study, 12 people collected more than 400 demonstrations in over 30 environments; the resulting policy worked in new settings, including outdoors.
Song describes transferring a manipulation policy to a quadruped by training a whole-body controller in simulation, without more human demonstrations. Follow-up projects add sensors and dexterous hands; DexUMI records finger movement and tactile contact. In response to a manufacturing question, she says high reliability can be improved by collecting data on failure cases through human DAgger, with real-world reinforcement learning as another possible approach.
Song says the team open-sources UMI hardware and algorithms and provides a tutorial. She describes launching the UMI Data website as a place to share demonstrations and related work. She encourages researchers to contribute data and improvements so others can build on them.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Shuran Song, a robotics researcher at Columbia and now Stanford, presents at the CMU Robotics Institute seminar on learning to use real-world dynamics in robot manipulation.
Shuran Song, a robotics researcher at Columbia University, presents her work on dynamic manipulation of deformable objects at the GRASP Lab Spring 2022 SFI.
