Learning Representations for Generalizable Robot Actions -- Congyue Deng

In this talk

1. 4:00

   Borrowing ideas from vision

   Deng frames imitation learning as a setting where robots train on demonstrations and then act in new but related scenarios. She asks whether visual representation methods can also provide useful structure for robot actions, and how to combine vision, language, and action in a shared latent space.

2. 8:44

   Using object features to guide policies

   Deng describes learning geometric, contrastive features from simulated point clouds, then training policies on top of them with robot demonstrations. The approach transfers across object shapes, textures, poses, and environments; simulation supplies feature pretraining, while real demonstrations train the policy.

3. 23:04

   Representing relations between hands and objects

   Deng observes that visual features tend to concentrate on object surfaces, although robot effectors also move meaningfully through free space. Her neural attention field represents relationships between scene points and hand points, using self-supervised cycle consistency to reorganize visual information. In tests, this reduced distracting responses in empty space and improved transfer in clutter.

4. 36:42

   Combining physical and semantic affordances

   Deng argues that action representations should capture both what a part means and how a robot can physically interact with it. For example, a blender component may be understood as a container semantically but treated as a button during an action. A language model decomposes appliances into parts and these two kinds of affordance, letting a robot reconsider its interpretation when an attempted action fails.

5. 44:32

   Structuring action learning around physics

   Deng notes that robots have access to dynamics, simulators, and opportunities to explore outcomes, unlike systems trained only on passive visual data. She suggests these active experiences may teach useful representations and that action should be integrated with vision and language in a way that reflects the physical world.