Mohit Shridhar on Acting with Perception and Language | Toronto AIR Seminar

In this talk

1. 1:04

   Connect perception directly to action

   Drawing on ideas from Rod Brooks and others, Shridhar questions robotics pipelines that first build human-designed world models and then plan actions. He argues that perception and action can instead shape each other, with actions helping determine which visual information matters.

2. 4:08

   A language-guided 3D manipulation policy

   Perceiver-Actor learns one six-degree-of-freedom manipulation policy from demonstrations, using spoken or written instructions to specify tasks. It takes an RGB-D voxel scene and a language goal, then predicts a location, rotation, and gripper state; the input locations also define possible actions. Shridhar shows examples including sorting produce, opening a drawer, and using a stick to move a ball.

3. 21:18

   Represent scenes as 3D patches

   Shridhar says 3D voxel patches can help with depth, camera changes, and the mismatch between 2D image transformations and real three-dimensional scenes. Because a voxel grid can produce thousands of patches, the system uses Perceiver IO to compress them into a smaller learned space before applying Transformer layers. The output returns to the voxel grid so each location can be scored as a candidate action.

4. 37:42

   Train from demonstrations and action outcomes

   The team records robot demonstrations with an HTC Vive controller and extracts key points when motion stops or the gripper state changes. A single trajectory can provide about 100 training examples, and the talk reports 53 demonstrations across tasks, with collection time varying from minutes to about an hour for sweeping. The model learns action locations and discretized rotations and gripper commands through supervised classification.

5. 43:14

   Where language and action-centered models fit

   Shridhar suggests that object detection and segmentation may emerge from systems trained to act, while acknowledging this as a longer-term idea with limited evidence so far. In discussion, he describes language as especially useful for task instructions and future interactions among people and robots, rather than as a replacement for physical practice. He recommends simulation for reproducibility alongside real-world work, and clarifies that the simulated and real-world policies in this project were trained separately.