SPRING 2022 GRASP on Robotics - Vincent Sitzmann, Massachusetts Institute of Technology

In this talk

1. 1:50

   Learning 3D scenes from images

   Sitzmann argues that robotics, graphics, and navigation rely on representations of the 3D world, while handcrafted formats can be difficult to learn and use with modern machine learning. He proposes neural scene representations trained from 2D images: a renderer recreates observed views, and image differences guide learning without ground-truth 3D data.

2. 4:52

   Neural fields and inference

   A neural field maps each 3D coordinate to properties such as geometry, color, or material. Sitzmann describes signed distance functions and ray marching for rendering, then compares two ways to infer a scene from limited views: a convolutional encoder and optimization of a latent code through the renderer. The latter handles unfamiliar camera distances and angles more reliably in his examples.

3. 22:48

   Planning from image-defined goals

   In a robotics project, a learned scene representation encodes the system state from images, while a dynamics model predicts how actions change that state. A robot can plan toward a goal specified by an image, including one from a camera viewpoint absent from training, by refining the image encoder’s initial state estimate through the renderer. The reported dataset included about 1,000 trajectories, each with 300 time steps and 30 views.

4. 35:12

   Matching task-relevant object features

   A second project uses demonstrations to transfer pick-and-place tasks to new mugs and poses. Descriptor fields identify corresponding features across object instances, such as a mug rim for grasping and its base for placement; an SE(3)-equivariant encoder helps handle rotated objects. The method estimates gripper poses, while conventional planning handles motion and collision avoidance.

5. 53:32

   Limits and open problems

   Sitzmann identifies limited generalization as a key constraint: the first robotics model was trained for a particular task, and its state representation could not readily accommodate added objects or tasks. He discusses compositionality, richer sensory inputs such as touch and audio, and whether some tasks might be represented in ray space instead of 3D. In the Q&A, he notes that representing many objects together can sharply increase the data needed, while factorized representations can reduce that burden.