2024 Winter Robotics Colloquium: Ani Kembhavi (Allen Institute for Artificial Intelligence (AI2))

In this talk

1. 0:59

   Why robotics needs more data

   Kembhavi argues that machine learning systems improve with large, varied datasets, while gathering robot demonstrations in the physical world is costly and slow. He proposes simulation as a way to generate data at much greater scale, while noting that existing simulators often lack environmental and object variety and can differ from reality.

2. 7:00

   Generating varied scenes and objects

   ProcTHOR uses procedural rules to create many apartment layouts, helping navigation agents perform better in unfamiliar spaces. Kembhavi then introduces Holodeck, which uses language models to turn scene descriptions into layouts and object lists; spatial constraints and optimization help place objects sensibly. Large web-sourced collections such as Objaverse and Objaverse-XL add diverse assets for furnishing those environments.

3. 15:43

   Training SPOC from simulated episodes

   AI2 generated more than 10 million imitation-learning episodes across over 100,000 simulated homes, using privileged planners to create tasks such as finding, fetching, or comparing objects. SPOC takes RGB images from two cameras and task instructions, then predicts actions from training sequences. Kembhavi reports that it can transfer to real apartments without real-world training, though detection errors and wall collisions remain problems.

4. 35:14

   What improves performance and enables customization

   More training episodes, a larger variety of house layouts, longer context windows within practical limits, and stronger visual encoders all improved results; changing the model architecture had less effect. Phone2Proc scans a user's home and generates varied simulated versions for tailored training. Kembhavi also describes promptable behaviors, where reward preferences such as safety or efficiency can shape a robot's actions at inference time.