Ep#2 Robot Utility Models

In this talk

1. 2:45

   From household demonstrations to utility models

   Pham describes Robot Utility Models as single-task policies intended to work in new environments without local fine-tuning. The earlier Dobb-E project required about five minutes of demonstrations in the target home and 15 minutes of fine-tuning. For RUM, the team trained on about 1,000 demonstrations per task, gathered across roughly 40 environments.

2. 16:03

   A portable tool for diverse data

   The team adapted a handheld tool with an iPhone so people could collect robot demonstrations without bringing a robot to each location. ARKit provides the phone’s pose over time, reducing the need for calibration and post-processing. Pham says the team used wrist-camera images as policy input, with actions at 3.75 Hz; the newer tool was shaped with help from Hello Robot.

3. 28:09

   Data diversity, quality, and policy choice

   In experiments with 1,000 demonstrations, data from 40 environments outperformed data from five on some tasks, especially object reorientation, where environmental diversity mattered substantially. Diffusion Policy and VQ-BeT had overlapping error bars, while smaller approaches such as ACT and MLP-BC performed somewhat worse. Pham also cautions that mixing lower-quality demonstrations can reduce performance, so dataset curation matters.

4. 10:38

   Retries, deployment, and open questions

   The base policies achieved about 75% success, rising to around 90% with retries guided by GPT-4o judging images from the robot’s head camera. Failures included unrecoverable changes to the environment and false success judgments, such as a vision model mistaking a hole in a door for an open door. The team’s roughly 80 MB policies ran at 30 FPS on a robot’s Intel NUC, and Pham described future work on chaining utility models, verification, and dexterous data collection.