Ep#31: Vision in Action: Learning Active Perception from Human Demonstrations

In this talk

1. 1:11

   Why robots need a moving camera

   Shong argues that fixed cameras can miss task-relevant views during manipulation, such as when reaching into a bag or working around an occluder. Wrist cameras can become blocked while a human operator still sees the scene, creating a mismatch between the demonstration and the robot’s observations. The project’s approach gives the robot an active camera so it can look for useful information.

2. 7:54

   Building a comfortable teleoperation setup

   The team used a six-degree-of-freedom neck and VR teleoperation to collect demonstrations. To reduce discomfort from control latency, they turned RGB-D observations into a world-frame point cloud, rendered from the operator’s latest headset pose at over 100 Hz, while camera and point-cloud updates ran asynchronously at 10 Hz. The point cloud reduced motion sickness for six of eight volunteers, though its lower fidelity and update latency can hinder dynamic tasks.

3. 16:42

   Training policies to search and act

   The demonstrations covered finding objects under an occluder, locating an item inside a bag, and searching shelves with randomized layouts. Shong says a pretrained RGB vision encoder helped the policy locate objects, while the point-cloud DP3 baseline often failed at semantic understanding. In one user study, six of eight volunteers preferred point-cloud rendering; the policy dataset contained roughly 150 to 300 demonstrations per task.

4. 25:00

   Testing the value of active vision

   In camera ablations, policies using chest and wrist cameras often failed when those views were occluded, while the active head camera could seek a better view. Adding wrist cameras to the active-camera setup slightly reduced performance in the team’s low-data experiments, which Shong attributed as a possibility to noisy inputs and overfitting. He notes that point-cloud policies performed better when test camera poses differed substantially from training, so representation choice depends on the training and deployment conditions.

5. 30:52

   Open questions and next steps

   Shong says VR teleoperation can be tiring and difficult to scale, and points to practical data-collection interfaces as an open problem. Future work could explore active perception with a mobile base, how to represent actions across the torso, neck, and arms, and whether attention mechanisms explain purposeful visual search. He also sees potential in RGB policies trained on diverse camera poses and notes that Vision Pro eye-gaze data was not available through an open API.