MIT Robotics - Kostas Bekris - Towards Closing the Perception-Planning and Sim2Real Gaps in Robotics

In this talk

1. 1:17

   Why robotics needs tighter feedback

   Bekris argues that robots deployed in homes and other unstructured settings need more than a one-pass sense, plan, and act pipeline. Perception can miss objects or hallucinate them, plans can fail to find workable actions, and execution can go wrong. He advocates continuous perception updates, adaptable planning with fallback actions, and compliant hardware that can absorb some errors.

2. 7:43

   Packing objects with perception and recovery actions

   In an e-commerce packing task, Bekris’s team used simulated labeled data and adversarial learning to improve object-boundary predictions on real images, then combined pose hypotheses with scene physics to avoid interpenetrating objects. Their system used toppling, adaptive pushing, and corrective actions to handle wrong object orientations and imperfect placements. Ablation experiments showed these actions helped pack objects more reliably than a basic pick-and-place pipeline.

3. 17:27

   Tracking objects for precise manipulation

   Because compliant hands can move objects unpredictably, the team developed fast visual tracking to update object poses during manipulation. For known objects, a network compares a real image with a rendered image from the previous estimate; Bekris reported tracking at about 90 hertz and demonstrated submillimeter-clearance insertions. For unknown objects, BundleTrack combines feature matches, point-cloud alignment, and pose-graph optimization, running at about 10 hertz while reducing drift.

4. 35:51

   Transferring demonstrations across object shapes

   The team used BundleTrack to follow a human demonstration and reproduce its trajectory with a different object from the same category. Their non-uniform NOCS representation adds scale to map objects into a shared space, while a moving key point near the insertion area helps adapt the trajectory and avoid collisions. Bekris said the method worked on battery shapes not seen during training.

5. 43:40

   Using simulation to guide compliant locomotion

   For a tensegrity robot developed with NASA collaborators, motion plans that worked in simulation initially failed on the physical system, which provided IMU measurements rather than the full simulated state. The team used planning solutions to bootstrap learning, then tuned a differentiable simulator with real data; tuning took about 20 to 30 minutes in one example. A controller trained with a deliberately rougher simulated slope transferred to a real 15-degree slope, and the talk closes with the principle of iterating from real data to simulation and back.