Winter 2023 Robotics Colloquium: Luca Carlone (MIT)

In this talk

1. 5:37

   Robot perception goals

   Carlone frames perception around robustness, scalability, and scene understanding. He argues that geometric SLAM maps can scale well but provide limited semantic knowledge, while supervised 2D and 3D methods add semantics but can be brittle outside their training data and harder to scale.

2. 9:40

   Hierarchical maps and scene graphs

   Carlone presents metric-semantic maps and 3D scene graphs as a way to represent geometry, objects, rooms, and larger places together. Organizing concepts hierarchically can reduce storage needs and represent relations such as objects being inside rooms. His group’s Hydra system builds these graphs incrementally in real time, including on a Xavier NX, and uses a neural tree to infer room labels such as kitchen or bedroom.

3. 33:01

   Certifiable robust estimation

   For safety-critical tasks such as estimating a car’s pose, Carlone says common methods like local optimization and RANSAC can return incorrect answers without detecting failure. His group develops certifiable methods that either establish optimality or flag an estimate as untrusted. Using polynomial optimization and semidefinite relaxations, their solvers sometimes find optimal solutions even with high outlier rates, including tests with up to 80–90% outliers.

4. 50:38

   Certification for self-supervised learning

   Carlone connects estimate certification to training perception networks without labels. The system treats certified pose estimates as pseudo-labels and uses them to improve the network on real data. In tests on object pose estimation, performance improved through training and compared well with supervised CosyPose, though Carlone notes there are no guarantees and that a weak starting model may produce too few usable examples.