CoRL 2024 Day 2 09:00-10:30

In this talk

1. 4:49

   Why Meta works on robotics

   Mustafa connects physical-world interaction to FAIR’s AI research goals, and robotics to Meta’s work on AI agents, planning, AR, and VR. He maps the research from navigation and mobile manipulation through hand dexterity, highlighting the PARTNR household coordination benchmark with nearly 100,000 language tasks and OpenEQA with nearly 1,600 embodied question-answering tasks.

2. 8:42

   Touch as a perception problem

   Mustafa argues that tactile sensing could help robots move beyond basic pick-and-place manipulation. He describes DIGIT, a camera-based sensor that reads deformation in an illuminated gel, and traces work applying pose estimation, 3D perception, and landmark-based tracking to touch. He frames touch perception as sensor fusion, while noting that touch is local, takes time to cover a scene, and changes the object being sensed.

3. 15:04

   Combining vision and touch

   The neural-fields work combines an RGB-D camera with four tactile sensors on an Allegro Hand to build an object shape representation and track pose during manipulation. In simulated camera placements with heavy occlusion, adding touch improved pose estimation more than it did when the object was clearly visible. Mustafa says this slower perception can support longer-term decisions, while quick reactions such as catching a slipping object need a faster system.

4. 19:41

   Shared tactile models and richer sensors

   SPUR is a general-purpose touch encoder trained on nearly 460,000 tactile images from varied sensors. Mustafa reports an average improvement of nearly 95% over task- or sensor-specific models; in one transfer example, 10 samples raised classification accuracy from 9% to almost 60%. He also describes Digit 360, with over 18 sensing features and visual resolution of seven microns, plus modalities such as vibration, temperature, pressure, and smell.

5. 25:57

   Making tactile hardware easier to use

   Digit Plexus brings tactile sensors into a shared hardware and software platform, and Mustafa says its circuit designs, CAD models, firmware, and fingertip sensor are open source. He describes plans with GIDE to commercialize Digit 360 and with Wonik Robotics to build a hand with integrated touch sensors. In the Q&A, he notes that soft sensor materials can wear over time, and suggests deformable-object perception may need task-relevant features rather than exact shape reconstruction.