MIT Robotics - Nima Fazeli - Dexterous Multimodal Robotic Tool-use

In this talk

1. 0:56

   Why robots need tools

   Fazeli defines tool use as grasping one object to manipulate another, such as using a spatula to move an egg. Tools extend what hands can do through new contact shapes and force transmission, or protect people and their surroundings. His group is especially interested in assistive robotics that could help people age in place.

2. 8:16

   Learning how compliant tools deform

   Fazeli introduces VERDO, which combines partial visual point clouds, robot motion, reaction forces, and contact information to estimate a compliant tool’s shape and predict its deformation. In spatula and chain demonstrations, it maintains estimates even when the tool is partly hidden. A related method, NDCF, predicts contact probabilities as well as geometry; tests against measured sponge contact show useful estimates from sparse views, with errors such as overly elongated contact patches.

3. 20:10

   Turning contact plans into actions

   The group uses predicted contact patches in model predictive control to guide tasks such as scraping around an obstacle. Calamari takes an image and a human command, such as wiping dust, and generates contact plans. Fazeli says the approach was trained in simulation and transferred to real tasks including wiping and button pressing.

4. 22:41

   Controlling tools through tactile sensing

   Fazeli describes soft bubble tactile sensors that use a camera to read deformations in an inflated membrane. The control system combines tactile observations, robot forces, and object geometry to adjust actions. Demonstrations include keeping an Allen key horizontal, drawing triangles despite slippage, pivoting a spatula, and inserting a box using tactile feedback after vision is turned off.

5. 31:47

   Limits and next steps

   Fazeli says the two research thrusts have yet to combine compliant tools, compliant sensors, vision, and high-resolution touch in one system. Models generalize across tools with similar geometry and mechanical properties, but can fail when an unseen tool has very different compliance. He also notes that tactile contact estimates and cluttered environments remain challenging; his group is exploring dual-arm sensing and fusion of vision with touch.