RI Seminar: Peter K. Allen : Multi-Modal Geometric Learning for Grasping

In this talk

1. 3:05

   Learning grasps from data

   Allen contrasts traditional pipelines for recognizing objects and planning grasps with systems that learn from large simulated datasets. His group built a database of 238,000 grasps across 7,200 objects, four scales, and four hands, then transferred grasps to new objects by matching their geometry. The method worked on examples such as a glove, a controller, and a ukulele, but relied on assumptions about material properties.

2. 11:39

   Finding multifinger grasps from images

   Allen describes training a CNN on simulated grasps to identify promising palm and fingertip locations in RGB-D images. The approach targets multifingered hands and grasps from varied directions, without requiring an object mesh or pose estimate. It can plan around visible geometry, but may collide with or misread regions hidden from the camera.

3. 18:10

   Completing hidden object shapes

   To address occlusion, Allen's group trained a 3D CNN to complete object shapes from single-view depth images. Training used more than 400,000 examples from 608 mesh models and 726 poses; the system produced completed voxel grids that could be smoothed into meshes for grasp planning. Allen reports a 93% grasp success rate in their experiments and says the method also helps with collision-aware path planning.

4. 32:55

   Adding touch to visual completion

   Allen explains that depth alone can miss features such as a mug handle or the far side of a rubber duck. His group fused visual data with sparse tactile contacts gathered by guarded exploratory moves, using simulated data from 600 objects and about 40 touch samples per object. On real YCB objects, the combined method improved shape and grasp results; the reported lift success was 87.5%.

5. 44:22

   Tactile sensing and open questions

   Allen summarizes the value of combining vision and touch, then describes flexible tactile sensors designed to wrap around fingers and detect force and slip. In questions, he notes unresolved challenges including calibration noise, overlapping objects, material properties, and uncertain shape completion. He says better tactile sensing could help gather useful information without moving objects during contact.