Zhenjia Xu on Deformable Object Manipulation beyond Pick-and-place | Toronto AIR Seminar

In this talk

1. 1:19

   Why deformable objects are hard to manipulate

   Xu describes ropes, cloth, and dough as common deformable objects. Their changing shapes, complex dynamics, and many degrees of freedom make them difficult to model and control with sparse robot contacts. He argues that pick-and-place alone is not enough, and that different materials and tasks can call for tailored actions or tools.

2. 6:57

   Using airflow to unfold cloth and open bags

   Xu presents “Dexterity with Air,” a system that uses an air pump alongside two robot arms to manipulate fabric. Airflow can apply force across a broad area, reach beyond the arms, and move quickly; the system uses visual feedback to choose and adjust actions. In tests, it unfolded a shirt in two steps and handled larger cloth that fling-based methods struggled with. For bag opening, a policy trained on 4,000 real-world samples collected over about ten hours learned to adjust its blowing angle and generalize across bags and grasp poses.

3. 33:02

   Adaptive cutting of multimaterial objects

   Xu introduces a robot cutting policy for objects with soft material surrounding a firmer core, using avocado and mango as examples. A differentiable simulator built with Taichi and the material point method supports trajectory optimization for removing soft material while avoiding collisions and limiting energy use. During real-world cutting, a low-cost force sensor supplies binary collision signals; the system updates its estimate of the hidden core and adapts the knife path. Xu says the same policy was used across several fruits, without explicit material-property estimates or visual feedback.

4. 47:05

   What the evaluations and discussion show

   Xu compares adaptive cutting with policies that follow the estimated core contour or do not adjust their path. Those baselines can get stuck, cut away too much material, or rotate the knife abruptly, while the adaptive policy can accept removing less material to avoid future collisions. For evaluation, success meant finishing within ten collisions and staying under a set energy threshold. Xu identifies material estimation, handling unstable objects, and improving state estimation as possible areas for further work.