David Held, a Carnegie Mellon University faculty member, presents research on perceptual robot learning at the UW Robotics Colloquium, introduced by Dieter Fox.
Held argues that robot perception should be designed alongside planning because robots need to predict how actions will change their surroundings. A general image description may miss details that matter to a task, such as cloth wrinkles or spilled droplets. His goal is to help robots manipulate unfamiliar, deformable, and visually challenging objects.
Held introduces SoftGym, a benchmark built on NVIDIA FleX with tasks including cloth folding, rope straightening, and pouring liquid. Its comparisons show a sizable gap between leading reinforcement learning methods and a planner with access to true state and dynamics, especially for cloth spreading and folding. He illustrates how latent-space planners can overlook small but important features, such as water droplets or cloth wrinkles.
For cloth smoothing, Held’s group estimates connections among visible points and uses a graph neural network to predict cloth movement. Trained on 2,000 simulated actions with one square towel, the method transferred to real cotton and silk cloths and a T-shirt, and handled unfamiliar shapes. For folding, a flow network predicts how cloth points should move toward a goal; a policy then selects grasp locations and infers placements, with actions using one or two arms as needed.
Held briefly describes applying the same object-motion-first approach to opening unfamiliar articulated objects, using predicted point movements to choose where and how to grasp. For cloth grasping, his group classifies corners and true edges, estimates grasp direction, and avoids locations with uncertain direction. He draws a shared lesson from these projects: reasoning about relationships among object parts, goals, and grasp directions can guide robot actions.
Held points to work on other deformable and transparent objects, tactile sensing, interactions between objects, and learning from visual demonstrations. In the discussion, he suggests graphs could represent interactions both within a cloth and between nearby objects. He also describes gradually tackling harder tasks as a path toward more general representations.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
David Held, faculty at Carnegie Mellon University, presents research on precise, generalizable robot manipulation at the 2026 Spring Robotics Colloquium, hosted by the Paul G. Allen School and introduced by Abhishek Gupta.
David Held, associate professor at Carnegie Mellon University and head of the RPAD Lab, presents research on precise, generalizable robot manipulation at a GRASP on Robotics seminar hosted by a Penn faculty member.
David Held, an assistant professor at Carnegie Mellon University's Robotics Institute, discusses perceptual robot learning and manipulation research with the umrobotics audience.
