Dieter Fox, senior research director at NVIDIA and professor at the University of Washington, discusses building reusable robot manipulation skills in a talk hosted by MIT Robotics.
Fox compares language and vision models with robot learning, arguing that their training resembles behavior cloning from large collections of meaningful examples. He expects these models to improve reasoning and perception, while physical reasoning and control still require robotics expertise and robust low-level skills.
Fox describes an object-centric pipeline that segments unfamiliar objects, generates candidate grasps, and checks for collisions. Training uses simulated scenes and thousands of object shapes; he reports that the grasping system transfers to real-world trials without additional real-world fine-tuning. When an object blocks the target, the robot can move the obstacle and try again.
Fox contrasts object-based methods with CLIPort and Perceiver-Actor, which use language instructions and demonstrations to predict gripper actions. Perceiver-Actor represents a 3D scene with voxels and learns from 53 real-world demonstrations; a later multi-view model reduced reported training time from 16 days to 3.3 days.
Fox wants broad skills such as open, close, pick up, place, and insert to work across objects, environments, and research labs. His proposed interface starts with a skill word and a pixel indicating the target. The team plans to generate cabinet scenes, tasks, and demonstrations in physics simulation, then use large-scale behavior cloning and test transfer to real robots.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Dieter Fox, senior robotics research director at NVIDIA and professor at the University of Washington, reflects on robotics’ progress and the prospects for open-world robots, with the IEEE Robotics and Automation Society.
Dieter Fox, professor at the University of Washington and robotics research lead at NVIDIA, discusses where robotics stands relative to generative AI and how simulation could help train more capable robots, during a CMU Robotics Institute seminar.
Caelan Garrett, Ankit Goyal, and Iretiayo Akinola, research scientists at NVIDIA Research, present robotics work at the University of Washington Robotics Colloquium hosted by Dieter Fox.
Dieter Fox, a University of Washington professor who also leads robotics research at NVIDIA, presents joint work from those teams at GRASP Lab on robot manipulation, with host Dan.
University of Washington professor and NVIDIA Senior Director of Robotics Research Dieter Fox presents his lab’s work on robot manipulation at the University of Toronto Robotics Institute seminar.
