Dieter Fox, professor at the University of Washington and robotics research lead at NVIDIA, discusses where robotics stands relative to generative AI and how simulation could help train more capable robots, during a CMU Robotics Institute seminar.
Fox says language and vision models benefit from vast datasets and scalable architectures, while robot manipulation lacks comparable action-labeled data. He proposes combining large-scale demonstrations with generative models and behavior cloning to improve manipulation, while stressing that this would not solve robotics on its own.
Fox describes using simulation to create labeled grasp data, train object segmentation, and develop manipulation policies that transfer to real robots. Examples include generating grasps for nearly 9,000 ShapeNet objects and training pose estimation on tens of thousands of simulated objects. He also covers contact-rich industrial tasks, where GPU simulation enabled 1,000 parallel runs in real time.
Fox outlines a pipeline for generating varied assets and scenes, creating tasks, solving them with planning or reinforcement learning, and rendering demonstrations for behavior cloning. He discusses procedural kitchens, AI-generated assets, language models for suggesting tasks, and task-and-motion planning for producing demonstrations at scale.
Fox contrasts systems where a vision-language model calls a fixed set of robot skills with end-to-end models that map images and language to robot controls. He suggests an intermediate approach that aligns language models with learned robot policies. He concludes that simulation data should be combined with real demonstrations and that real-world fine-tuning and reinforcement learning will still be needed.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Caelan Garrett, Ankit Goyal, and Iretiayo Akinola, research scientists at NVIDIA Research, present robotics work at the University of Washington Robotics Colloquium hosted by Dieter Fox.
Dieter Fox, a University of Washington professor who also leads robotics research at NVIDIA, presents joint work from those teams at GRASP Lab on robot manipulation, with host Dan.
Dieter Fox, senior robotics research director at NVIDIA and professor at the University of Washington, reflects on robotics’ progress and the prospects for open-world robots, with the IEEE Robotics and Automation Society.
Dieter Fox, senior research director at NVIDIA and professor at the University of Washington, discusses building reusable robot manipulation skills in a talk hosted by MIT Robotics.
University of Washington professor and NVIDIA Senior Director of Robotics Research Dieter Fox presents his lab’s work on robot manipulation at the University of Toronto Robotics Institute seminar.
NVIDIA founder and CEO Jensen Huang presents the company’s AI strategy and product announcements at GTC Berlin, with European partners and customers.
