Dieter Fox, a University of Washington professor who also leads robotics research at NVIDIA, presents joint work from those teams at GRASP Lab on robot manipulation, with host Dan.
Fox contrasts explicit object and environment models with approaches that learn from sensor data. With models, a Franka robot can put a sugar box in a cabinet, account for the door’s motion, and plan a multi-step search through drawers. These systems support detailed planning, but inaccurate pose estimates can make them fail.
Fox describes training grasp networks on point clouds, using simulated grasps across nearly 9,000 objects to create examples of stable holds. In clutter, a learned collision checker can identify which item blocks a requested grasp, so the robot can move that item first. The same approach can place objects relative to one another and is being explored for handovers.
Fox presents early work on SORNet, which uses a transformer to form separate embeddings for objects from images. Training relies on logical questions such as whether one block rests on another, rather than object masks or 6D pose labels. Although the embeddings were trained on those predicates, additional readout networks found spatial information in them and used it to guide a gripper.
CLIPort combines CLIP’s image and language features with Transporter Networks for precise pick-and-place actions. One network learned a range of tasks from 179 demonstrations, including folding cloth, stacking blocks, and closing a rope loop. Fox reports transfer across colors and object descriptions, while noting that instructions remain scripted and generalization is limited.
In the discussion, Fox says CLIPort uses word meaning, including color and object names, but handles verbs and temporal meaning less well. He argues that robotics researchers still need foundations in estimation, control, and physics, even when developing deep learning methods. He also points to open challenges in training useful representations and extending current systems beyond top-down manipulation.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Dieter Fox, professor at the University of Washington and robotics research lead at NVIDIA, discusses where robotics stands relative to generative AI and how simulation could help train more capable robots, during a CMU Robotics Institute seminar.
Caelan Garrett, Ankit Goyal, and Iretiayo Akinola, research scientists at NVIDIA Research, present robotics work at the University of Washington Robotics Colloquium hosted by Dieter Fox.
Dieter Fox, senior robotics research director at NVIDIA and professor at the University of Washington, reflects on robotics’ progress and the prospects for open-world robots, with the IEEE Robotics and Automation Society.
Dieter Fox, senior research director at NVIDIA and professor at the University of Washington, discusses building reusable robot manipulation skills in a talk hosted by MIT Robotics.
University of Washington professor and NVIDIA Senior Director of Robotics Research Dieter Fox presents his lab’s work on robot manipulation at the University of Toronto Robotics Institute seminar.
NVIDIA founder and CEO Jensen Huang presents the company’s AI strategy and product announcements at GTC Berlin, with European partners and customers.
