Caelan Garrett, Ankit Goyal, and Iretiayo Akinola, research scientists at NVIDIA Research, present robotics work at the University of Washington Robotics Colloquium hosted by Dieter Fox.
Caelan Garrett describes using task and motion planning to generate demonstrations for Optimus, a visual motor transformer that maps camera observations to end-effector poses at about 50 Hz. In simulation, the team generates varied tasks and trains the model on successful plans; Garrett reports that it can outperform the planner in runtime and success rate on the tested distribution. For precise actions such as insertion, a human can handle the difficult segment while planning manages the rest, and those demonstrations can later train an autonomous policy.
Ankit Goyal presents RVT, which represents a scene through images rendered from five virtual viewpoints, alongside RGB, depth, and 3D coordinate channels. A transformer predicts target positions in each view, which are projected back into 3D, reducing token growth from cubic to square in image resolution compared with voxel processing. Goyal reports faster training and inference and strong real-world results with 10 demonstrations per task; thin objects and unseen objects remain challenges.
Iretiayo Akinola explains how signed distance fields and contact clustering speed up simulation of nut-and-bolt interactions. Clustering reduces contacts from as many as 16,000 to about 300; with a Jacobi solver, the team reports increasing parallel simulations from 20 to more than 8,000 on a desktop GPU. Reinforcement learning trains separate pick, place, and screw skills, which are then chained for assembly.
Akinola describes handling real-world gaps such as noisy pose estimates, reflective parts, and differences in friction, damping, and sensor readings. The team adds observation noise during training, estimates friction and damping from free-space robot movements, and randomizes dynamics in simulation. Akinola also shows tactile simulation and a real-robot peg placement example where the policy uses tactile input to orient the peg.
In the discussion, Akinola says sensor tuning and filtering help with RealSense data, though reflective metal remains difficult to capture. Garrett says the lab is exploring ways to combine projects, including using data from his imitation-learning work to train RVT. Akinola explains that contact reduction speeds the solver, while separate training penalties address excessive interpenetration during reinforcement-learning exploration.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Dieter Fox, professor at the University of Washington and robotics research lead at NVIDIA, discusses where robotics stands relative to generative AI and how simulation could help train more capable robots, during a CMU Robotics Institute seminar.
Dieter Fox, a University of Washington professor who also leads robotics research at NVIDIA, presents joint work from those teams at GRASP Lab on robot manipulation, with host Dan.
RoboStrategy head of research Scott Walter, an aerospace engineer and robotics-company co-founder, joins host Herbert to discuss Tesla Optimus V3’s design, production readiness, and remaining engineering challenges.
Andrew King, CEO of RoboStrategy, discusses humanoid robotics and investment with John Koetsier on NEXT.
Jason Calacanis, host of This Week in Startups, and co-host Lon Harris discuss Beijing’s humanoid robot games, AI business models and labor, cyber threats, autonomous weapons, and Star Wars.
Scott Walter joins The Humanoid Hub host to analyze Tesla’s Optimus hand and production plans, Figure’s robot output, and humanoid robots’ performance in the Beijing half marathon.
