Physical AI for the Real World: A Vision From NVIDIA Robotics Research

In this talk

1. 0:20

   NVIDIA’s three-computer robotics strategy

   Yash describes DGX for training foundation models, Omniverse and Cosmos for simulation and evaluation, and AGX for running models on robots. He presents Isaac Lab with Newton as suited to known physics, and Cosmos as useful when the world’s physics are less known.

2. 4:18

   Planning and learning for manipulation

   Yash contrasts planning from a world model with learning policies from data. Schedule Stream jointly plans actions and motions for robot arms, reaching a reported 99% average success rate on diverse problems in under two seconds. Dexter RGB and Dexter E2E use simulation training and distillation to enable stereo RGB grasping in the real world, with a reported 93% success rate on sorting and bin placement.

3. 10:22

   Simulation for contact and robot dynamics

   Yash explains classical simulation for contact-rich tasks, including Factory’s GPU-based methods and simulated tactile sensing. He reports contact simulation speedups of up to 20,000 times. Neural Robot Dynamics learns a robot’s dynamics from simulated actions and states, then serves as a fast, differentiable surrogate simulator that can be fine-tuned on real-world data.

4. 14:59

   Robots assembling compute, and a path to robot learning

   Yash describes work on assembling GB300 systems: motion primitives for rigid parts, sim-to-real transfer for cable routing, and simulation pretraining plus real-world residual reinforcement learning for cable insertion. Jim then frames robotics progress around world models, action fine-tuning, and physical reinforcement learning, describing Cosmos, DreamZero, GR00T N2, and DreamDojo. He also reports a dexterity scaling relationship from training on human egocentric video.

5. 29:33

   Humanoid control and next research steps

   Yu Khor presents Sonic, a humanoid motion-generation policy trained on 700 hours of motion-capture data and designed to run onboard. The team trained it across thousands of simulated environments and demonstrated gamepad control on a Unitree G1 at 50 hertz on Jetson Orin. Yu Khor says the next step is to add scene awareness for navigation, obstacle avoidance, and manipulation.