Closing the Physical AI Data Gap With Simulation and Foundation Models

In this talk

1. 0:03

   Reasoning for autonomous vehicles

   NVIDIA Alpamayo enables vehicles to explain actions such as changing lanes and steering around a car blocking their path. The presenter also demonstrates following a spoken request to accelerate.

2. 0:46

   Simulation fills gaps in real-world data

   Robots must handle varied, unpredictable conditions and rare edge cases, which real-world examples alone cannot cover. The presenter argues that simulated data is essential for training physical AI.

3. 1:13

   A pipeline for training robot models

   Developers pre-train world models on large collections of video and human demonstrations, then assess them before further training. NVIDIA's tools include Isaac Lab for simulated training and assessment, Newton for GPU-based physics simulation, Cosmos for model-generated simulation, and GR00T for robot reasoning and action generation.

4. 2:06

   Examples across robotics and research

   Periton AI combines Isaac Lab and Cosmos to expand training data for an operating room assistant. Skild AI generates additional training data and uses reinforcement learning across many variations; Humanoid and Hexagon Robotics also train policies with Isaac Lab. Foxconn and Noble Machines fine-tune GR00T models, while Disney Research uses Newton and Isaac Lab with its Camino simulator to train character robots.