Eric Heiden on Reducing the Reality Gap in Robotics via Inference | Toronto AIR Seminar

In this talk

1. 0:54

   Why simulator accuracy matters

   Heiden describes simulators as physics-based models that support safe, low-cost prototyping, but says their incomplete models often fail to match real behavior. Rather than focusing only on making control policies robust through methods like domain randomization, he aims to make the simulators themselves more accurate.

2. 4:20

   Inferring uncertain simulation parameters

   The inverse simulation problem is to find parameters that make simulated trajectories match real observations. Because noisy measurements, chaotic dynamics, and multiple plausible parameter settings make a single best guess unreliable, Heiden estimates distributions with Stein Variational Gradient Descent. His constrained method uses parallel differentiable simulation and multiple shooting, which divides long trajectories into shorter windows to ease optimization.

3. 21:43

   Testing constrained inference on real systems

   Heiden compares the method with alternatives on an 11-parameter real double pendulum and a Panda robot shaking a weighted box. He reports closer trajectory fits and parameter estimates than the tested MCMC, cross-entropy, and gradient-based methods. In the Q&A, he notes that longer MCMC runs could improve results, but the experiments limited iterations for comparison.

4. 27:12

   Building an interactive simulator from video

   The second project aims to infer a simulator from RGB or depth video, without requiring a predefined URDF or motion-capture markers. The pipeline detects and tracks rigid bodies, infers their joints using relative motion and a minimum spanning tree, then estimates simulation parameters. He demonstrates the approach on a cart-pole, a coupled pendulum from video, and wooden construction pieces; the resulting model can support tasks such as model predictive control.