Spring 2026 GRASP SFI - Stone Tao, University of California, San Diego

In this talk

1. 0:01

   Why use simulation for robot learning

   Tao says robotics research needs more data, evaluations, and scalable infrastructure, while simulation can help with tasks where real-world data is difficult to collect. It can also provide training data for other models and support compute-scalable evaluation. He focuses on sim-to-real reinforcement learning.

2. 4:45

   Simulating cables and simplifying training

   For cable routing with Vayu, Tao's team models cables as chains of capsule links with joints, using simple elasticity and plasticity rules in ManiSkill. The goal is broad coverage of cable shapes and behaviors rather than exact physical fidelity. They use motion-planning primitives to reset training near the routing task, shortening the learning horizon and keeping the reward function simple.

3. 11:16

   Putting simulation in the deployment loop

   Tao describes Silo, or Simulation in the Loop: a synchronized digital twin feeds observations to the policy, then the simulator applies each policy action and the real robot tracks its joint angles. Modeling the wire harness in simulation provides collision handling that helps keep the real robot from striking it. The approach assumes a perception pipeline, a partial digital twin, and a mostly static task setup.

4. 17:34

   Designing simulation tools for faster research

   Tao discusses design choices in MuJoCo 3 and ManiSkill, including object-oriented access to simulation data, merging heterogeneous environments, and reducing tensor indexing and memory management. The team also tunes rendering and training tools for visual reinforcement learning, provides minimal single-file implementations, and uses per-environment video views to make evaluation and debugging easier.

5. 31:42

   Tradeoffs in fidelity and evaluation

   In the Q&A, Tao explains that cable perception uses stereo, segmentation, and point-cloud processing to produce 3D points, rather than feeding raw images to the policy. He argues that state coverage can matter more than exact transition dynamics for reactive tasks like cable routing, while tasks such as throwing may need more accurate dynamics. He also says evaluation systems should be designed to answer specific questions about model capabilities.