RoboBoom
VinMotion· May 25, 2026· 1:26:13

VINMOTION INTELLIGENCE PIONEERS SEMINAR #2 | Synthetic Data for Robot Learning - Dr. Fan Shi

Dr. Fan Shi, an assistant professor at NUS, discusses synthetic data for robot learning at VinMotion’s seminar with Professor Quan and the VinMotion team.

In this talk

  1. 3:54

    Why robot learning needs scalable data

    Shi frames dexterous manipulation, reliable deployment, and simulation-based co-design as closely connected challenges. Real robot data is costly and sparse, especially for dexterous hands, while decentralized collection makes it hard to capture failures and recovery.

  2. 11:33

    Fast, accurate simulation for manipulation

    Contact-rich manipulation needs more accurate physics than locomotion because small contact errors can change the result. Shi describes a GPU-parallel method combining local-global deformation solving with a nonsmooth Newton approach for contact. On an RTX 5090, it can run over 300 environments; a towel-folding policy trained in about an hour ran continuously for more than an hour in a live demonstration.

  3. 25:13

    Differentiable simulation and real-to-sim

    Shi explains how analytic and neural differentiable simulators can provide gradients for policy learning, reducing the need for large numbers of trial rollouts. Examples include depth-based obstacle avoidance, deformable-object control, and calibrating contact parameters from a few real-world trajectories. In one graph-network simulator study, three real trajectories supported calibration before scaling to 3,000 simulated contact-rich trajectories.

  4. 37:43

    Finding controller weaknesses and improving safety

    Shi presents learned adversarial tests that expose sequential failure cases in locomotion controllers. A computational attack brought down a simple controller in 1.4 seconds, compared with four seconds for the best of 100 ETH experts, and the method also revealed risks in a robust controller under small IMU errors. He discusses using discovered failures to refine controllers, while noting that overly aggressive fixes can create new weaknesses.

  5. 50:01

    Hardware protection and practical guidance

    Shi describes a soft protective material co-designed through simulation and fitted to a humanoid robot, which remained safe in high-impact falls, including a second-floor drop. In the discussion, he recommends accurate digital assets for contact tasks, automatic parameter calibration such as CMA-ES, and balancing simulation accuracy with domain randomization to avoid overly conservative policies.

Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.