VINMOTION INTELLIGENCE PIONEERS SEMINAR #2 | Synthetic Data for Robot Learning - Dr. Fan Shi
Dr. Fan Shi, an assistant professor at NUS, discusses synthetic data for robot learning at VinMotion’s seminar with Professor Quan and the VinMotion team.
In this talk
- 3:54
Why robot learning needs scalable data
Shi frames dexterous manipulation, reliable deployment, and simulation-based co-design as closely connected challenges. Real robot data is costly and sparse, especially for dexterous hands, while decentralized collection makes it hard to capture failures and recovery.
- 11:33
Fast, accurate simulation for manipulation
Contact-rich manipulation needs more accurate physics than locomotion because small contact errors can change the result. Shi describes a GPU-parallel method combining local-global deformation solving with a nonsmooth Newton approach for contact. On an RTX 5090, it can run over 300 environments; a towel-folding policy trained in about an hour ran continuously for more than an hour in a live demonstration.
- 25:13
Differentiable simulation and real-to-sim
Shi explains how analytic and neural differentiable simulators can provide gradients for policy learning, reducing the need for large numbers of trial rollouts. Examples include depth-based obstacle avoidance, deformable-object control, and calibrating contact parameters from a few real-world trajectories. In one graph-network simulator study, three real trajectories supported calibration before scaling to 3,000 simulated contact-rich trajectories.
- 37:43
Finding controller weaknesses and improving safety
Shi presents learned adversarial tests that expose sequential failure cases in locomotion controllers. A computational attack brought down a simple controller in 1.4 seconds, compared with four seconds for the best of 100 ETH experts, and the method also revealed risks in a robust controller under small IMU errors. He discusses using discovered failures to refine controllers, while noting that overly aggressive fixes can create new weaknesses.
- 50:01
Hardware protection and practical guidance
Shi describes a soft protective material co-designed through simulation and fitted to a humanoid robot, which remained safe in high-impact falls, including a second-floor drop. In the discussion, he recommends accurate digital assets for contact tasks, automatic parameter calibration such as CMA-ES, and balancing simulation accuracy with domain randomization to avoid overly conservative policies.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.