Ep#56: GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulation

In this talk

1. 0:41

   Building a photo-realistic simulator

   GSWorld combines 3D Gaussian splatting with the ManiSkill physics simulator to create digital twins of real robot scenes. The team uses COLMAP and ArUco tags to build a metric reconstruction, then aligns it with the simulated robot using point registration. The result supports policy training, benchmarking, data collection, and virtual teleoperation.

2. 5:32

   Zero-shot transfer and learning from failures

   The team trained Action Chunking Transformer policies on simulated data and deployed them in the real world, where all four reported tasks succeeded. For closed-loop learning, they collected 100 demonstrations per iteration, recorded policy failures, and used motion planning to gather corrective examples from those states. Across five iterations and up to 500 demonstrations, this approach outperformed training from scratch on the Franka tasks.

3. 17:52

   Evaluating policies in simulation

   Across four Franka tasks, the team reports that simulation success rates correlate with real-world results, making GSWorld a cheaper way to compare policies. The host notes that simulation appears to underestimate π₀ and overestimate ACT; Guantani attributes some of that difference to where each policy was trained. Simulation also lets researchers run more evaluation trials without repeatedly using real hardware.

4. 24:06

   Limits in reconstruction and physics

   The pipeline needs textured scenes for COLMAP feature matching, and its collision meshes can be larger than the rendered meshes, making grippers appear to miss objects they touch in simulation. The team treats most of the scene as a static background, so objects such as chairs do not move. Guantani says object mass, friction, and gripper force limits remain difficult sim-to-real gaps, and reconstruction still requires substantial labeling.

5. 35:18

   Reinforcement learning and next steps

   With GSWorld’s photo-realistic rendering, a SAC policy trained in simulation transferred to the real world without extensive domain randomization. Training took roughly 10 to 20 hours on one RTX 4090, using mostly color changes and a third-person camera to limit visual mismatch. Future directions include dexterous hands, improving simulated physical properties, and reconstructing deformable objects.