A Data-Driven Path to Real2Sim -- Ruining Li

In this talk

1. 1:17

   Training 3D generators for physical stability

   Li says visually convincing 3D generators can produce objects that collapse under gravity, including when prompted with images of objects from their training sets. His team’s Direct Simulation Optimization uses simulator preferences to fine-tune a generator, making stable outputs more likely without differentiable simulation. The approach improved results on difficult examples such as alarm clocks and motorcycles, generalized from MuJoCo to Isaac Gym, and produced a stable 3D-printed object.

2. 10:46

   Scaling articulated object estimation

   For robotics evaluation and simulation training, Li argues that interactive objects with articulated parts are needed alongside static assets. His Particulate model estimates part segmentation and joint properties from a mesh, so it can complement image-to-3D systems. Early models trained on about 4,000 objects struggled with unfamiliar structures, leading the team to expand its training data.

3. 14:10

   Using prompts and synthetic data for articulation

   The team labeled about 20,000 synthetic meshes by rendering multiple views, prompting a vision-language model for part and kinematic structure, and mapping generated 2D masks back onto the 3D meshes. Because annotations differed even for similar objects, they added kinematic prompts that specify the desired structure, including joint types. Li says the larger dataset improved performance on varied categories and small components such as oven knobs.

4. 22:12

   An agent that designs simulation-ready assets

   Li describes an agent that writes code to construct articulated 3D objects, checks its own designs with tests, and revises them using compilation feedback and tools for inspecting points in 3D. The team generated 10,000 diverse objects, many of which worked in Isaac Sim; primitive geometry kept models light enough for simulation. The assets are open source and were also used to train the articulation model.

5. 30:41

   Synthetic experience for long-horizon robotics

   Li identifies latency and cost as barriers to using frontier models for dexterous real-world tasks: a pen-picking example took 20 minutes and cost $13, compared with roughly one second for a human. He proposes an efficient System 2 model to manage memory and long tasks, directing faster System 1 policies. Agents that generate simulated interactions could provide training examples for this system, including tasks with sustained contact that are hard to collect from video.