Ep#102: EgoSuite-Open100K from Lightwheel

In this talk

1. 1:09

   Releasing and checking EgoSuite-Open100K

   Lightwheel is releasing 100,000 hours of egocentric data in stages, with more than 10,000 hours already available across over 15,000 scenes and tasks. Sier describes collection guidance, near-real-time AI checks, agent-driven QC, and two rounds of human review to catch issues such as unnatural movement, hands leaving the frame, and errors in timing or 3D understanding.

2. 8:18

   Building diversity and useful annotations

   Sier identifies video quality, hand pose, language annotations, and full-body pose as key parts of the data pipeline. Lightwheel collects in real homes, hospitality sites, shops, gyms, logistics locations, offices, and industrial settings, and uses an internal task agent to manage tasks and assignments. He says precise hand annotations and atomic action-level language labels remain challenging, so the team combines automated methods with human review.

3. 16:54

   Connecting human data with simulation

   Sier says Lightwheel began with simulation, then added egocentric data as a scalable, robot-agnostic source for training and for building simulations grounded in real-world tasks. He sees human data as useful for model training and simulation as a scalable way to evaluate models. Lightwheel’s RoboFinals benchmark turns real-world task descriptions into simulation exams, with sim-to-real checks on robot fleets beginning with tabletop tasks.

4. 37:16

   Scaling simulation assets and physics

   Lightwheel maintains separate pipelines for customer-specific digital twins and more general simulation benchmarks derived from egocentric data. Its SimReadyGen tool can generate simulation-ready scenes from an image or text prompt for Isaac Sim and MuJoCo, supporting USD and MJCF. Sier says converting video into accurate geometry and physics remains difficult, and describes collecting physical force data, including measurements for opening microwave doors and handling cables.

5. 50:41

   Planning for larger-scale data collection

   Sier says Lightwheel aims to support multiple collection devices, from phones to headsets, with raw, standardized, and Pro data tiers. He argues that the 10-billion-hour target depends on scaling the full data engine, including edge processing and eventually shadow mode to identify robot failures and collect useful edge cases. He also describes the Home challenge as an early effort combining deformable-space simulation with sim-to-real evaluation.