Precision Home Robotics w/Real-to-Sim-to-Real

In this talk

1. 0:15

   Building a digital kitchen

   The goal is to send robots into homes to take on chores such as laundry, dishwashing, and cleaning. A person can scan a kitchen with an iPhone and upload it so researchers can create a digital version for robot training.

2. 1:01

   Practicing safely in simulation

   In the digital kitchen, a robot can repeat tasks at very large scale without damaging real objects. The presenter says this practice can produce a robust policy that adjusts when shelves move or objects change position.

3. 1:34

   Scanning objects and learning the task

   The team scans scene objects such as a mug and shelf, then imports them into the simulator for faster practice. Reinforcement learning rewards the robot for placing the mug on the shelf; camera images are converted into point clouds to represent the scene in 3D.

4. 2:30

   Testing in the real world

   The robot reaches for a cup while Marcel moves it, and completes the task despite not having trained in the physical environment. The presenter notes that liquids and deformable objects remain difficult to simulate.

5. 2:52

   Scaling simulated data collection

   The demonstration points toward gathering large quantities of training data in simulation. The team says further updates are expected.