Generalizable and Scalable Robot Learning in the Physical World - Fangchen Liu 0327

In this talk

1. 0:36

   Why physical robot learning is difficult

   Liu contrasts internet-trained digital models with robots, which have limited high-quality data, complex continuous controls and costly real-world failures. She argues that useful robot learning needs multiple data sources: demonstrations for specific behaviors, diverse sensorimotor experience for basic skills, and internet-scale data for broader task understanding.

2. 6:54

   Learning skills from exploratory data

   Liu describes a NeurIPS 2022 method that stores exploration trajectories and reuses them for downstream tasks. Its masked autoencoding model learns both forward and inverse dynamics, then supports goal reaching, multigoal planning and skill generation by changing which parts of a trajectory are hidden. The results are from simulated locomotion, where goal-reaching performance improved over an autoregressive baseline.

3. 16:00

   Combining visual language reasoning with robot control

   For open-world manipulation, Liu presents MoCA, which uses GPT-4V to select grasp, interaction and waypoint points on images, then executes them with programmed motion primitives. She notes that the system can fail because of object detection, model decisions or low-level control, and that 2D waypoints require hardcoded heights. Her later task-aware visual feature work uses intermediate CLIP representations to connect instruction words with relevant image patches, helping the policy handle new objects and tasks.

4. 36:14

   Scaling real-world demonstrations and future directions

   Liu discusses Tactile-Enriched UMI, which adds tactile images to visual input to improve dexterous interaction, including with transparent objects. She also describes VR tracking as an alternative to fragile RGB SLAM, while noting that wearable data collection still faces inverse-kinematics and collision-avoidance limits. Future directions include combining simulation, human video and other data with robust low-level controllers.