Rong Xiong -- Integration of Robotics and AI: Changes and Challenges

In this talk

1. 0:07

   From explicit models to learned systems

   Xiong describes building humanoid robots that played table tennis, using engineered perception and modeled estimation, planning, balance, and control. That approach supported mobile robots, quadrupeds, and manipulation systems, but depended on expert modeling and tuning and struggled with complex objects and unfamiliar situations.

2. 4:37

   AI helps, but robot skill data is scarce

   Deep learning and reinforcement learning shifted robotics toward models learned from neural networks and data. Xiong gives examples in mobile robotics, cluttered-scene manipulation, and legged locomotion, while noting that learned methods still require expert-designed training pipelines. Robot control is especially hard to train because physical interaction data is costly, multimodal, and difficult to simulate reliably.

3. 9:35

   Mechanism-based methods can supervise learning

   Xiong’s group combines established controllers with learned policies to reduce the burden of collecting and labeling robot data. For bronchoscopy, a simulated airway and an artificial expert generate training examples; the resulting controller turns a doctor’s discrete commands and camera images into steering actions. In reported experiments, students using the system approached or exceeded expert doctors’ accuracy.

4. 15:43

   Structural priors improve manipulation control

   For visual servoing, the group used ground-truth pose information and a controller architecture informed by traditional methods. A hypernetwork helped adapt to different target poses, cutting manual image annotation from thousands of examples to 10 to 30; calibration embeddings supported operation with unknown camera parameters. For peg-in-hole tasks, mapping force signals to a localization problem helped a controller trained on square holes generalize to several unseen socket shapes.

5. 20:56

   New mechanisms address impact and localization

   Xiong describes a nonlinear controller inspired by shear-thickening fluids that aims to let collaborative robots yield to human traction while resisting impacts. The group also developed invariant representations for global localization, reporting improved success and accuracy on vision and LiDAR data when combined with learning. He concludes that integrating mechanisms with data-driven methods can improve robustness while reducing data and annotation demands.