S3 E12 Rocky Duan, from OpenAI to Covariant: Building Foundation Models for Robotic Automation

In this talk

1. 2:47

   Why warehouse picking is difficult

   Duan says Covariant builds AI-powered robotic systems for warehouse and logistics tasks such as picking, scanning, and placing goods. These tasks are hard to automate because objects arrive in unstructured layouts and vary too widely for robots to follow a fixed script. Covariant trains systems from data and robot experience so they can generalize to unfamiliar situations.

2. 8:11

   How Covariant systems work

   Covariant supplies camera systems and software that interpret warehouse scenes and direct robotic arms, sometimes alongside automation partners. Duan describes a robotic put-wall system that sorts picked items into 20 to 80 orders, as well as goods-to-robot stations where arms pick from totes and place items into compartments. He explains that deep learning represents data with flexible neural networks, while reinforcement learning lets robots improve through trial and error, such as learning from missed picks or damaged goods.

3. 22:09

   Foundation models and reliable adaptation

   Duan sees foundation models as a way to apply knowledge learned from broad data to many robotic tasks. Internet-scale images, videos, and text may help robots handle rare cases, such as deciding what to do with a box whose lid is open. He also highlights system reliability, uncertainty detection, and fast adaptation, including a sale that introduces bottles with surfaces a robot's gripper cannot handle.

4. 39:22

   From research to real-world robotics

   Duan says Covariant could expand automation upstream and downstream of its current systems, including packing orders after items reach a put wall. He is also interested in robotic science labs, where automation could take on precise, repetitive, or hazardous experiments and give researchers more time to design studies and analyze results. He says he left OpenAI because commercial deployments could provide relevant tasks and data at scale, creating a cycle in which deployed systems generate value and improve the learning system.

5. 50:13

   Meta-learning, productivity, and personal routines

   Duan traces his interest in meta-learning to the limits of reinforcement learning that depended on extensive exploration and transferred little between tasks. He describes meta-learning as using data to improve the learning process itself, and connects that idea to language models that adapt to tasks from prompts and examples. For productivity, he recommends learning tools from productive colleagues, building useful software, and regularly checking whether daily work serves the larger goal; he also mentions hiking, piano, and time with his rabbit, Puffin.