Ken Goldberg - Is Data All You Need Large Robot Action Models and Good Old Fashioned Engineering

In this talk

1. 0:37

   Art, work, and manipulation roots

   Goldberg connects robotics with art, from the 1995 Telegarden to Breathless, an eight-hour dance performance with robotics researcher and dancer Katie Quan about work and automation. He also recalls early research proving that a parallel-jaw gripper could orient polygonal objects, later extended to algebraic parts.

2. 3:46

   Why data alone may fall short

   Goldberg questions whether internet-scale data will solve robotics as it helped vision and language. He points to robotics’ higher-dimensional tasks and limited training data, then compares self-driving companies: Tesla has an estimated 500 times more data than Waymo, while Goldberg says Waymo performs much better. He attributes Waymo’s results partly to modular systems, algorithms, and metrics.

3. 8:33

   Dex-Net and lessons from deployment

   Goldberg describes Dex-Net, which used simulation and perturbation analysis to generate millions of grasp examples. It cleared bins of unfamiliar objects with success rates above 90 percent, and its technology became part of AmbiSort. The company’s 60 machines sort a million packages each week and record picks for monitoring and algorithm tests.

4. 11:28

   Adding engineering structure to learned models

   Goldberg presents the Manipulation Modular Architecture, developed with Toyota Research Institute colleagues, and a bagging task that is difficult for robots. A learned system using ultraviolet paint and cameras achieved about 50 percent success; adding SLIP, an interactive perception module to distinguish bag layers, improved reliability and generalization to other bags. Goldberg closes by urging measured expectations while recognizing that new model structures may bring further breakthroughs.