Jitendra Malik, professor of EECS at UC Berkeley and vice president and distinguished scientist at Amazon, discusses robot learning and dexterous manipulation at a CMU Robotics Institute seminar introduced by Shubham.
Malik frames robotics as connecting perception to action and surveys navigation and locomotion. In Project GOAT, robots learn to find requested objects in unfamiliar homes while building semantic maps; he says a modular system trained across about 200 homes generalized better than fully end-to-end policies. He also describes humanoids walking outdoors, including on San Francisco’s Bradford Street, which has a 41% slope.
Malik argues that reliable manipulation remains unsolved, despite progress in language and vision. He says robotics requires reasoning at three levels: task goals, movement through space and time, and contact forces. Language models can help with plans such as making an omelette, but they do not provide the detailed trajectories and force control needed to crack eggs or handle objects.
Malik compares teleoperation, human video, and simulation as ways to train robots. Teleoperation captures robot state but can be slow and limited to the settings where data is collected; human video offers more varied examples but lacks contact forces and requires adapting human motion to robot bodies. Simulation has helped locomotion, while manipulation is harder because its rewards are difficult to specify.
Drawing on Alan Turing and child development research, Malik proposes combining observation with practice in the robot’s own body. He describes VideoMimic, which converts human movement videos into 3D scenes, trains humanoid locomotion in simulation, and transfers the resulting policy to the real robot. He suggests a similar path for manipulation: reconstruct hand and object motion from video, then train in simulation.
Malik says manipulation datasets need 3D or 4D records of hands interacting with objects, and notes that reconstructing these trajectories from video remains a work in progress. He advocates multifingered hands and tactile sensors, reporting that object rotation performed better when vision and touch were combined than with either alone. He concludes that teams should build reliable, generalizable basic skills that can later be composed into larger tasks.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Wei of Tsinghua University, Ren of Renmin University of China, Wen, Jad, Priya, Joey of Remix, and UC Berkeley professor Jitendra Malik present CoRL 2024 research on robot learning, manipulation, and perception during the Conference on Robot Learning oral session.
Jitendra Malik, professor at UC Berkeley and researcher at Meta, discusses robot learning inspired by child development at a Paul G. Allen School talk hosted by Steven Seitz.
Jitendra Malik, Berkeley professor and part-time research science director at Meta, talks with Peter on The Robot Brains Podcast about sensorimotor learning, robotics, computer vision, and how AI research develops.
