Jitendra Malik, Berkeley professor and part-time research science director at Meta, talks with Peter on The Robot Brains Podcast about sensorimotor learning, robotics, computer vision, and how AI research develops.
Malik argues that seeing, moving, and manipulating objects are foundational forms of intelligence that evolved before language and abstract reasoning. He connects this to Moravec’s paradox and says AI will remain incomplete until it can handle basic sensorimotor tasks as well as language and games.
Malik describes how children first connect words such as “ball” and “give” to objects and actions, before learning more abstract concepts from reading and context. He favors building physical intelligence before language, while acknowledging that the order is an open research question and multimodal models may combine both.
Malik frames his robotics work as learning sensorimotor skills, from walking to catching and slicing, so that later skills can be acquired faster. His group’s Rapid Motor Adaptation approach uses proprioceptive feedback to infer terrain and adjust a robot dog’s gait within about a second. The same learned policy handled rocks, stairs, leaves, and planks; vision helped with harder tasks such as climbing stairs.
Malik contrasts learned control with hand-designed controllers that depend on models and parameters that can change across terrain. He sees simulation as a way to train and identify systems, then improve the simulator based on failures in the real world. His group uses both real-world training and simulation, and he expects both approaches to matter.
Malik says recognition and image segmentation have advanced substantially through larger datasets and benchmarks, while single-image 3D reconstruction and vision for action remain open challenges. He recalls how benchmark discussions helped prompt the ImageNet work that demonstrated deep learning’s strength on natural images, alongside GPUs and large-scale data. He describes academia and industry as complementary, supporting both small exploratory projects and large team efforts.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Jitendra Malik, professor of EECS at UC Berkeley and vice president and distinguished scientist at Amazon, discusses robot learning and dexterous manipulation at a CMU Robotics Institute seminar introduced by Shubham.
Wei of Tsinghua University, Ren of Renmin University of China, Wen, Jad, Priya, Joey of Remix, and UC Berkeley professor Jitendra Malik present CoRL 2024 research on robot learning, manipulation, and perception during the Conference on Robot Learning oral session.
Jitendra Malik, professor at UC Berkeley and researcher at Meta, discusses robot learning inspired by child development at a Paul G. Allen School talk hosted by Steven Seitz.
