Wei of Tsinghua University, Ren of Renmin University of China, Wen, Jad, Priya, Joey of Remix, and UC Berkeley professor Jitendra Malik present CoRL 2024 research on robot learning, manipulation, and perception during the Conference on Robot Learning oral session.
Wei’s method uses foundation models to provide policy, value, and success priors for reinforcement learning. In a button-pressing simulation, foundation-guided actor-critic learned in about 100,000 frames, while vanilla RL remained stuck; on five real-robot tasks, it reached 86% success after an hour, compared with 22% for the prior policy. Wei also described using vision-language or diffusion-based models to guide real-robot tasks such as watering plants, opening a door, and playing golf.
Ren introduced MSPOT, which predicts task stages and uses them to dynamically fuse visual, audio, and tactile information. The approach was evaluated on pouring beads to a target mass and inserting a pack into a keyway, where it outperformed simple concatenation, LSTM, and self-attention baselines. Ren said language models can help label stages from task descriptions, though the audience asked how stage definitions and discrete phases should be chosen.
Wen’s TIEBot estimates a sequence of cloth meshes from one human RGB-D demonstration, uses physics simulation to preserve knot topology, and trains a policy in simulation before deploying it on a dual-arm robot. Its simulation policy achieved 60% success; real-robot trials tied a knot, though the pipeline still needs human tuning and an initial-state mesh. Jad presented embodied Gaussians, which connect visual 3D Gaussians to physics particles so a system can track scenes at 30 Hz using prediction and visual correction; initialization assumes a controlled scene and objects visible from several viewpoints.
Priya’s RT Sketch turns hand-drawn scene sketches into spatial goals for a robot policy. A model trained with 500 paired image-sketch examples generated sketches for a dataset of 80,000 trajectories; the method handled varied sketches and clutter better than a goal-image baseline in the reported tasks. Joey’s REMIX method reweights robot datasets by estimating each domain’s remaining learning potential. Its recipe includes action normalization, discrete behavior-cloning losses, and early stopping; the resulting mixture improved reported generalization performance by 32% over the comparison mixture.
UC Berkeley professor Jitendra Malik connected robot learning to child development and argued that robotics should join perception with physical action. He reviewed work on locomotion, navigation, and manipulation, highlighting online adaptation from robot movement, combined vision and touch, and simulation-to-real learning. Malik proposed treating common manual actions as a small repertoire of composable skills, learned through reinforcement learning, teleoperation, and visual imitation. In discussion, he said automatic skill hierarchies remain an open challenge and that dynamics models can support control.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Jitendra Malik, professor of EECS at UC Berkeley and vice president and distinguished scientist at Amazon, discusses robot learning and dexterous manipulation at a CMU Robotics Institute seminar introduced by Shubham.
Jitendra Malik, professor at UC Berkeley and researcher at Meta, discusses robot learning inspired by child development at a Paul G. Allen School talk hosted by Steven Seitz.
Jitendra Malik, Berkeley professor and part-time research science director at Meta, talks with Peter on The Robot Brains Podcast about sensorimotor learning, robotics, computer vision, and how AI research develops.
