Dr. Dinesh Jayaraman, a faculty member in computer and information science at the University of Pennsylvania and GRASP Lab, discusses visual robot learning and ways to make robots more capable in uncertain settings.
Jayaraman contrasts predictable factory settings with homes, roads, and workplaces, where people and objects introduce uncertainty. He argues that robots need frequent perception and action updates in these environments, and that learned controllers can help turn visual input into complex behavior.
Jayaraman describes self-supervised keypoints that let robots track objects, tools, robot bodies, and tactile observations. He then presents a hierarchical keypoint method that learns coarse and fine scene structure without manually specified body joints, reporting lower error than single-level approaches on human keypoint prediction and more stable semantic associations in video.
Jayaraman explains how goal-conditioned reinforcement learning can use previously collected data even when it was not gathered as task demonstrations. A goal image, such as a robot holding a bottle, helps select relevant parts of that data; he says the approach worked in simulation and on a real multifingered hand manipulating a screw.
Jayaraman extends active perception from inspecting the world to testing whether a task outcome earned reward. For example, a robot can pull a door handle to determine whether the hidden latch is locked, or turn a bottle upside down to check whether its cap is tight. He reports that policies trained with interactive reward checks performed better than baselines on several tasks.
Jayaraman groups the research around observation, representation, and decision-making. His closing point is that choosing useful abstractions can help robot learning use experience efficiently and reduce the computation needed during execution.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
A GRASP Lab seminar host and panelists Nadia Figueroa, Dinesh Jayaraman, and Mark Yim discuss robotics research, deployment challenges, and future opportunities at the University of Pennsylvania.
