Xiaolong Wang, robotics and vision researcher at the University of California San Diego, presents his group’s work on generalization for robotics at the Fall 2023 GRASP SFI.
Wang argues that data driven learning has made progress in recognition and language, but robot learning still often centers on simple pick and place with parallel grippers. More demanding skills, such as using tools, reorienting objects in the hand, and throwing and catching, require 3D geometry and contact reasoning. His goal is to train strategies that work across different objects.
Wang describes a simulated training setup for an Allegro Hand fitted with 16 inexpensive force-sensitive resistors. The policy uses binary touch readings, recent actions, and joint information to rotate objects; training across multiple shapes makes touch more useful than training on a single object. In real-world demonstrations, the approach handles an unseen prop dock and rotates a real tomato without squeezing it too hard.
Wang extends sim-to-real learning to two-hand throwing and catching, using soft objects such as sandbags because rigid objects and hands are difficult to match in these motions. He also describes robot manipulation guided by language and 3D scene representations that combine rendered features from stable diffusion with scene geometry. Demonstrations include turning a faucet, opening a microwave, and moving a teapot, though he says the grasping component is limited and staged.
For mobile manipulation, Wang describes a generalizable NeRF representation that updates as a robot navigates, helping it map obstacles and locate objects such as a duck. He also discusses feature splatting, which uses learned features to localize objects in 3D for a physics simulator, and a method that progressively builds a Gaussian-splatting scene while estimating camera poses without COLMAP. He notes limitations, including failures when camera tracking breaks and artifacts in extracted meshes.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Xiaolong Wang, assistant professor in electrical and computer engineering at UC San Diego and visiting professor at NVIDIA, discusses data-driven robotics research at the 2025 Winter Robotics Colloquium hosted by the Paul G. Allen School.
