7 Gerhard Neumann, Karlsruhe Institute of Technology - BMVA: Robotics Foundation & World Models
Gerhard Neumann, professor at Karlsruhe Institute of Technology, explains how to improve world models for robot control, presenting three research directions at the BMVA event.
In this talk
- 2:53
Why world models mishandle uncertainty
Neumann examines how recurrent state-space models handle aleatoric uncertainty from noisy environments and epistemic uncertainty from limited data. He argues that the usual inference assumptions can mistake a surprising observation for a sudden change in the system, like believing a hamster turned into a cat rather than correcting the earlier belief. His results suggest that modeling both uncertainty types matters when observations are incomplete or transitions are noisy.
- 14:40
Use different learning objectives for different sensors
Robots combine high-dimensional camera images, which may contain distracting backgrounds, with compact signals such as proprioception. Neumann’s CORAL method applies contrastive learning to images and reconstruction to proprioception within one state model. In simulated tasks, this combination reached up to 70% success, compared with roughly 40% for approaches using a single loss or simple concatenation.
- 21:31
Build predictions across multiple time scales
Neumann describes multi-time-scale state-space models for extending predictions beyond the next step. A slower, abstract state summarizes activity over a window and influences the faster model’s dynamics; randomized imputation helps train predictions across missing inputs. On examples including excavator dynamics, varied terrain, and Franka Kitchen, the approach produced stronger long-horizon predictions than LSTMs and Transformers.
- 29:23
Practical lessons for model-based control
Neumann summarizes three design lessons: use sound inference when uncertainty is high, tailor representation learning to each sensor, and model dynamics at multiple time scales for long-horizon prediction. He notes that uncertainty modeling may have little effect on standard benchmarks with low aleatoric uncertainty, while partially observed or noisy tasks expose larger differences.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.