Jang Ren, a Peking University pre-PhD researcher collaborating with Galbot, Tsinghua University, SG2, Calia, and NVIDIA, discusses LDA-1B and its approach to training robot action models from mixed embodied data with RoboPapers hosts Chris and Michael.
Ren presents LDA as a way to train one model on expert robot demonstrations, imperfect robot trajectories, and human video. Expert data supports policy learning and dynamics prediction; imperfect trajectories contribute dynamics information without teaching the policy to imitate their mistakes. Human video supports visual forecasting despite lacking robot action labels.
The model uses task embeddings and a shared diffusion transformer with action and video streams. Video is represented in DINO feature space, which Ren says emphasizes structure and depth over appearance details. The team assembled more than 30,000 hours of data, roughly split between robot and human sources, and reformatted it into a shared action representation across embodiments.
Ren reports that performance improved as training data grew from about 5,000 to 30,000 hours, while action prediction error fell. The full model could use actionless human video, unlike policy-only baselines; the team also found larger models performed better and DINO latents helped action learning compared with pixel representations. Real-world evaluations covered contact-rich, drawer, and long-horizon tasks, with cross-embodiment adaptation using one hour of task-specific post-training data.
Ren says human egocentric video is cheaper to expand, but robot teleoperation data remains valuable and should also grow. One-hour task-specific post-training reportedly reached about 90% success on a demonstrated task, while Ren distinguishes this practical fine-tuning goal from academic work on zero-shot generalization. Future directions include larger datasets, exploring visual representations, and using limited tactile data.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Arhan, a PhD student at the University of Washington, and Carl, a robotics researcher at Physical Intelligence, discuss PolaRiS and how it uses real-to-simulation methods to evaluate robot policies with RoboPapers’ Chris and other guests.
Jason Ma, a robotics researcher at the University of Pennsylvania who worked on the paper at Google DeepMind, joins RoboPapers hosts Michael and Chris to discuss using vision-language models to assess robot task progress and data quality.
