Self-Distillation Enables Continual Learning - Idan Shenfeld

In this talk

1. 2:07

   Continual learning for foundation models

   Shenfeld frames continual learning as updating a model as new tasks arrive while preserving the abilities it already has. He argues that language models face this challenge across pretraining, instruction tuning, and successive reinforcement learning stages, and may increasingly learn from user interactions as internet data becomes less abundant. He gives medical question answering and learning a new software API as examples of capabilities a model might need to acquire after pretraining.

2. 12:23

   Why on-policy learning can forget less

   On-policy methods collect examples from the student’s own behavior, then use a teacher or reward to guide it. Shenfeld revisits DAgger’s lane-driving example: a car trained only on expert trajectories may not know how to recover after drifting toward a wall, while querying an expert on the car’s own states supplies recovery examples. In experiments spanning language tasks, an environment simulator, and MNIST variants, he reports that reinforcement learning often achieved a better balance between new-task learning and retention than supervised fine-tuning.

3. 33:51

   Self-distillation turns context into training

   Shenfeld’s method uses the same model in two modes: a student answers without extra context, while a teacher sees helpful context such as a demonstration or feedback. The model then learns to match the teacher’s output distribution on the student’s own responses, transferring the context’s benefit into its parameters. Because evaluation uses the student without the extra context, the method aims to make the improvement persist beyond the prompt.

4. 37:56

   Sequential tasks and richer feedback

   On the Tulu dataset with Qwen 7B, Shenfeld reports that self-distillation with demonstrations improved task accuracy with less loss on earlier capabilities than ordinary fine-tuning. Across sequential tool-use, scientific question answering, and medical tasks, he says the method retained all three capabilities by the end, with some decline during later training. For code generation, he reports that using execution errors and failed tests as contextual feedback outperformed GRPO on LiveCodeBench by nearly 10%; a separate experiment using 10,000 WildChat conversations found small gains across several evaluations.

5. 52:03

   Limits and robotics directions

   The method depends on a teacher capable of using context and handling student-generated responses that may be outside its usual distribution; Shenfeld says a 3B model performed worse than fine-tuning in his comparison. For robotics, he identifies a missing ingredient: capable vision-language-action models with in-context learning and access to history. He points to limited robotic data as a possible factor and raises the question of how training can preserve the contextual abilities present in vision-language models.