iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs
A research training method and codebase where a student model learns from a teacher's demonstrations while a constraint limits how far it can drift, aimed at fine-tuning a model on a new task without forgetting old ones.
iSDFT builds on on-policy self-distillation fine-tuning: a student model generates a response, and a teacher model conditioned on a demonstration provides token-level guidance, but instead of always matching the teacher's full output distribution, iSDFT constructs the closest distribution to the current student that still satisfies a teacher-information constraint, with a separate anchor back to the frozen initial model limiting drift during training.
Across four base models and two specialization tasks, the authors report iSDFT improving on a prior self-distillation baseline in most settings while retaining more of the base model's performance on a six-benchmark retention suite. The repository includes the trainer, the tool-use and science training datasets and their evaluators, trained checkpoints on Hugging Face, and requires Python 3.12 with a CUDA GPU that supports BF16 and enough memory to hold a student, an exponential-moving-average teacher and a frozen base model at once.