# Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

This project studies why on-policy distillation of large language models succeeds or fails: it finds the technique needs a student and teacher with compatible thinking patterns and genuinely new capability, then proposes two ways to recover a failing run.

This repository accompanies a research paper investigating the training dynamics and mechanisms of on-policy distillation (OPD) of large language models. It identifies two conditions that govern whether OPD succeeds or fails: the student and teacher must share compatible thinking patterns, and the teacher must offer genuinely new capability beyond what the student has already seen in training, which the authors confirm through weak-to-strong reverse distillation experiments.

At the token level, successful OPD shows progressive alignment on high-probability tokens at states the student actually visits, concentrated in a small shared token set. The paper proposes two practical strategies for recovering a failing run, an off-policy cold start and teacher-aligned prompt selection, and raises an open question about whether OPD can scale to long-horizon distillation. The code builds on verl for reinforcement learning and LlamaFactory for supervised fine-tuning.

Website: https://github.com/Thinking-Space/Rethinking-OPD