Awesome LLM On-Policy Distillation
A curated collection of papers and resources on on-policy distillation for large language models, accompanying a survey paper and a companion site with full-text search.
Awesome LLM On-Policy Distillation tracks papers and resources on on-policy distillation, a technique where a student model generates its own trajectories and is then evaluated against a teacher model, reward model or verifier, rather than learning only from a teacher's demonstrations as in traditional off-policy distillation.
It accompanies a survey paper that provides a taxonomy of on-policy distillation methods by objective design, signal source and training-stabilization technique, now in its fourth revision, and links a companion site with full-text search and filters by section, loss, domain, signal source, rollout frequency, student size and year.
The repository also references a related unified framework covering more than ten on-policy distillation methods, released as a companion toolkit to the survey.