RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation
Research code for a reinforcement-learning method that trains reasoning models with dense token-level supervision by contrasting teacher-student gaps under correct and incorrect reference solutions.
RLCSD is the source code for a paper on on-policy self-distillation for reasoning models. Standard on-policy self-distillation gives a student model dense, token-level supervision by aligning its output distribution with a teacher conditioned on a verified solution, but this signal can concentrate on style tokens rather than the tokens that carry the actual task content, a problem the paper calls privilege-induced style drift.
RLCSD contrasts the teacher-student gap under a correct reference solution against the gap under an incorrect one, using the same prompt template for both, to suppress shared stylistic shifts and sharpen the remaining signal on task correctness. That signal is used to modulate the verifier-derived advantage from group relative policy optimization while preserving its update direction.
The method is evaluated on Qwen3 (1.7B, 4B and 8B) and Olmo-3-7B-Think, using a math dataset for training with evaluation on AMC23, AIME24 and AIME25, and a Knights and Knaves benchmark for logical reasoning in and out of its training distribution.