TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents
The official research codebase for a training method that stabilizes distilling a large teacher agent's behavior into a smaller student agent across long, multi-turn tasks.
TCOD is the official codebase for a training method addressing an instability the authors call trajectory-level KL instability, where a student agent trained by on-policy distillation from a larger teacher drifts far from the teacher's behavior as it explores longer multi-turn interaction trajectories, such as in ALFWorld, WebShop, or ScienceWorld, causing the supervision signal to become unreliable. Instead of exposing the student to a full trajectory from the start of training, TCOD applies a temporal curriculum that progressively expands the trajectory depth used for training, keeping the student within the teacher's guidance range throughout.
Two complementary trajectory orderings are provided: a backward-to-forward strategy that starts distillation from the later, clearer steps of a trajectory and extends supervision toward the beginning, and a forward-to-backward strategy that starts from the early, most on-distribution steps and extends toward longer horizons. The authors report up to an eighteen-point improvement over standard on-policy distillation, more stable training curves, and student agents that in some cases surpass their teacher.
Its code implements multi-turn training workflows for all three environments with runnable configuration files, and the underlying paper was accepted at COLM 2026.