Agentic-GRPO-LongHorizon
A research ablation study that diagnoses why GRPO training collapses on long-horizon, multi-tool conversational agents and proposes a combined PRM-Lite and LATA method that achieves a 37 percent relative gain over vanilla GRPO on the tau-bench airline benchmark.
This project identifies three causes of training collapse when applying Group Relative Policy Optimization to long-horizon, multi-tool agents on the 50-task tau-bench airline benchmark: binary outcome rewards saturating a training group to all-correct or all-wrong so the learning signal vanishes, policies memorizing teacher trajectories on tasks with available teacher data while other tasks stay near zero, and linear length-normalization of the advantage encouraging short, low-quality reasoning that trades depth for a better-looking score.
Its proposed fix combines a lightweight process reward model, PRM-Lite, with a length-aware training adjustment called LATA, reaching an overall pass rate of 0.240 at its best checkpoint versus 0.175 for vanilla GRPO, a 37 percent relative improvement, with a 55 percent improvement in generalization to unseen tasks and a 30 percent drop in error rate. The repository documents the ablation results, including a finding that training-time validation reward is not a reliable proxy for true evaluation performance.