The Alignment Handbook
A collection of training recipes for aligning language models with human and AI preferences, spanning supervised fine-tuning, DPO, RLHF and related post-training techniques.
The Alignment Handbook provides recipes to continue pretraining and to align language models to human or AI preferences, aiming to fill a gap in public resources on what data to collect and what to measure for good downstream performance after supervised fine-tuning.
Its releases include the full post-training recipe behind SmolLM3-3B, fine-tuning recipes for SmolLM2-Instruct and SmolLM-Instruct, a Zephyr 141B (A35B) recipe built with Argilla and KAIST AI to fine-tune Mixtral 8x22B with ORPO, a recipe for StarChat2 15B coding assistants, a Zephyr 7B Gemma recipe using RLAIF, a recipe to align open LLMs with Constitutional AI, and a released suite of evaluations comparing DPO, KTO and IPO.
It is written in Python and licensed under Apache-2.0.