Liger Kernel
A collection of Triton kernels for LLM training, from LinkedIn, that reduces GPU memory use and increases training throughput with a one-line change.
Liger Kernel implements Hugging Face-compatible RMSNorm, RoPE, SwiGLU, CrossEntropy, and FusedLinearCrossEntropy operations, among others, as fused Triton kernels, reporting roughly 20% higher multi-GPU training throughput and 60% lower memory use in its benchmarks, which in turn allows longer context lengths and larger batch sizes on the same hardware. It works alongside Flash Attention, PyTorch FSDP, and Microsoft DeepSpeed.
It also adds optimized post-training kernels for alignment and distillation losses, including DPO, CPO, ORPO, SimPO, KTO, and JSD, reporting up to 80% memory savings for those workloads. It provides both high-level and low-level APIs and a published technical report describing its kernel designs.
It is written in Python and distributed under the BSD-2-Clause license.