NVIDIA Model Optimizer
A library of model optimization techniques, quantization, pruning, neural architecture search, distillation and speculative decoding, for compressing deep learning models ahead of deployment.
NVIDIA Model Optimizer accepts Hugging Face, PyTorch or ONNX models as input, provides Python APIs to compose the optimization techniques and export an optimized quantized checkpoint, and integrates with NVIDIA Megatron-Bridge, Megatron-LM and Hugging Face Accelerate for training-time optimization. Exported checkpoints are ready for deployment in SGLang, TensorRT-LLM, TensorRT or vLLM, and its unified Hugging Face export API supports both transformers and diffusers models.
Documented tutorials in the material include an end-to-end NVFP4 weight-and-activation quantization plus quantization-aware distillation recipe reaching up to 1.30x vLLM throughput over BF16 with 3.1x smaller checkpoints, an automatic mixed-precision assignment method, a quantization-aware distillation recipe for a Nemotron model, and a pruning-plus-distillation-plus-FP8 recipe reaching 2.6x throughput and 2.6x memory reduction.
It is written in Python and licensed under Apache-2.0.