AngelSlim
A large-model compression toolkit supporting quantization, speculative decoding and distillation for LLMs, VLMs and diffusion models, aimed at making compressed models more usable and efficient to deploy.
AngelSlim is a toolkit for compressing large models, including LLMs, vision-language models and diffusion models, through quantization (including FP8-static and FP4), speculative decoding, and distillation. It supports quantization-aware distillation on Megatron-Core for MoE models with distributed training, and has released standalone drafter weights and a training framework (AngelSpec) for speculative decoding.
Its speculative-decoding components include a sparse-attention algorithm that accelerates the prefill stage of long-context LLMs by dynamically selecting top-k key blocks, a block-diffusion speculative decoding framework with layer-wise fusion, and an adaptive verification-depth pruning technique, reporting substantial end-to-end speedups over autoregressive decoding in its released benchmarks.
It publishes a technical report and documentation, and released models are available on Hugging Face and ModelScope.