LLMLingua
A research toolkit that compresses LLM prompts and key-value caches by removing non-essential tokens, reporting up to 20x compression with little quality loss.
LLMLingua uses a small, separately trained language model, such as GPT2-small or LLaMA-7B, to identify and drop tokens from a prompt that contribute little to the answer, so a larger model can be called with a shorter, cheaper prompt while keeping most of its accuracy. A related method, LongLLMLingua, targets the tendency of long-context models to lose information placed in the middle of a prompt, and is reported to improve retrieval-augmented generation quality while lowering cost.
A newer variant, LLMLingua-2, is a faster, smaller compressor trained specifically for the compression task rather than reused from a general-purpose model. The repository has been integrated into frameworks including LangChain, LlamaIndex, and Prompt flow, and its examples/ directory covers RAG, online meetings, chain-of-thought, and code use cases.
It is written in Python, is associated with EMNLP 2023 and ACL 2024 papers, and is distributed under the MIT license.