llama-zip
A lossless compression utility that uses a user-provided LLM as the probabilistic model for an arithmetic coder, achieving high compression ratios on structured or natural-language text.
llama-zip is a lossless compression utility that uses a user-provided LLM as the probabilistic model for an arithmetic coder, so text the model predicts with high confidence needs very few bits to encode. This gives it high compression ratios on structured or natural-language text.
A sliding context window means it is not limited by the LLM's own context length and can compress strings of arbitrary length. It also handles arbitrary binary data, not just text, by encoding invalid UTF-8 bytes as code points in Unicode's private-use areas, though at a reduced compression ratio compared to text input.
Its compression ratios were benchmarked on the Calgary Corpus text files against other compression utilities, including cmix, paq8px, zpaq, brotli, bzip2, lzma, xz, zstd and gzip, using three open-weight LLMs at two context lengths each.