bitnet.cpp
The official inference framework for 1-bit large language models such as BitNet b1.58, with optimized kernels for fast, lossless inference on CPU and GPU.
bitnet.cpp provides optimized kernels for running 1.58-bit (ternary) BitNet language models on CPU and GPU, with NPU support planned. In the project's own benchmarks, it reports speedups of 1.37x to 5.07x on ARM CPUs and 2.37x to 6.17x on x86 CPUs over its baseline implementation, alongside lower energy use, and states it can run a 100-billion-parameter BitNet b1.58 model on a single CPU at speeds comparable to human reading.
The project also releases models built on this format: BitNet-b1.58-2B-4T, a 2.4-billion-parameter ternary language model trained on 4 trillion tokens, and BitNet embedding models at 0.6B and 270M parameters that use I2_S quantization for faster CPU inference.
It is written in C++ and released under the MIT license.