BitNet
A PyTorch implementation of BitNet, a 1-bit Transformer architecture that replaces standard linear layers with a quantized BitLinear layer to scale large language models with lower-precision weights.
BitNet is a PyTorch implementation of the linear methods and model described in the paper on scaling 1-bit Transformers for large language models. Its core building block, BitLinear, replaces a standard linear layer with a pipeline of layer normalization, binarization and absolute-max quantization, so a Transformer can be built by swapping in BitLinear modules in place of ordinary linear projections.
The repository also implements a full BitNetTransformer with multi-head attention and BitFeedforward layers, residual and skip connections, and a BitAttention variant built on multi-grouped query attention. A model must be fine-tuned or trained from scratch to use BitLinear; changing the linear layers of an already-trained model does not work.
It installs as a Python package and includes example scripts for using the BitLinear and BitNetTransformer components directly.