iFairy
A 2-bit complex-valued large language model in which every weight is one of four complex values, achieving efficient compression with minimal accuracy loss compared with real-valued 2-bit models.
iFairy, also written Fairy plus-minus i, is described as the first 2-bit complex-valued large language model, with all weights constrained to the four values plus-one, minus-one, plus-i and minus-i. By introducing a complex-valued architecture alongside a new quantization scheme, it achieves efficient compression while reporting minimal accuracy loss.
Trained from scratch on 100 billion tokens, it is reported to consistently outperform real-valued 2-bit baselines such as BitNet b1.58 and symmetric INT2 on language modeling and zero-shot benchmarks, and at 1.3 billion parameters is reported as the first 2-bit model to surpass a full-precision real-valued model of the same size.
The repository includes model code compatible with HuggingFace's remote-code loading, training scripts using Accelerate and DeepSpeed, evaluation scripts for perplexity and zero-shot tasks, and data processing scripts for the RedPajama-Data-1T corpus. Training and evaluation require a Linux machine with an NVIDIA Ampere-or-newer GPU and FlashAttention-2; macOS and Windows are not supported. It is licensed under Apache-2.0.