Bankai: Ultra-Sparse Adaptation of 1-Bit LLMs via XOR Patches
A research method for adapting true 1-bit large language models after training, using sparse XOR bitmask patches that flip individual binary weights instead of requiring gradients or continuous weights.
Bankai patches are found by a greedy search, apply with a single bitwise XOR operation, add no inference overhead, and compress to around one kilobyte; the method is validated on Bonsai 8B, an 8.2-billion-parameter true 1-bit language model, since existing adaptation methods such as LoRA, fine-tuning and quantization-aware training all need continuous weights or gradients that binary models lack.
A series of experiments shows that binary MLP weights carry redundancy that scale-guided bit flips exploit far better than random flips, that patches trained on only a few examples memorize rather than generalize while patches trained on diverse probe variations generalize to held-out prompts, fixing several previously-wrong problems with no new breakage once measured correctly, that stacking two patches through XOR is mechanically sound but behaviorally lossy, and that a safety check on a math benchmark shows no degradation to general reasoning. It is early-stage research, reproducible on Apple Silicon, and licensed Apache-2.0.