# MatMul-Free LM

An implementation of a language model architecture that removes matrix multiplication operations, compatible with Hugging Face Transformers, with pretrained checkpoints from 370 million to 2.7 billion parameters.

MatMul-Free LM implements a language-model architecture that eliminates matrix-multiplication operations, using more efficient ternary weights in some layers instead, while remaining compatible with the Hugging Face Transformers library so it can be loaded with AutoModel and used with the standard text-generation APIs. The project provides pretrained checkpoints at 370 million, 1.3 billion, and 2.7 billion parameters, trained on 15 to 100 billion tokens.

Its scaling-law evaluation compares this architecture against a standard Transformer++ baseline under identical treatment of operations, and reports a steeper scaling curve, suggesting the architecture uses additional compute more efficiently. A tagged release corresponds to the archived software accompanying a Nature Computational Science paper. It requires PyTorch 2.0+, Triton 2.2+, and einops, and is Apache 2.0 licensed.

Website: https://github.com/ridgerchu/matmulfreellm