Kimi K3 — a from-scratch implementation
An independent, from-scratch implementation of the Kimi K3 language model architecture described in its paper, reproducing its published parameter counts and running the same code path from a 20-million-parameter laptop model up to its 2.8-trillion-parameter configuration.
This project implements the architecture, pre-training protocol, systems co-designs and post-training framework described in the Kimi K3 paper. Model scale is purely a matter of configuration: the same code runs a 20-million-parameter model on a laptop CPU or describes the paper's full 2.78-trillion-parameter configuration, though only the small end is actually trained and verified here; cluster-scale training is out of scope.
Parameter counts for the full configuration reproduce the paper's published Table 1 closely: 2.778 trillion total parameters against a paper figure of 2.78 trillion, and 104.108 billion activated parameters per token against 104.2 billion. No GPU is required; the test suite, the parameter audit, five runnable demos and a small nano training run all complete on CPU, with a GPU used automatically when one is available.
Preprocessing deduplicates raw text by exact SHA-256 hash with an incremental per-file cache by default, with an optional fuzzy near-duplicate mode. On a corpus of about 48,000 documents, the model's loss falls from 8.13 to 0.32 over 400 training steps on synthetic data.