esp32-tinyLLM
A Rust inference engine that runs a 28.9-million-parameter language model on an ESP32-S3 microcontroller, ported from a C reference implementation and benchmarked to run faster than it.
esp32-tinyLLM runs a small language model on an ESP32-S3, a microcontroller with 512 KB of SRAM. The model fits because most of its parameters live in a flash-mapped per-layer embedding table and the rest are 4-bit quantized; it can write short children's stories but cannot answer questions or follow instructions.
The Rust implementation is benchmarked stage by stage against the C reference it was ported from, and is reported as faster at every stage, generating around 10.5 tokens per second versus 8.2 for the C version, with a further increase available through an experimental memory-bus clock flag. Both firmwares are checked for byte-identical output: every emitted token is folded into a digest that a Rust test on the host recomputes, so a change in behavior is caught even if it makes the model faster.
The project includes benchmark data files and a plotting script so its performance tables are regenerated from measurements rather than typed by hand, along with a suite of tests comparing golden logits against PyTorch and CLI output against the compiled C binary. It is licensed under Apache-2.0.