frink
A GGUF model inference engine written entirely in Rust, with quantized kernels for CPU, Apple Metal and CUDA, and an OpenAI-compatible server benchmarked directly against llama.cpp.
frink loads GGUF model checkpoints and runs them without depending on llama.cpp bindings or a ggml wrapper; its loader, quantized kernels, attention and expert routing are all written in Rust. It ships as a single binary that also includes a benchmarking tool, a quantizer, an importance-matrix tool and a GGUF-splitting tool, and it aims to be a drop-in replacement for llama.cpp's own command-line flags and tools, producing byte-identical quantized output where the two overlap.
Weights stay quantized in memory and are dequantized inside the matrix multiplication, so an 8-billion-parameter model can fit on a laptop, and its OpenAI-compatible server supports multiple concurrent clients sharing one batched decode worker, paged key-value cache shared across conversations, runtime model swapping, and speculative decoding. Every reported speed number in its documentation was measured against llama.cpp on the same machine and model file.