llama-cpp-rs
Safe Rust bindings to llama.cpp that track the upstream project closely, exposing a high-level API alongside raw bindgen bindings, with example programs for chat, embeddings and an OpenAI-compatible server.
The project packages llama.cpp as two crates: a safe, high-level Rust API and a lower-level crate of raw generated bindings. It tracks a specific recent llama.cpp commit that includes TurboQuant, multi-token-prediction speculative decoding, and DeepSeek V4's MTP and DSpark support, surfaced through the high-level crate's model-loading parameters.
Example programs cover single-turn text completion from either the command line or a downloaded Hugging Face model, an interactive multi-turn chat REPL, batch embedding with cosine similarity, loading sharded or split GGUF files, an OpenAI-compatible HTTP server supporting chat, completions, embeddings, tool calls and file inputs, multimodal vision and audio inference, GGUF quantization with a fully typed API, and speculative decoding both with a draft model and with a draft-free n-gram lookup that is verified to be byte-identical to plain greedy decoding.