mistral.rs
A fast, flexible LLM inference engine in Rust, with an agentic runtime, OpenAI- and Anthropic-compatible serving, and Rust and Python SDKs.
mistral.rs is a fast, flexible inference engine for large language models, with Rust and Python SDKs. It loads GGUF files directly or downloads a published quantized artifact, and its OpenAI-compatible server now also exposes Anthropic-compatible message endpoints alongside its OpenAI-compatible API.
Its agentic runtime adds web search, local Python code execution, shell execution, OpenAI-compatible Skills, session management and custom tool hooks. It supports multimodal models handling text, image, video and audio input, including recent block-diffusion text generation with full integration of paged attention, prefix caching, in-situ quantization and tool calling.
The project publishes CUDA benchmarks comparing its UQFF quantized inference against llama.cpp's GGUF format across several GPUs, reporting higher prefill and decode throughput on models such as Gemma 4.