# Ferrum

A Rust-native LLM serving engine, distributed as a single binary, that serves local models over an OpenAI-compatible API with no Python runtime needed.

Ferrum is a single-binary LLM serving engine written in Rust, running with Metal acceleration on Apple Silicon and CUDA on supported NVIDIA GPUs, with no separate Python runtime required. It exposes OpenAI-compatible Chat Completions and stateless Responses APIs with streaming, tool calls, and structured output.

It supports continuous batching, a paged KV cache, a prefix cache, and typed admission control, along with optional 8-bit KV storage on newer Metal and CUDA paths (FP16 remains the default). Model formats include GGUF on both Metal and CUDA, plus GPTQ and safetensors on CUDA; example supported models include Qwen3.5 4B, Qwen3 30B-A3B, and Llama 3.1 8B dense.

The installer verifies release checksums, adds the binary to the shell PATH, and falls back from a CDN to GitHub if needed; Windows x64 supports CPU inference and compatible NVIDIA GPUs.

Website: https://ferrum.pandaailabs.com/