runNburn
A memory-aware GGUF inference runtime in Rust that runs quantized language models across CPU, NVIDIA CUDA, Apple Metal and Android without expanding weights into oversized resident copies.
runNburn keeps GGUF model weights file-backed rather than expanding them into a larger in-memory format, so host residency and accelerator caches stay bounded to detected RAM, VRAM and backend capability rather than requiring a converted model or an expanded resident copy. Model architecture, including dense attention, GatedDeltaNet, Mamba-style recurrence and sparse mixture-of-experts models, determines execution semantics, while available hardware determines where computation is placed.
It supports common GGML quantization formats from Q2_K through Q6_K plus Q4_0 and Q8_0 across x86, ARM NEON, CUDA and Metal, and exposes the same model-loading and memory-policy behavior through a CLI, a Rust API, an Android C ABI and an OpenAI-compatible server. The project is pre-1.0 and under active development, with CPU as the default path and CUDA and Metal acceleration still evolving.