SPARKINFER
A native C++ and CUDA inference runtime for running mixture-of-experts language models on Blackwell GPUs, serving an OpenAI-compatible endpoint from a small binary with no Python dependency.
SPARKINFER is a native C++ and CUDA runtime for decoding mixture-of-experts and other large language models on NVIDIA's Blackwell GPU generation, from desktop RTX cards up to workstation-class hardware. It has no Python stack, compiles to a binary of about 2.5 megabytes, and uses kernels written specifically for Blackwell, which the project reports as faster than llama.cpp on its reference model.
A speculative decoding mode, using a smaller drafting model alongside the main one, can be turned on with one extra argument when starting the server; the server refuses to start rather than silently fall back to plain autoregressive generation if the drafting model fails to load, and switches automatically between the speculative and lossless autoregressive paths depending on the request. The server exposes OpenAI, Anthropic, Ollama and LM Studio-compatible endpoints, serves text, images and video, and reports its live context and output limits over its own status endpoint.
It ships as a container image with weights that download automatically on first run.