tinyserve
A Python library and CLI that runs mixture-of-experts language models larger than a consumer GPU's memory by offloading unused experts to CPU, reaching real-time-usable throughput on an 8 GB laptop GPU.
tinyserve lets a mixture-of-experts model that does not fit in a consumer GPU's memory run anyway, by keeping expert weights mmap'd as raw quantized bytes in pinned CPU memory and loading only the experts a given token actually needs, with no dequantization step. On an 8 gigabyte laptop GPU running a 20-billion-parameter mixture-of-experts model, the project reports 30 tokens per second, and flat throughput out to 32,000 tokens of context using a streaming attention mode that caps the attention window rather than letting cost grow with context length.
It stores the key-value cache in FP8 by default to roughly halve memory use versus BF16, supports native MXFP4 and GGUF quantized formats through ggml CUDA kernels, and can be used as a Python library that loads and offloads a Hugging Face model, or as a command-line tool that serves an OpenAI-compatible API, runs an interactive terminal session, or reports a model's architecture profile.