FreeToken
An edge-native mixture-of-experts serving engine that runs frontier-scale open-weight language models on consumer GPUs, with CPU-GPU co-execution, semantic caching and Anthropic/OpenAI-compatible APIs.
FreeToken is an edge-native mixture-of-experts serving engine for running frontier-scale open-weight language models on personal and consumer hardware, treating a machine's GPUs, CPUs, host memory and interconnects as one elastic inference platform. Its runtime uses bandwidth-adaptive CPU-GPU co-execution, full-layer double-buffered prefill streaming, a global least-recently-used expert cache, and its own fast weight format to serve large mixture-of-experts models efficiently.
Semantic anchor checkpoints for recurrent state and key-value caches let repeated agentic context edits, such as tool calls or thinking blocks, avoid recomputing context that has not actually changed, and VRAM can be reallocated between expert caches and key-value memory at runtime without restarting the engine or reloading weights.
It supports open-weight mixture-of-experts models such as DeepSeek-V4-Flash, Qwen3.6 and GLM-5.2 across several quantization formats, exposes Anthropic- and OpenAI-compatible APIs for existing coding and tool-calling agents, and runs on NVIDIA RTX 30, 40 and 50 series GPUs, either as a Windows or Linux desktop app with a GUI or as a command-line tool installed with pip or uv.