TokenSpeed
An open-source LLM inference engine for agentic workloads that separates a C++ control plane, which manages request lifecycles and KV cache safety at compile time, from a Python execution plane for fast iteration.
TokenSpeed is an inference engine built specifically for agentic LLM workloads, aiming for performance close to TensorRT-LLM with the usability of vLLM. Its scheduler implements request lifecycle and KV cache ownership as a finite-state machine in C++, using the type system to enforce safe cache reuse at compile time rather than at runtime, while the execution plane is implemented in Python for faster iteration by researchers and engineers.
Its kernel system is a pluggable, layered subsystem with a portable public API and a centralized registry, including a fast Multi-head Latent Attention implementation tuned for Blackwell-generation hardware, and a low-overhead asynchronous entrypoint handles request scheduling on the CPU side. It is backed by the non-profit LightSeek Foundation rather than a single company, with hardware and engineering support from vendors including NVIDIA and AMD, and is part of the PyTorch Ecosystem.
It is released under the MIT license.