Arctic Inference
An open-source vLLM plugin from Snowflake that combines advanced parallelism, speculative decoding and model optimizations to raise inference throughput and lower latency for large language models and embeddings.
Arctic Inference patches vLLM automatically once installed, so existing vLLM APIs and CLI commands keep working while its optimizations, including Arctic Ulysses sequence parallelism, Shift Parallelism, an Arctic Speculator, Suffix Decoding and SwiftKV, run underneath. Reported results show a single deployment reaching 3.4 times faster request completion and 1.06 times higher throughput than the best throughput-optimized baseline, and 1.7 times higher throughput with 1.28 times faster completion than the best latency-optimized baseline.
For embeddings, it reports 1.4 million tokens per second per GPU, 16 times faster than plain vLLM on short sequences and 2.4 times faster than Text Embeddings Inference. It installs with pip install arctic-inference[vllm].