Olla
A lightweight proxy and load balancer for self-hosted LLM infrastructure, routing requests across local and remote inference backends with automatic failover and a unified model catalogue.
Olla is a proxy and load balancer for managing LLM infrastructure, intelligently routing requests across self-hosted inference backends such as Ollama, LM Studio, vLLM, llama.cpp and sglang, with model discovery and a unified model catalogue per provider.
It offers priority-based load balancing with automatic failover and connection retry, KV-cache-aware sticky sessions that pin a multi-turn conversation to the same backend, glob-based profile and model filtering, continuous endpoint health checks with circuit breakers, and automatic model discovery refresh when an endpoint recovers. It can pass through or translate the Anthropic Messages API for backends without native support, and handles outbound authentication to backends that require it.
It ships two proxy engines, a simple Sherpa engine and a higher-performance Olla engine with circuit breakers and connection pooling, configured through a single CLI application and one config file, and is written in Go.