LoRAX
A multi-LoRA inference server that serves thousands of fine-tuned language models from a single GPU by dynamically loading LoRA adapters onto one shared base model.
LoRAX (LoRA eXchange) is a framework for serving many fine-tuned large language models from a single GPU by keeping one shared base model, such as Llama, Mistral or Qwen, and dynamically loading task-specific LoRA adapter weights per request. Adapters can be pulled from Hugging Face, Predibase or any filesystem, loaded just in time without blocking concurrent requests, and merged per request to build ensembles.
It batches requests for different adapters together to keep latency and throughput close to constant as the number of concurrent adapters grows, and asynchronously prefetches and offloads adapters between GPU and CPU memory to optimize aggregate throughput. Production features include tensor parallelism, precompiled CUDA kernels, quantization, token streaming, prebuilt Docker images, Kubernetes Helm charts, Prometheus metrics, distributed tracing, an OpenAI-compatible chat API, and per-tenant adapter isolation.
It is licensed under Apache-2.0.