# ollamaMQ

A high-performance Rust proxy that queues requests from multiple users in front of one or more Ollama or LM Studio instances, dispatching them in parallel with a least-connections, round-robin, model-aware scheduler and a real-time terminal dashboard.

ollamaMQ is an asynchronous message queue dispatcher and load balancer that sits in front of one or more Ollama or LM Studio API instances. It queues incoming requests from multiple users and dispatches them in parallel across backends using a fair-share, least-connections-plus-round-robin scheduler, automatically detecting each backend's API type (Ollama-style or OpenAI-style) so a request only ever reaches a compatible server.

It identifies the model requested in a call's body and routes only to backends that list it, preferring a backend where the model is already loaded, then a cold backend with nothing loaded, and only as a last resort a backend that would have to evict a different loaded model; model-name matching handles tags, case and quantization-suffix variants without matching unrelated substrings. Models can be loaded or unloaded on connected backends directly from the terminal UI or the admin HTTP API, and the model-listing endpoints aggregate every compatible backend's models into one response.

It is written in Rust and processes multiple requests to different backends simultaneously rather than one at a time.

Website: https://github.com/Chleba/ollamaMQ