exo
exo connects a user's own devices into a single AI cluster, letting a model larger than any one device's memory run by splitting it across machines, with day-0 RDMA-over-Thunderbolt support for lower latency.
Devices running exo discover each other automatically with no manual configuration, and a topology-aware scheduler decides how to split a model across the available devices based on their resources and the real-time latency and bandwidth between them; tensor parallelism is reported to give up to roughly a 3.2x speedup across four devices. It uses MLX as its inference backend and for distributed communication between devices.
It is compatible with the OpenAI Chat Completions API, the OpenAI Responses API, the Claude Messages API and the Ollama API, so existing clients and tools can point at an exo cluster unchanged. A built-in dashboard manages the cluster and lets a user chat with models directly, and custom models can be loaded from the Hugging Face hub.
Python, Apache-2.0.