Mesh LLM
A distributed inference system that pools GPU and memory across machines into one OpenAI-compatible API, routing or splitting models across peers as needed.
Mesh LLM pools GPUs and memory across machines and exposes the combined result as a single OpenAI-compatible API. Starting one node and adding more later lets the mesh decide whether a given model runs locally, is routed to a peer over the network, or is split across peers using staged model splitting for models too large for one machine.
It installs from a release executable through a shell script, PowerShell script, or Homebrew, with versioned formulas, Ubuntu and Arch packages, checksums, software bills of materials and OCI images produced by a companion packaging repository. Once set up, a single command picks a backend, downloads a suitable model, joins the best discovered public mesh, and starts both a local API and a web console; a private mesh can instead be started with a specific model, and a headless flag hides the web console for server deployments.
It is written in Rust and released under the Apache 2.0 license.