llamacpp-distributed-inference
A cross-platform Electron desktop app that coordinates several computers on a local network to run distributed LLM inference together, using llama.cpp's RPC backend.
The app automatically discovers other nodes on the local network, or lets a user add one manually by IP, and scans a local models folder for GGUF files while letting each node's GPU layer count be adjusted individually. A local API server and remote RPC nodes cooperate on inference, with a multi-tab log view, real-time status updates, node connection checks and a display of the machine's own network addresses.
It bundles llama.cpp's rpc-server and llama-server binaries for Windows, macOS and Linux, and is built with Node.js and Electron; a packaged release excludes model files to keep the installer small, so GGUF models are added manually to a models folder afterward. It requires at least 8GB of RAM, with an optional NVIDIA GPU for acceleration, and is licensed Apache-2.0.