Distributed Llama
An inference engine that links multiple home computers into a cluster over Ethernet to run large language models faster by splitting the work with tensor parallelism.
Distributed Llama connects several home devices into a cluster to accelerate inference of large language models such as Llama, Qwen, and DeepSeek. It splits a model's computation across the machines using tensor parallelism and synchronizes them over a standard Ethernet network, so adding more devices increases speed.
It runs on Linux, macOS, and Windows, with optimizations for both ARM and x86_64 AVX2 CPUs, and more recently added GPU support through Vulkan. A single command downloads a chosen model and its tokenizer and launches the root node, and the project also documents running on a Raspberry Pi or on a GPU, and converting models from Hugging Face.
Distributed Llama is written in C++ and released under the MIT license.