Lifeboat
A local inference server that puts an OpenAI- and Anthropic-compatible API in front of open-weight language models running on hardware you control, from a laptop to a GPU cluster.
Lifeboat serves open-weight language models on hardware you control, from a laptop to a workstation, a GPU server or a cluster, and puts an OpenAI- and Anthropic-compatible API in front of them with a load balancer and a web console, keeping all data on your own network.
It installs through pip without Docker or a GPU, as a Docker or Kubernetes deployment, or as a native desktop app, and a doctor command reports a device's hardware and the largest model it can serve before any weights are downloaded. Linux builds run on hardware from an Atom processor to a current server, including Raspberry Pi boards, mini-PCs and NVIDIA Jetson devices, and an Android build runs on phones, though it is not yet publicly downloadable. Downloaded builds carry only the GGUF engine.
In the project's own benchmarks on the same hardware and model file, Lifeboat runs faster than llama.cpp and ollama.