Self-Hosted AI Stack
A ready-to-run Docker Compose bundle of local AI services, including Ollama, a chat UI, an LLM gateway, embeddings, speech-to-text, text-to-speech and an MCP gateway.
Self-Hosted AI Stack packages Ollama for running local models, LiteLLM as a gateway that also routes to more than a hundred external providers, AnythingLLM as a password-protected web chat UI, Whisper and WhisperLive for speech-to-text, Kokoro for text-to-speech, an embeddings service for semantic search and RAG, an MCP Gateway exposing filesystem, fetch, GitHub, search and database tools, and Docling for converting documents to structured text.
All services auto-configure and generate their own API keys on first start, an optional Caddy overlay adds automatic TLS, and the stack can run with as little as about 4.5 GB of RAM in its lightweight configuration or with NVIDIA CUDA acceleration for heavier models. It supports both amd64 and arm64 Linux hosts and starts with a single docker compose command.
It is distributed as shell scripts and Docker Compose files under the MIT license.