Awesome Private AI
A curated list of tools and frameworks for running, building, and deploying AI privately on-prem, air-gapped, or self-hosted, spanning inference engines, model serving, and privacy tooling.
Awesome Private AI is a curated list of tools, frameworks, and resources for running AI privately, keeping data, models, and infrastructure under the user's own control instead of exposing them to third parties. It is organized into sections covering inference runtimes and backends, model management and serving, fine-tuning and adapters, quantization and compression, vector databases and embeddings, agents and orchestration, VS Code plugins and extensions, privacy/security/governance, observability and evaluation, models suited to private deployment, UI and interaction layers, image and video generation, speech and audio, datasets and data preparation, learning resources and research, and AI routers and API aggregators.
Listed inference engines and runtimes include vLLM, sglang, mlx-lm, Jan, LM Studio, Cherry Studio, Ollama, llama.cpp, and llamafile, each with a short note on what distinguishes it, such as prefix caching, paged SSD KV caching, or OpenAI- and Anthropic-compatible local serving.