Awesome Local AI
A curated list of tools, frameworks, and guides for running AI models on a user's own hardware without a cloud API, spanning inference engines, quantization, fine-tuning, retrieval, orchestration, and monitoring.
Awesome Local AI is a curated reference list for anyone who runs models on their own GPU instead of calling a hosted API, organized into sections such as inference engines, Python and language bindings, quantization and optimization, fine-tuning, retrieval and knowledge tools, orchestration and agents, model management, monitoring and profiling, user interfaces, code assistants, voice and audio, image and video, privacy-first tools, unified gateways, and hardware guides.
Its inference-engines section alone spans a wide range of approaches, from llama.cpp and Ollama for general local inference, to vLLM and SGLang for high-throughput serving, MLX LM for Apple Silicon, LocalAI as an OpenAI-compatible drop-in server, and single-executable options such as llamafile and KoboldCpp, each annotated with a short note on what it is best suited for. Every listed tool is described as working offline once its initial setup is complete.