llama.cpp
A C/C++ library and set of tools for running large language model and vision-language model inference with minimal setup across a wide range of hardware, from CPUs to GPUs.
llama.cpp is a plain C/C++ implementation for running LLM and VLM inference locally or in the cloud, aiming for minimal setup and high performance across hardware. It includes a command-line tool, a server with an OpenAI-compatible API, and a built-in web UI, and it can download and run models directly from Hugging Face.
It supports many backends and architectures, including Apple silicon through ARM NEON, Accelerate and Metal, AVX, AVX2, AVX512 and AMX on x86, RISC-V extensions, and GPU backends for NVIDIA through CUDA, AMD through HIP, Moore Threads through MUSA, Vulkan and SYCL. Models can be quantized from 1.5-bit to 8-bit integers to reduce memory use, and CPU and GPU hybrid inference can run models larger than the available VRAM.
It is built on the ggml library and distributed under the MIT license.