Running Local LLMs
A guide to running large language models on local hardware using frameworks such as llama.cpp, Ollama, Hugging Face Transformers, vLLM and LM Studio, with setup steps and performance notes.
This guide walks through running large language models on a local machine for privacy, offline access and cost efficiency, listing hardware and software requirements such as a multi-core CPU, at least 16GB of RAM, an optional 8GB-or-larger GPU, and Python and CUDA versions needed for GPU acceleration.
Each covered framework gets its own installation and usage section: llama.cpp for CPU- and GPU-optimized inference with quantization, Ollama for a simple local API and one-line model pulls, Hugging Face Transformers including quantized loading with BitsAndBytes and TorchAO, vLLM, and LM Studio, along with a performance comparison across them.