Running (and coding with) Local AI on a Mac
A practical, hands-on guide to running large language models and coding agents locally on Apple Silicon, covering inference engine choice, memory sizing, model selection and setting up agent tool calling.
This is a comprehensive guide, gathered from scattered sources into one document, on running AI models and coding agents locally on a Mac instead of relying on a hosted service. It targets Apple's unified memory architecture, where the GPU can access nearly all of a Mac's memory, arguing this makes even a mid-range Mac with about 32 gigabytes of memory a viable alternative to a discrete consumer GPU for many workloads, while noting that dedicated GPUs still offer more raw throughput.
It is organized into three parts: architecture decisions covering inference engines, API servers, model types, recommended models and quantization; running models with LM Studio or llama.cpp, including context window, sampling, prompt caching and multi-token prediction settings; and agentic workflows, covering configuring an API server, sizing the context window and parallel requests for agent use, and setting up coding agents such as OpenCode and Claude Code against a locally served model.
It assumes some existing familiarity with large language model concepts and basic system configuration.