jamesob's guide to running SOTA LLMs locally
A hardware and configuration guide for running state-of-the-art large language models and speech-to-text locally, covering everything from a $2k single-GPU setup to a $40k multi-GPU rig.
This repository is a guide to running state-of-the-art large language models locally, covering the hardware the author uses, why he bought what he bought, configuration tips for it, how he runs speech-to-text locally, and ready-to-run Docker configurations for models he considers good.
It documents a base system built from a last-generation EPYC CPU and eBay DDR4 memory, four RTX Pro 6000 GPUs for 384GB of combined VRAM, PCIe4 switching hardware so the GPUs can talk to each other directly during tensor-parallel training, the BIOS, kernel and ACS settings needed to make that switching work, and GPU power limiting for running the setup on a 110V circuit. It reports resulting gen4 line-rate bandwidth and sub-microsecond latency between cards.
The repository's runners directory has ready-to-run serving configurations, including a vLLM Docker Compose setup for a large model and a speech-to-text configuration, plus a script for benchmarking peer-to-peer GPU bandwidth and latency.