PocketLLM
An experimental C++/CUDA and PyTorch inference stack for running large language models on consumer multi-GPU systems, with model-specific kernels, low-bit quantization, tensor and expert parallelism, and benchmarks tuned for older hardware such as the RTX 2080 Ti.
PocketLLM began as a project to run DeepSeek-V4 on four RTX 2080 Ti cards and has since added validated runtimes for several other models, including MiniMax-M2.7, GLM-5.2, Qwen3.8-27B, DeepSeek-V4.1-Flash, MiMo-V2.6-Flash and Ternary-Bonsai-2-27B; rather than one universal backend, each model runs on a runtime matched to its own architecture and checkpoint format, combining model-specific CUDA kernels, low-bit number formats, and tensor or expert parallelism with a chosen CPU and GPU placement.
Four of its supported models are served end to end through an OpenAI-compatible API: Qwen3.8-27B-FP8 and Ternary-Bonsai-2-27B through its native C++ runtime, the latter on a single card, and DeepSeek-V4.1-Flash and MiMo-V2.6-Flash through dedicated serving backends. The project describes itself as research and engineering software, where every published number is a measurement from a specific checkpoint and hardware configuration rather than a general performance guarantee.
It is written in C++, CUDA and Python, and is licensed under MIT.