KTransformers
A framework for running and fine-tuning large language models through CPU-GPU heterogeneous computing, letting large mixture-of-experts models do inference and LoRA fine-tuning on consumer hardware with CPU expert offload.
KTransformers is a research project for efficient inference and fine-tuning of large language models using CPU-GPU heterogeneous computing. It exposes two user-facing capabilities: inference and supervised fine-tuning (SFT).
It adds native support for a range of open models, including DeepSeek, Kimi, GLM and MiniMax variants, often on the day each model is released, and supports LoRA and full-parameter fine-tuning with BF16, FP8 and INT8 precision. CPU expert offload lets large mixture-of-experts models run on a single consumer GPU or on an Ascend NPU, and CPU-only backends cover AVX512 and AVX2 x86 processors, including AMD servers, without requiring AMX.
It is written in Python and released under the Apache 2.0 license.