# LLM.Q

A CUDA and C++ implementation of quantized large language model training, aimed at single-node, multi-GPU training setups, inspired by llm.c.

LLM.Q implements quantized large language model training directly in CUDA and C++, without a Python training framework in the critical path, though optional Python bindings are available. It targets medium-sized training setups, specifically a single node with multiple GPUs, and depends on NCCL for communication and cuDNN for fast attention, with multi-GPU runs supported through either multi-process mode via OpenMPI or multi-thread mode.

A bundled tokenization script prepares token files for a limited set of datasets, and the project includes a worked example that fine-tunes a small Qwen model on a Shakespeare dataset, showing options such as bf16 precision, e4m3 matmul precision, gradient accumulation and activation-recomputation settings for fitting training onto a given amount of GPU memory.

Website: https://github.com/IST-DASLab/llmq