# USAF — Ultra Sparse Adaptive Fine-Tuning

A sparse fine-tuning method that trains a small fraction of a Mixture-of-Experts model's expert weights and router, making MoE fine-tuning possible on a 12GB consumer GPU.

USAF trains a small, fixed fraction of a Mixture-of-Experts model's expert parameters plus its router directly, rather than freezing the base weights and training small adapter matrices the way LoRA, QLoRA and DoRA do. On Qwen3-30B-A3B it trains about 24.2 million of 4.83 billion expert parameters, roughly 0.5 percent, and the project states it is the only method in its comparison that runs on 12GB of VRAM, that works on AMD GPUs, and that trains both expert weights and the router.

It was built and measured on a Radeon RX 6750 XT with 12GB of VRAM on Windows, completing a 180-step run in 7.64 hours with an in-domain perplexity of 2.76. The project states plainly that only the USAF column in its comparison table is measured; the LoRA, QLoRA, DoRA and full-fine-tuning rows are reasoned expectations rather than benchmarks, since those methods do not load on the tested hardware at all.

Website: https://github.com/tsuyu122/usaf