FluxForge Distillation Suite
A memory-optimization toolkit for fine-tuning SDXL-class and ANIMA diffusion models on 12 GB consumer GPUs, combining a custom optimizer that reduces optimizer-state memory with a per-layer scheduler that skips computation on layers whose gradients have gone flat.
FluxForge Distillation Suite targets fine-tuning SDXL-class latent diffusion models on 12 GB of VRAM, a budget the project positions against server-cluster or cloud-rental training. Its optimizer, described as Hybrid Momentum Distillation, combines low-order and high-order gradient moment corrections to cut optimizer-state memory by a stated 60 percent compared with AdamW, while a per-layer scheduler monitors each layer's loss contribution and reduces computation on layers whose gradients have gone near zero for a batch.
The project also describes a memory-folding mechanism that compresses memory-intensive components into overlapping representations during the forward pass and reconstructs them only as needed for the backward pass, distinguishing this from quantization or pruning since no precision or parameters are discarded.
It targets SDXL and ANIMA diffusion architectures, including rectified-flow and velocity-prediction training.