# Custom Diffusion

A research method and PyTorch implementation for fine-tuning text-to-image diffusion models on a handful of example images to add a new concept, without retraining the whole model.

Custom Diffusion is a method and code release for fine-tuning text-to-image diffusion models, such as Stable Diffusion, on 4 to 20 images of a new concept. It fine-tunes only a subset of model parameters, the key and value projection matrices in the cross-attention layers, which keeps training fast, about 6 minutes on 2 A100 GPUs, and adds only about 75MB of storage per additional concept.

The method also supports combining multiple learned concepts, such as a new object together with a new artistic style, or several new objects in one generation. It uses a small set of regularization images to reduce overfitting, and for personal categories it adds a new modifier token in front of the category name.

The code accompanies the CVPR 2023 paper on multi-concept customization of text-to-image diffusion and builds on Stable Diffusion.

Website: https://www.cs.cmu.edu/~custom-diffusion