6 Transfer Learning Techniques for Training Generative Models with Limited Data

Training generative models like GANs or Diffusion Models with limited data often fails.

Standard techniques either lead to non-converging loss or severe mode collapse. The generator might simply memorize and reproduce the training data in extreme cases, such as the few-shot data regime (e.g., 10 samples from the target domain).

Transfer learning is used in literature as a powerful solution, starting with a pre-trained source generator Gₛ​ (trained on a large, diverse dataset) and fine-tuning it on limited target data. The key challenge is balancing the preservation of the source knowledge while adapting to the target distribution.

For our recent survey, I reviewed over 200 papers on this topic.

There are six effective transfer learning techniques in literature to train generative models under various scenarios, from thousands to few or even zero target-domain images.

Let’s dive in!

1. Add a regularizer to the training objective to preserve source knowledge