For the #huggan event we decided to fine-tune a cloob-conditioned latent diffusion model. Training largely followed the guidelines in https://github.com/JD-P/cloob-latent-diffusion The (lightly modified) training script is included in this model repo, but I'd recommend starting in the github repository linked above. The model was fine-tuned on Wikiart (huggan/WikiArt) for ~12 hours on 2 A6000 GPUs kindly provided by Paperspace. The large files in this repository are
- df_model.ckpt: The latent diffusion model
- ae_model.ckpt: The autoencoder in whose latent space we operate
- ae_model.yaml: the config file needed to load said autoencoder
The final component is CLOOB, but the code for sampling will include downloading the right pretrained CLOOB model since we didn't modify that bit.
More info coming soon.
For now if you really want to try it, this colab (https://colab.research.google.com/drive/1HPu6tz44brMnKOU4G6Luhy1_iMgDl1fw#scrollTo=2s0I4RksiHtA) will download the models and set up a gradio interface for it.