Model description
This is a Stable Diffusion model fine-tuned using Dreambooth on Calvin and Hobbes images 🐯. Part of the Keras Dreambooth Event
Intended uses & limitations
- For experimentation and curiosity.
Training and evaluation data
- This model was fine-tuned on images of Calvin and Hobbes.
Training procedure
Starting from the provided Keras Dreambooth Sprity - HuggingFace - GITHUB, the provided IPYNB was modified to accomidate user images and optimze for cost. The entire training process was done using Colab. Data preperation can complete using the free tier, but you will need a premium GPU (A100 - 40G) to train. Generate the images for class-images
using free tier Colab, this will take 2-3 hours for 300 images. Once complete, you have 2 options, create the folder/root/.keras/datasets/class-images
and copy the images to that directory, or create a tar.gz file and download the files for backup using the snippit below. Downloading the file will save you the compute of having to redo it again as it can simply be uploaded into the contents
folder at a later time. You can collect the images for my_images
into a folder in either PNG or JPG format, and upload the TAR.GZ version of the folder containing the images.
import glob
import tarfile
output_filename = "class-images.tar.gz"
with tarfile.open(output_filename, "w:gz") as tar:
for file in glob.glob('class-images/*'):
tar.write(file)
To use tf.keras.util.get_file
with a tar.gz file located on your running VM you will need instance_images_root = tf.keras.utils.get_file(origin="file:///LOCATION_TO_TARGZ_FILE/my_images.tar.gz",untar=True)
get_util - Tensorflow Docs. The command, in my case, places the files in '/root/.keras/datasets/my_images'
. This will not work. The following work around resolved the issue:
# From within the Colab Notebook
!mkdir /root/.keras/datasets/class-images
!mkdir /root/.keras/datasets/my_images
!cp /root/.keras/datasets/my_images.tar.gz /root/.keras/datasets/my_images/
!cp /root/.keras/datasets/class-images.tar.gz /root/.keras/datasets/class-images/
!tar -xvzf /root/.keras/datasets/class-images/class-images.tar.gz -C /root/.keras/datasets/class-images
!tar -xvzf /root/.keras/datasets/my_images/my_images.tar.gz -C /root/.keras/datasets/my_images
Having more my_images
will improve the results. Current results used 10, but 20-30 is recommended based on Implementation of DreamBooth using KerasCV and TensorFlow - Notes on preparing data for DreamBooth training of faces.
If you decide to train using Lambda GPU - Dreambooth - GIT be sure the python -m pip install tensorflow==2.11
instead of python -m pip install tensorflow
as 2.12 will be downloaded by default and there will be a CUDA version mismatch, causing computation to occur on the CPU instead of the GPU. A second note is setting export XLA_FLAGS=--xla_gpu_cuda_data_dir=/usr/lib/cuda
and export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
from a terminal will not transfer to a notebook. Be sure to set the environment variables in the jupyter notbook to enable the proper CUDA libraries to be used otherwise the GPU will not be accessable. %env MY_ENV_VAR=value
can help and must be the first code block of the notebook.
Training results
Training hyperparameters
The following hyperparameters were used during training:
Hyperparameters | Value |
---|---|
inner_optimizer.class_name | Custom>RMSprop |
inner_optimizer.config.name | RMSprop |
inner_optimizer.config.weight_decay | None |
inner_optimizer.config.clipnorm | None |
inner_optimizer.config.global_clipnorm | None |
inner_optimizer.config.clipvalue | None |
inner_optimizer.config.use_ema | False |
inner_optimizer.config.ema_momentum | 0.99 |
inner_optimizer.config.ema_overwrite_frequency | 100 |
inner_optimizer.config.jit_compile | True |
inner_optimizer.config.is_legacy_optimizer | False |
inner_optimizer.config.learning_rate | 0.0010000000474974513 |
inner_optimizer.config.rho | 0.9 |
inner_optimizer.config.momentum | 0.0 |
inner_optimizer.config.epsilon | 1e-07 |
inner_optimizer.config.centered | False |
dynamic | True |
initial_scale | 32768.0 |
dynamic_growth_steps | 2000 |
training_precision | mixed_float16 |
Recommendations
- Access to A100 GPUs can be a challenege. A variety of providers, at the time of this writing (April 2023), either do not have additional resources or may be excessivly expensive. Spot instances, or breaking the data prep and training onto different types of machines can alleviate this issue.
Environmental Impact
Carbon emissions were estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019). In total roughly 10 hours were used primarily in US Central, with AWS as the reference. Additional resources are available at Our World in Data - CO2 Emissions
- Hardware Type: Intel Xeon (VM), with NVIDIA A100-SXM 40GB
- Hours Used: 10 hrs
- Cloud Provider: Colab Free + Colab Premium
- Compute Region: US Central
- Carbon Emitted: 1.42 kg (GPU) + 0.59 kg (CPU) = 2 kg (the weight of 2 liters of water) - 2 kg offset = 0
Model Plot
Citation
@online{ruizDreamBoothFineTuning2023,
title = {{{DreamBooth}}: {{Fine Tuning Text-to-Image Diffusion Models}} for {{Subject-Driven Generation}}},
shorttitle = {{{DreamBooth}}},
author = {Ruiz, Nataniel and Li, Yuanzhen and Jampani, Varun and Pritch, Yael and Rubinstein, Michael and Aberman, Kfir},
date = {2023-03-15},
number = {arXiv:2208.12242},
eprint = {arXiv:2208.12242},
eprinttype = {arxiv},
url = {http://arxiv.org/abs/2208.12242},
urldate = {2023-03-26},
abstract = {Large text-to-image models achieved a remarkable leap in the evolution of AI, enabling high-quality and diverse synthesis of images from a given text prompt. However, these models lack the ability to mimic the appearance of subjects in a given reference set and synthesize novel renditions of them in different contexts. In this work, we present a new approach for "personalization" of text-to-image diffusion models. Given as input just a few images of a subject, we fine-tune a pretrained text-to-image model such that it learns to bind a unique identifier with that specific subject. Once the subject is embedded in the output domain of the model, the unique identifier can be used to synthesize novel photorealistic images of the subject contextualized in different scenes. By leveraging the semantic prior embedded in the model with a new autogenous class-specific prior preservation loss, our technique enables synthesizing the subject in diverse scenes, poses, views and lighting conditions that do not appear in the reference images. We apply our technique to several previously-unassailable tasks, including subject recontextualization, text-guided view synthesis, and artistic rendering, all while preserving the subject's key features. We also provide a new dataset and evaluation protocol for this new task of subject-driven generation. Project page: https://dreambooth.github.io/},
pubstate = {preprint}
}
@article{owidco2andothergreenhousegasemissions,
author = {Hannah Ritchie and Max Roser and Pablo Rosado},
title = {CO₂ and Greenhouse Gas Emissions},
journal = {Our World in Data},
year = {2020},
note = {https://ourworldindata.org/co2-and-other-greenhouse-gas-emissions}
}
@article{lacoste2019quantifying,
title={Quantifying the Carbon Emissions of Machine Learning},
author={Lacoste, Alexandre and Luccioni, Alexandra and Schmidt, Victor and Dandres, Thomas},
journal={arXiv preprint arXiv:1910.09700},
year={2019}
}
- Downloads last month
- 2