Paper: Effective Data Augmentation With Diffusion Models (ICLR-2024)

Reviewer comment (Experiments):
DA-Fusion outperforms the much simpler RandAugment method mostly on extremely low-shot settings (less than 16 images per class). The improvements of DA-Fusion seem to be marginal when there are more than 16 images per class. Also, it is unfair to compare with RandAugment, which only uses the default hyperparameters, while DA-Fusion is “fine-tuned” on the target data (i.e., textual inversion, selecting $M$, and other parameters). The authors should search for the optimal number of operations and augmentation magnitude for RandAugment for a fairer comparison.

Author response:
DA-Fusion and RandAugment use a fixed set of hyperparameters across all tasks in the paper (excluding Leafy Spurge, which we are updating in the manuscript per the additional experiment noted in a later response). We did not fine-tune the hyperparameters of DA-Fusion per task, and doing so for either method would risk biasing the evaluation. The hyperparameters of RandAugment were shown in Cubuk et al. 2019 (https://arxiv.org/abs/1909.13719) to work well across several standard datasets.

__“The authors should search for the optimal number of operations and augmentation magnitude for RandAugment for a fairer comparison.”__

This would risk unfairly favoring RandAugment in our evaluation. We use a uniform set of hyperparameters with DA-Fusion and do the same with RandAugment and Real Guidance. Note that standard data augmentation can be applied in parallel with DA-Fusion (see Figure 3 in the paper), and improvements in performance due to fine-tuning RandAugment per task would also improve the performance of DA-Fusion by a corresponding amount.

Reviewer comment (Experiments):
The authors tested their method on COCO and PASCAl VOC datasets for pure classification but not object detection. Being a general image-to-image diffusion model, the generation step in DA-Fusion may alter the position of the objects in input images. It seems that DA-Fusion can only be applied to classification tasks.

Author response:
Our focus in this work is classification, but our public code supports masks for objects, allowing DA-Fusion to independently transform objects in the image, and the background. (This is the relevant section)[https://github.com/anonymous-da-fusion/da-fusion/blob/master/semantic_aug/augmentations/textual_inversion.py#L152 ] of the anonymous code. By using masks, DA-Fusion can preserve the location of objects in the image. Examples using DA-Fusion with masks to preserve the locations of objects are in the following anonymous google drive link. This functionality allows our method to be applied to datasets that require object locations to be preserved during augmentation, such as object detection and segmentation tasks. We are adding a section discussing this application of DA-Fusion in the appendix of the manuscript.
https://drive.google.com/drive/folders/15O0HWkQsEskmzmI8F14ISf2xNdwOlKBG?usp=sharing

Reviewer comment (Experiments):
The authors used a different number of augmented images per real image, $M$ ($M$=50 for spurge and $M$=10 for other data). Are there any guidelines for selecting $M$?

Author response:
We are happy to include a discussion of the role and selection of M in the next revision of the manuscript. Introduced in Section 4.2, this hyperparameter controls the number of synthetic images generated per real image before training starts. The larger M, the higher the computational cost of DA-Fusion, but the higher performance can be attained. We ablate M in Figure 11 and find that our method is not sensitive to its value, observing that larger values of M perform marginally better than smaller values. M should be chosen based on the computational constraints of the user, but we find M=10 is an effective value across standard datasets.

__“The authors used a different number of augmented images per real image, M (M=50 for spurge and M=10 for other data).”__

We have evaluated DA-Fusion on the new Leafy Spurge task with the same value of M as the rest of the datasets in Figure 5 and will add this in the manuscript. The new experiment is consistent with the existing results in the paper: that DA-Fusion continues to outperform RandAugment and Real Guidance. Results can be viewed at this anonymous link:
https://drive.google.com/drive/folders/1cSPeNTOmZK-vTTrxF5xvtEtBtzrYRMXd?usp=sharing

Reviewer comment (Experiments):
As pointed out in the weakness, the technical contribution of this work is slightly limited. It is recommended to compare DA-Fusion with more and stronger data augmentation baselines (instead of just RandAugment and Real Guidance) to claim a more significant empirical contribution.

Author response:
At time of submission, the most recent and performant generative data augmentation is Real Guidance, adapted from He et al. 2023 (https://arxiv.org/abs/2210.07574). Being the state-of-the-art method, Real Guidance is the strongest and most relevant baseline to include. RandAugment is chosen for inclusion due to being recent, popular, and performant. We include an additional comparison against CutMix (Figure 12), another recent, popular, and performant data augmentation, and results are consistent with the existing comparison to RandAugment.

If you have any specific additional data augmentation baselines to compare to DA-Fusion, we are happy to include them in the next revision of the manuscript.

Reviewer comment (Experiments):
The proposed technique is only applicable for classification tasks. It is not clear how it can be applied for object detection and segmentation tasks. My thinking is that one of the major drawbacks of such generative model-based augmentation method vs. traditional method may be that it can not simultaneously generate the segmentation mask and bounding box annotation for the augmented images.

Author response:
__“one of the major drawbacks of such generative model-based augmentation method vs. traditional [data augmentation] may be that it can not simultaneously generate the segmentation mask and bounding box annotation for the augmented images”__

Applying DA-Fusion to images with segmentation mask and bounding box annotations requires either (1) generating the annotations with the image, or (2) taking existing annotations and augmenting the corresponding input images in a manner that preserves the annotations. Our anonymous public code __supports capability (2) out-of-the-box__, and (here is the relevant section)[https://github.com/anonymous-da-fusion/da-fusion/blob/master/semantic_aug/augmentations/textual_inversion.py#L152] of the anonymous code. By using masks, DA-Fusion can preserve the location of objects in the input images. Examples using DA-Fusion with masks to preserve the locations of objects are in the following anonymous google drive link. We are adding a section discussing this application of DA-Fusion in the appendix of the manuscript.
https://drive.google.com/drive/folders/15O0HWkQsEskmzmI8F14ISf2xNdwOlKBG?usp=sharing

Reviewer comment (Experiments):
I am not clear how the data-centric leakage prevention is performed. The paper mention that "switching from a prompt ... is sufficient" and "Section 4.1 goes into detail". If data-centric leakage refers to using prompt like "a photo of a Class3", what is the setting difference between Fig.5 and Fig.9?

Author response:
__“If data-centric leakage refers to using prompt like ‘a photo of a Class3’, what is the setting difference between Fig.5 and Fig.9?”__

Data-centric leakage prevention modifies the prompt used with Stable Diffusion to omit the name of the class, and we discuss the technical details in the paragraph titled “Data-Centric Leakage Prevention” in Section 6.1 of the manuscript. The high-level difference between Figure 5 and Figure 9 is whether Stable Diffusion knows the name of the class of the image. If the class name is known as in Figure 5, we prompt Real Guidance with “a photo of a <class-name>” and we initialize the learned pseudo-prompt for DA-Fusion with the class name.

If we don’t know the class name as in Figure 9, we prompt Real Guidance with “a photo” and we initialize the learned pseudo-prompt for DA-Fusion with the word “the”. These values are discussed in Table 1 of Appendix G - Hyperparameters. The ablation in Figure 9 shows that DA-Fusion works well for concepts that Stable Diffusion may not have learned the name of.

Reviewer comment (Evaluation):
Desiderata – Controllable (content). Is textual inversion (as it is used in the paper) controllable? I.e. Is it easy to control for whether the token learns the style or object? Or if certain objects tend to co-occur with another, e.g., train and rails, it may not be necessarily learning the token for trains but also the objects that co-occur with it.

Author response:
# Controlling What DA-Fusion Generates: 

__“Is it easy to control for whether the token learns the style or object? … it may not be necessarily learning the token for trains but also the objects that co-occur with it.”__

Generative models are uniquely promising for data augmentation because they allow the user to control the content and style of images in training datasets. DA-Fusion implements three simple interfaces that control the content and style of what is generated by our method.

1) guiding what’s generated with prompts

2) guiding what Textual Inversion learns with prompts

3) erasing the ability to generate specific concepts written in prompts

Each control method has a simple natural language interface. 

__Control Method 1__: The word embeddings learned by DA-Fusion are flexible, and the user can modify the content and style of generations after learning the word embeddings. For example, if the prompt “a photo of a <class>” leads to the images named “original generation” at the following anonymous link, changing the prompt to “a photo of a <class> wearing a bow-tie”, leads to the images named “generation with bow-tie” at the following anonymous link:
https://drive.google.com/drive/folders/1gnCa0SBd9tadL9o2SK0jmNNYoNvC76Sc?usp=sharing
We are updating the manuscript to discuss __Control Method 1__ in the appendix.

__Control Method 2__: Suppose we show DA-Fusion the image named “original image” at the following anonymous link and learn a “<class>” word embedding using this image. The image has a cat and a dog. If we want the “<class>” word embedding to focus on the cat and ignore the dog, we can prompt Stable Diffusion with “a photo of a <class> sitting on the right of a dog” when learning the “<class>” word embedding. If we generate images using an inference-time prompt “a photo of a <class>” the generations only have cats, showing the “<class>” word embedding has ignored the dog and focused on the cat in the original image.
https://drive.google.com/drive/folders/1lmr7e7qBZHwyGJyIkxt-0bUAxcn0934M?usp=sharing
We are updating the manuscript to discuss __Control Method 2__ in the appendix.

Controlling the style of what tokens learn can be done with Textual Inversion. We build on the Textual Inversion script released by HuggingFace, and by using `--learnable_property=style` the user can switch modes and cause tokens to learn style rather than object-based concepts.

__Control Method 3__: Finally, we can erase Stable Diffusion’s generation ability for specific concepts. Suppose we generate images of roads using the prompt “a photo of a <class>” and we realize that cars often show up in the generations, but our real data never has cars. We can erase the “car” concept and prevent Stable Diffusion from generating it, resulting in the following images. Additional examples for erasing concepts are in Figure 7 of the manuscript.
https://drive.google.com/drive/folders/1eDkXFKhpN94ZEhQtih97PdXsJtjzqKuz?usp=sharing
We discuss __Control Method 3__ in Section 6.1 of the manuscript.

Reviewer comment (Evaluation):
Desiderata – Performant. Gains in accuracy should justify the additional computational cost of generating images from Stable Diffusion. There does not seem to be any comparisons with compute cost relative to the baseline Rand Aug. DA-Fusion requires performing textual inversion, which may be expensive.

Author response:
# Computational Cost Analysis:

__“There does not seem to be any comparisons with compute cost relative to the baseline Rand Aug. DA-Fusion requires performing textual inversion, which may be expensive.”__

The trade-off of performance and computational cost is important for users considering the viability of generative data augmentations. We agree the cost of DA-Fusion is higher than traditional data augmentation, but our method is parallelizable, and the cost is once upfront, whereas the cost of traditional data augmentation is at every training iteration.

We provide statistics for DA-Fusion in the following table, assuming a dataset with 100 classes, 4 images per class, a training batch size of 32, trained for 10,000 steps, with M=5. The machine has 4 Nvidia RTX 6000 ada GPUs, and image generation is parallelized across all the GPUs.

| Method | Total Time (minutes) |
| ----------- | ----------- |
| DA-Fusion (25-step diffusion sampler) | 21:19 |
| DA-Fusion (12-step diffusion sampler) | 14:47 |
| RandAugment | 12:18 |
| No Augmentation | 08:15 |

DA-Fusion is only 20.2% more expensive than RandAugment with the 12-step diffusion sampler according to wall-clock time in this configuration, and performs up to 24.2% better for fine-grain concepts, which is discussed in the paper in Section 6, Figure 6 in the manuscript. We are adding this cost analysis to the paper.

Reviewer comment (Experiments):
Desiderata – Performant. It would also be interesting to compare the performance with standard augmentations like Rand Aug for different M. For example, Sariyildiz et al. [1] showed that training on synthetic data outperforms when the generated dataset size is much larger than the original dataset size. It is possible that at smaller M, standard augmentations outperform synthetic data, with the additional benefit that it is also fast to compute.

Author response:
__“It is possible that at smaller M, standard augmentations outperform synthetic data, with the additional benefit that it is also fast”__

In Figure 11 of the manuscript, we ablate M, and find that our method is not sensitive to this hyperparameter for values $M \in \{ 5, 10, 20 \}$. Smaller values of M perform marginally worse than larger values of $M$, and have the advantage of a smaller computational cost. Based on the cost analysis, DA-Fusion is only 20.2% more expensive than RandAugment according to wall-clock time in our configuration, and performs 24.2% better for fine-grain concepts.

Reviewer comment (Experiments):
How does the method perform zero shot? If the learned tokens did capture the class characteristics, and there is no distribution shift between training and test, it seems like DA-Fusion should do well.

Author response:
__“How does the method perform zero shot?”__

Requiring a handful of real images is an intuitive interface for users of DA-Fusion because they can show our method what kind of images they want to generate. Being fully zero-shot, the user would have to describe all aspects of the generated images in the prompt, which can be difficult. While DA-Fusion could be adapted to this setting, we found it works best when real images are used as a guide (see the paragraph titled “Generating Synthetic Images” in Section 4.1). Previous works already show synthetic data from Diffusion Models improves classification when no real data is available, such as in Azizi et al. 2023, https://arxiv.org/abs/2304.08466.

Reviewer comment (Experiments):
Is the leafy spurge task a binary classification problem, i.e., detecting if there is leafy spurge in the image or not? If that is the case, how often do other plants occur in the data? This relates to wondering how much fine-grained details textual inversion can capture.

Author response:
__“Is the leafy spurge task a binary classification problem i.e., detecting if there is leafy spurge in the image or not? If that is the case, how often do other plants occur in the data? I was wondering how much fine-grained details can textual inversion capture.”__

The leafy spurge task is binary: the target is present or absent. Other plant species are always present in the leafy spurge images. The composition of plant species was highly varied and capturing this diversity was intentional on the part of the authors, who hoped to represent the breadth of plant communities. Many spurge plants are immediately adjacent to other plants, and often partially obscured by them. For example, a shrub or tree may contribute overstory and obscure leafy spurge plants. Additionally, the authors captured spurge at various stages in the plant’s life history, where some individuals in an image show distinct yellow flowers, while others do not. Leafy spurge plants may cluster spatially as their colonies expand, forming large contiguous areas. The degree to which plants are clustered is also varied throughout the dataset and even in individual images. You might encounter an individual in one region of the image, and a continuous mat of individuals in another region of the same image.

Reviewer comment (Experiments):
How much variance is there in the class word embedding? If textual inversion is performed again with a different seed, how different would the generations be? If they are different, it could be another way to introduce diversity.

Author response:
__“How much variance is there in the class word embedding?”__

Different seeds of Textual Inversion lead to different image generations. We have uploaded generations for different Textual Inversion seeds for cats from the PASCAL VOC task at the following anonymous link. This could provide additional diversity for DA-Fusion.
https://drive.google.com/drive/folders/16uOtwsks53mTnXe2DEaDCLFNhQ_zc-yJ?usp=sharing

Reviewer comment (Experiments):
Complexity and Computational Cost: The proposed method involves fine-tuning pseudo-prompts for each concept, which can be computationally expensive and time-consuming. This approach might not be as practical as traditional data augmentation techniques that are computationally efficient and easy to implement.

Author response:
__“This approach might not be as practical as traditional data augmentation techniques that are computationally efficient and easy to implement.”__

The trade-off of performance and computational cost is important for users considering the viability of generative data augmentations. We agree the cost of DA-Fusion is higher than traditional data augmentation, but our method is parallelizable, and the cost is once upfront, whereas the cost of traditional data augmentation is at every training iteration.

We provide statistics for training a classifier with DA-Fusion in the following table, assuming a dataset with 100 classes, 4 images per class, a training batch size of 32, trained for 10,000 steps, with M=5. The machine has 4 Nvidia RTX 6000 ada GPUs, and image generation is parallelized across all the GPUs.

| Method | Total Time (minutes) |
| ----------- | ----------- |
| DA-Fusion (25-step diffusion sampler) | 21:19 |
| DA-Fusion (12-step diffusion sampler) | 14:47 |
| RandAugment | 12:18 |
| No Augmentation | 08:15 |

Training a classifier with DA-Fusion is only 20.2% more expensive than RandAugment with the 12-step diffusion sampler according to wall-clock time in this configuration, and performs up to 24.2% better for fine-grain concepts, which is discussed in the paper in Section 6, Figure 6 in the manuscript. We are adding this cost analysis to the paper.

__“The proposed method involves fine-tuning pseudo-prompts for each concept, which can be computationally expensive and time-consuming.”__

We agree that fine-tuning pseudo-prompts is time-consuming. Each concept requires 10 minutes of fine-tuning on our machine, and this can be parallelized. One fine-tuning experiment requires 11GB of RAM, and we are able to fit 4 of these in parallel on an NVIDIA RTX 6000 ada GPU, resulting in 16 parallel fine-tuning experiments with 4 GPUs. With our 100 class dataset, this requires 10 * 100 / 16 = 62.5 minutes of time spent fine-tuning.

Faster methods for learning pseudo-prompts are being developed. Recent work by Voronov et al. 2023 (https://arxiv.org/abs/2302.04841) could reduce this cost by a factor of 8, resulting in a total cost of 7.8 minutes of time spent fine-tuning. Additionally, HyperNetworks are being developed (Ruiz et al. 2023 https://arxiv.org/abs/2307.06949) that may eliminate the fine-tuning step by directly predicting the pseudo-prompt embeddings with a negligible cost.

Reviewer comment (Experiments):
Lack of Control Over Augmentations: While DA-Fusion introduces the concept of modifying images while respecting their semantic attributes, it does not explicitly provide fine-grained control over how the augmentations are performed. This lack of control may limit its applicability to specific use cases where precise image modifications are required.

Author response:
__“This lack of [fine-grained control over how the augmentations are performed] may limit its applicability to specific use cases where precise image modifications are required.”__

Generative models are uniquely promising for data augmentation because they allow the user to control the content and style of images in training datasets. DA-Fusion implements three simple interfaces that control the content and style of what is generated by our method.

1) guiding what’s generated with prompts
2) guiding what Textual Inversion learns with prompts
3) erasing the ability to generate specific concepts written in prompts

Each control method has a simple natural language interface. 

__Control Method 1__: The word embeddings learned by DA-Fusion are flexible, and the user can modify the content and style of generations after learning the word embeddings. For example, if the prompt “a photo of a <class>” leads to the images named “original generation” at the following anonymous link, changing the prompt to “a photo of a <class> wearing a bow-tie”, leads to the images named “generation with bow-tie” at the following anonymous link:
https://drive.google.com/drive/folders/1gnCa0SBd9tadL9o2SK0jmNNYoNvC76Sc?usp=sharing
We are updating the manuscript to discuss __Control Method 1__ in the appendix.

__Control Method 2__: Suppose we show DA-Fusion the image named “original image” at the following anonymous link and learn a “<class>” word embedding using this image. The image has a cat and a dog. If we want the “<class>” word embedding to focus on the cat and ignore the dog, we can prompt Stable Diffusion with “a photo of a <class> sitting on the right of a dog” when learning the “<class>” word embedding. If we generate images using an inference-time prompt “a photo of a <class>” the generations only have cats, showing the “<class>” word embedding has ignored the dog and focused on the cat in the original image.
https://drive.google.com/drive/folders/1lmr7e7qBZHwyGJyIkxt-0bUAxcn0934M?usp=sharing
We are updating the manuscript to discuss __Control Method 2__ in the appendix.

Controlling the style of what tokens learn can be done with Textual Inversion. We build on the Textual Inversion script released by HuggingFace, and by using `--learnable_property=style` the user can switch modes and cause tokens to learn style rather than object-based concepts.

__Control Method 3__: Finally, we can erase Stable Diffusion’s generation ability for specific concepts. Suppose we generate images of roads using the prompt “a photo of a <class>” and we realize that cars often show up in the generations, but our real data never has cars. We can erase the “car” concept and prevent Stable Diffusion from generating it, resulting in the following images. Additional examples for erasing concepts are in Figure 7 of the manuscript.
https://drive.google.com/drive/folders/1eDkXFKhpN94ZEhQtih97PdXsJtjzqKuz?usp=sharing
We discuss __Control Method 3__ in Section 6.1 of the manuscript.

Reviewer comment (Presentation):
Citations starting from second page are not in brackets like that in the first page. Would be nice to have consistency and keep them in brackets.

Author response:
__“Some nitpicks: Citations starting from second page are not in brackets like that in the first page. Would be nice to have consistency and keep them in brackets.”__

Thanks for catching this! We are updating the citation format in the manuscript to keep citations consistent by changing citations to brackets from the second page and after.