Paper: Do Generated Data Always Help Contrastive Learning? (ICLR-2024)

Reviewer comment (Experiments):
It would be better if the authors could compare AdaInf with the other existing data inflation methods for CL.

Author response:
Upon our survey of the literature, there are several works studied **only using generative models** for SL / CL [1,2,3], and one explored combining real and generated data for supervised learning [4]. As far as we see, **no existing work explores the mixed training strategy for self-supervised CL as done in ours**. Another key difference is that they often utilize generative models trained from **external datasets or supervision**, which leads to an unfair comparison. Instead, we make sure that only **the unsupervised pretraining dataset** is available for all settings, and we utilize the same dataset to train the generative model and perform contrastive learning.

**References:**

[1] Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. arXiv preprint arXiv:2306.00984, 2023.

[2] Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning. In ICLR, 2022.
[3] Victor Besnier, Himalaya Jain, Andrei Bursuc, Matthieu Cord, and Patrick Pérez. This dataset does not exist: Training models from generated images. In ICASSP, 2020.
[4] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip H. S. Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? In ICLR, 2023.

Reviewer comment (Reproducibility):
The description of AdaInf is in qualitative way. For example, it claims to use different weights on real and generated data and adopt milder data augmentation. However, how to get an appropriate weighted and the strengths of the data augmentation remains unknown.

Author response:
Since our theoretical and empirical discoveries are general and not limited to a particular reweighting or augmentation strategy, we use **AdaInf to refer to a general designing principle for data inflation, that we should adapt data reweighting and data augmentation to the chosen diffusion model and inflated data size.** The word “adaptive” here is used to highlight the data dependence. 

Nevertheless, since modern CL methods adopt multiple hand-designed augmentations — each with their own hyperparameters (eg cropping, color jittering) — it is hard to find a universal law for adjusting them, and it is a common practice to select a well-performing default configuration (e.g., SimCLR, MoCo, BYOL). Thus, we also resort to a default strategy that works well across multiple datasets. **The proposed AdaInf principle can provide guidelines for large-scale pretraining with generated data, and our simple AdaInf strategy can serve as a good starting point that delivers significant gains.** With large enough computing, one may further uncover a scaling law for AdaInf, and we leave it for future work. We have revised the definitions of AdaInf in  Sec 3.3 to avoid confusion.

Reviewer comment (Experiments):
It would be better if the authors could provide some experiments on more generative models like GAN.

Author response:
Sure we can. Following your suggestion, we have also added evaluations on GANs as another SOTA generative model, also with good generation quality. We find that similar to DDPM, vanilla inflation leads to worse performance, while AdaInf can bring significant improvements. Comparably, diffusion models with better generation quality (e.g., STF with lower FID) can achieve better accuracy. The results are added to **Appendix A**.

*Inflated contrastive learning with 1M generated images on CIFAR-10.*

| Model | FID | No Inflation | Vanilla Inflation | AdaInf |
| --- | --- | --- | --- | --- |
| StyleGAN2-ADA | 2.50 | 91.33 | 90.06 | 91.93 (+1.87%) |
| STF (diffusion) | 1.94 | 91.33 | 91.35 | 93.57 (+2.02%) |

Reviewer comment (Theory):
For equation (1), should there be a negative sign before the first term?

Author response:
Yes. We have fixed it in the revision.

Reviewer comment (Experiments):
Are there any experiments to show that the two tricks in AdaInf can also improve the performance of CL with data generated by more generative models like GANs?

Author response:
Please see **A3**.

Reviewer comment (Writing):
On page 3, there is a statement that better generative quality often requires larger models and/or slower sampling. What does the slower sampling mean?

Author response:
To obtain better generation quality in the diffusion model, one way is to adopt many intermediate denoising steps when drawing a sample (e.g., 1000 steps in DDPM), which slows the sampling speed significantly. We have added a short explanation here in the revision.

Reviewer comment (Presentation):
The purpose of Figure 4 is a little bit confusing to me. Why can this figure be used to show stronger augmentations create more overlap between different training samples?

Author response:
We are sorry for the confusion. We have revised this figure to be clearer. Now Figure 4b shows that weak augmentations still preserve the main image semantics, so different images cannot be effectively bridged together. Instead, **strong augmentations can create overlap views between different samples through common objects**, such as the wheels in the two images. In this way, stronger augmentations create more overlap between different training samples.

Reviewer comment (Experiments):
For the experiments of Figure 6, is the batch size for CL, CL with vanilla data augmentation and CL with AdaInf the same? If this is the case, does it mean the training cost is the same when the total training steps are the same for the three methods?

Author response:
For a fair analysis, we adopt the same training configuration (including batch size) for all experiments: CL, CL with vanilla data augmentation, and CL with AdaInf. Thus, as you said, the training cost is indeed the same when the total training steps are the same for the three methods.

Reviewer comment (Evaluation):
What is the pertaining cost of the diffusion model on the evaluated datasets?

Author response:
We conducted a time test for pretraining diffusion models with 4 NVIDIA GTX 3090 GPUs on CIFAR-10, and the total training time is shown below. We can see that diffusion models with better quality (EDM, STF) generally require longer training. In practice, since these models have public checkpoints, we do not need to train on our own for CIFAR-10.

| Model | DDPM | STF | EDM |
| --- | --- | --- | --- |
| Training Time | 71h | 87h | 91h |

Reviewer comment (Evaluation):
It would have been useful to have some more discussion on the issues of using synthetic data, e.g. bias.

Author response:
Indeed, using generated data will have downsides on generalization, particularly if the generated data is biased, e.g., a distribution gap $D(P_g,P_d)$. **Our Theorem 4.1 theoretically characterizes the influence of this distribution gap on downstream generalization, and shows that data reweighting can help mitigate this bias.** For concerns on social bias (if this is what you were referring to), we believe that generated data reflect the intrinsic bias in real data (if their distributions are close enough). Please let us know if there is more to clarify.

Reviewer comment (Theory):
Lemma 4.2 uses random sub-graph with probability \(p\) of selecting each edge. Is random edge selection reflective of subsampling data? Random node selection seems more appropriate.

Author response:
We note that in the InfoNCE loss (Eq. 1), the positive samples are drawn as a pair from the data, which represents an edge in the graph. In other words, during contrastive learning, the influence of inflation size is reflected in the form of sampled edges. Thus, we believe that it also makes sense to consider edge subsampling for modeling their difference. Another important reason is that upon our research, we only found an existing result of Chung and Horn (2007) guaranteeing the spectral gap on edge subsampling. If you happen to know any node subsampling results, we would be happy to include them as well.

Reviewer comment (Theory):
Thm 4.1 argues about connectivity and non-zero \(\lambda_{k+1}\) for the training set, but Saunshi et al. 2022 show lack of connectivity in the training set for standard augmentations, implying \(\lambda_{k+1}=0\). This would make the bound in Theorem 4.1 vacuous without any . Does that contradict the claims? The results in HaoChen et al. are non-vacuous because that uses the eigenvalue of the population distribution in the bound rather than eigenvalue of training data.

Author response:
This is also a very good question! Indeed, [1] argued that finite data may suffer from zero augmentation overlap and thus $\lambda_{k+1}=0$. If it is really true, contrastive learning should have bad generalization (proven in [2]). However, in practice, it is not. [1] attributed this to the inductive bias of neural networks that implicitly bring augmentation overlap, and [3] established a formal guarantee on it. Therefore, with the help of architectural inductive bias, we will not have zero augmentation overlap even with finite training data. Since incorporating inductive bias is rather independent of the main interest of this work, we omit this part for simplicity. For completeness, we have added a footnote to explain it on page 6.

 **References:**

[1] Saunshi et al. Understanding Contrastive Learning Requires Incorporating Inductive Biases. In ICML. 2022.

[2] Wang et al. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In ICLR, 2022.

[3] Haochen and Ma. A Theoretical Study of Inductive Biases in Contrastive Learning. In ICLR. 2023.

Reviewer comment (Writing):
The preliminary section is not well-written.

Author response:
**A1**. Thanks for your careful reading. We have carefully revised the writing and fixed the typos that you mentioned (more details below). Please let us know if there is more to modify.

Reviewer comment (Novelty):
There is a concern about the generality of AdaInf. It seems that the method is hand-crafted rather than adaptive.

Author response:
**A2**. Since our theoretical and empirical discoveries are general and not limited to a particular reweighting or augmentation strategy, we use **AdaInf to refer to a general designing principle for data inflation, that we should adapt data reweighting and data augmentation to the chosen diffusion model and inflated data size.** The word “adaptive” here is used to highlight the data dependence. 

Nevertheless, since modern CL methods adopt multiple hand-designed augmentations — each with their own hyperparameters (e.g., cropping, color jittering) — it is hard to find a universal law for adjusting them, and it is a common practice to select a well-performing default configuration (e.g., SimCLR, MoCo, BYOL). Thus, we also resort to a default strategy that works well across multiple datasets. **The proposed AdaInf principle can provide guidelines for large-scale pretraining with generated data, and our simple AdaInf strategy can serve as a good starting point that delivers significant gains.** With large enough computing, one may further uncover a scaling law for AdaInf, and we leave it for future work. We have revised the definitions of AdaInf in Sec 3.3 to avoid confusion.

We also acknowledge that the name Adaptive Inflation could be a little confusing despite general principles, we only offer a default choice in practice. We would like to know which alternative name you may find suitable.

Reviewer comment (Novelty):
There is a lack of motivation for considering diffusion models in this paper. All theoretical results can be generalized to any generative model.

Author response:
**A16.** We want to note that the data inflation strategy explored in this work is **model-agnostic**, and **we can use *any generative models* that have good generation quality, in theory, a lower distribution gap $D_{\rm TV}(P_d,P_g)$.** We pick diffusion models only because it has SOTA generation quality to date. As shown in the revision **(Appendix A)**, our AdaInf also **delivers significant improvements over vanilla inflation with StyleGAN2 (quoted below)**. This model-agnostic property makes AdaInf more generally applicable than strategies specifically designed for one class of models.

Accordingly, our theory is also general and not specified for diffusion models. Thus, the theory also fits for our practice as well.

*Inflated contrastive learning with 1M generated images on CIFAR-10.*

| Model | FID | No Inflation | Vanilla Inflation | AdaInf |
| --- | --- | --- | --- | --- |
| StyleGAN2-ADA | 2.50 | 91.33 | 90.06 | **91.93 (+1.87%)** |
| STF (diffusion) | 1.94 | 91.33 | 91.35 | **93.57 (+2.02%)** |

Reviewer comment (Writing):
Some notation and experiment results are not consistent.

Author response:
**A3**. Thank you for your detailed reading and kind suggestions. We have fixed the typos and writing errors in the revision (details below).

Reviewer comment (Theory):
Eq. (1) is wrong. In addition, the authors should cite this paper for Eq (1): “Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere”.

Author response:
**A4.** Thanks. We have fixed it.

Reviewer comment (Theory):
Eq. (2) is also wrong. Again, the citations are not chosen well. Ho et. al 2020 should be a better reference here.

Author response:
**A5**. Thanks. We have fixed it.

Reviewer comment (Experiments):
Why do we need to sample 1M data while CIFAR-10 has only 60k samples? Speaking of which, instead of increasing $\beta$ we can decrease $|\mathcal{D}_g|$.

Author response:
**A6**. This is because using more generated data can bring diverse input and facilitate generation. We have shown the gain of using more generated data in **Figure 8**. As quoted below, decreasing  $|\mathcal{D}_g|$ from 1M to 0.2M will lead to worse performance (about -0.5%). **Even without reweighting (first row)**, we need to inflate **at least 0.5M** generated samples (10x CIFAR-10) to attain the best performance of ****93.12%. With reweighting, we can allow to use a larger 1M dataset, and **attain a better performance of 93.57%**. Based on this evidence, we adopt 1M dataset + reweighting as our default strategy.

| Generated data　size | 0.2M | 0.5M | 1M | 2M | 5M |
| --- | --- | --- | --- | --- | --- |
| No replication | 92.84 | 93.12 | 91.35 | 92.42 | 92.14 |
| 10:1 Replication | 93.09 | 92.97 | 93.57 | 93.39 | 93.54 |

Reviewer comment (Evaluation):
Section 3.2: Are there any intuitions or reasonings why larger data size needs weaker data augmentation? Or do generated images need weaker data augmentation and they dominate the inflated data set?

Author response:
**A8**. Yes! To understand it more thoroughly, we give a theoretical characterization on this phenomenon in Sec 4.2, and the intuitive logic is as follows. As pointed out by many papers [1,2], contrastive learning generally relies on **strong data augmentation to create connectivity** between samples, in order to get good downstream generalization. However, strong augmentation also has downsides as it **introduces large labeling errors** with distribution shifts. To resolve this issue, **we wish to use as weak augmentations as possible if only we could have the same level of connectivity,** while data inflation is exactly such a technique that can improve graph connectivity by enlarging the training set. Therefore, as pointed out in Sec 4.2, data inflation and data augmentation have complementary effects. With inflation sharing the burden of graph connectivity, we can use weaker augmentation to minimize its downsides (labeling error) and gain better downstream performance. 

> Or do generated images need weaker data augmentation and they dominate the inflated data set?
> 

This seems a common misunderstanding of our method. Although it is an intuitive logic, it is **not exactly true**. Fig 3a shows that **subsampling real data (Half CIFAR-10) would also require stronger data augmentation (min crop 0.02) to attain the optimal performance.** In turn, it means that scaling from Half CIFAR-10 and CIFAR-10 **with real data** also requires weaker augmentation (min crop 0.08). Therefore, it is NOT the real-synthetic distribution gap that requires us to adjust data augmentation**,** but the inflated data size. ****Indeed, ****our theory above suggests that even if the synthetic data are perfect ($P_d=P_g$), it is also helpful to decrease data augmentation strength under more data inflation, which aligns with the observation in Fig 3a.

**References:**

[1] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In NeurIPS, 2021.

[2] Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In ICLR, 2022.

Reviewer comment (Experiments):
Section 3.3: How to choose the correct strength of data augmentation in AdaInf? The word "adaptive" seems misleading here because the hyperparameters are chosen manually rather than auto adjusted or learned.

Author response:
**A2**. Since our theoretical and empirical discoveries are general and not limited to a particular reweighting or augmentation strategy, we use **AdaInf to refer to a general designing principle for data inflation, that we should adapt data reweighting and data augmentation to the chosen diffusion model and inflated data size.** The word “adaptive” here is used to highlight the data dependence. 

Nevertheless, since modern CL methods adopt multiple hand-designed augmentations — each with their own hyperparameters (e.g., cropping, color jittering) — it is hard to find a universal law for adjusting them, and it is a common practice to select a well-performing default configuration (e.g., SimCLR, MoCo, BYOL). Thus, we also resort to a default strategy that works well across multiple datasets. **The proposed AdaInf principle can provide guidelines for large-scale pretraining with generated data, and our simple AdaInf strategy can serve as a good starting point that delivers significant gains.** With large enough computing, one may further uncover a scaling law for AdaInf, and we leave it for future work. We have revised the definitions of AdaInf in Sec 3.3 to avoid confusion.

We also acknowledge that the name Adaptive Inflation could be a little confusing despite general principles, we only offer a default choice in practice. We would like to know which alternative name you may find suitable.

Reviewer comment (Experiments):
Section 5.1: Why is the min scale of random resized cropping (0.2) different from the optimal value (0.3) in Figure 3?

Author response:
**A10**. In Figure 3, we do not add data reweighting (1:1) for ablating their effects, while we adopt 10:1 reweighting in Table 3 for a comprehensive analysis, so there is a slight difference. Overall, 0.2 and 0.3 perform similarly (gap $\leq$ 0.3%) in two settings.

Reviewer comment (Presentation):
The result in Table 3 is slightly different from that in Figure 3 where 0.3 yields the best performance.

Author response:
**A10**. In Figure 3, we do not add data reweighting (1:1) for ablating their effects, while we adopt 10:1 reweighting in Table 3 for a comprehensive analysis, so there is a slight difference. Overall, 0.2 and 0.3 perform similarly (gap $\leq$ 0.3%) in two settings.

Reviewer comment (Experiments):
Figure 8: Is the replication N fixed to 10? Again, the question is whether N = 10 is universal.

Author response:
**A11**. Yes, we adopt 10:1 across all experiments in the experiment section and appendix section. For the latter question, please see **A7**.

Reviewer comment (Theory):
The labeling error definition seems weird. It is better to define it as Eq (8). Then, the term $\phi^y$ in Eq (6)  is less than $2 \alpha$. If the definition is kept as in Theorem 4.1, we have $\phi^y = \alpha$. Note that $\alpha$ in HaoChen et al. 2021a has a different meaning.

Author response:
**A12**. Thanks for pointing it out! Indeed, we made a typo here, and it would be better to define $\alpha$ as Eq. 8, which is consistent with Haochen et al. We have fixed this in the revision.

Reviewer comment (Writing):
The usage of $B^\star$ can be safely removed.

Author response:
**A13**. Here, $B^*$ denotes the optimal weight of the linear classifier. We have revised the formulations in Sec 4 to be clearer.

Reviewer comment (Writing):
N is used for replication times in Section 3.1 but later used in Section 4.1 with different meanings. What is the meaning of N in Section 4.1?

Author response:
Yes. We abuse the notations here. We change $N$ in Sec 4.1 to $n$, where it denotes the number of training data.

Reviewer comment (Writing):
Eq (6), what is $\mathcal{X}$?

Author response:
It denotes the set of all augmented data. We have added definitions.

Reviewer comment (Writing):
Is $w_{x x'}$ defined as $A_{x, x'}$ in the main text?

Author response:
Yes. We have modified them to be $A_{xx'}$ to be consistent.

Reviewer comment (Writing):
Overall, the notation is similar to HaoChen et al. 2021a. However, the way the authors use them is not good and sufficient.

Author response:
Thanks for pointing it out! We have revised them in the revision.

Reviewer comment (Experiments):
Theory and empirical experiments are not well connected. Leveraging synthetic data requires defining how it will be mixed with real data. As image statistics of synthetic data might be different from real data, the existing augmentations applied to contrastive learning might be suboptimal — hence, re-tuning both aspects of the model could help boost performance. A better way to connect theory and empirical observations could be interesting, but in the current outline of the paper, both tracks seem pretty independent.

Author response:
**Q1.** Theory and empirical experiments are not well connected, in my opinion (I would be happy to get corrected on this during the rebuttal).

**A1. First**, let us explain how we connect theory and practice. After we discover the two AdaInf strategies in Sec 3, we provide a rigorous theoretical justification in Sec 4. In particular,  Theorem 4.1 establishes **a theoretical bound that takes the reweighting-related terms ($\beta$ and $D_{\rm TV}(P_d,P_g)$) and the augmentation-related terms ($\alpha$ and $\lambda_{k+1}$) into consideration**. We then provide **a detailed explanation in Sec 4.2** to show that the theoretical bounds imply the empirical phenomena that we observed. To make it clearer, we now explicitly link each explanation to each strategy in the revision (see page 6). 

**Second**, we want to point out that there are some misunderstandings in your descriptions below.

> Here is how I read the paper: Leveraging synthetic data requires to define how it will be mixed with real data. Also, as image statistics of synthetic data might be different from real data, the existing augmentations applied to contrastive learning might be suboptiomal --- hence, re-tuning both aspects of the model could help to boost performance.
> 

We highlight that in our theoretical analysis, the distribution gap is the cause of the **data reweighting strategy** (because reweighting can provably close the gap)**, but NOT the cause of the data augmentation** strategy. Instead, **even if the synthetic data are perfect ($P_d=P_g$)**, it is also helpful to decrease data augmentation strength under more data inflation. Fig 3a gives the evidence by showing that **subsampled real data (Half CIFAR-10) would also require stronger data augmentation to attain the optimal performance, and vice versa.** We theoretically justify this phenomenon with three paragraphs in Sec 3, and the logic is as follows. Since data inflation enhances the connectivity of augmentation graph, we can achieve a smaller labeling error under the same connectivity, so the optimal augmentation strength will become smaller.  

So, to be precise, the takeaway of our theoretical analysis is that, ***for data inflation, we could adjust reweighting coefficients to mitigate the distribution gap, and adjust augmentation strength to fully unleash the benefits of enlarged data size**.* In this way, we can gain more from generated data and benefit downstream tasks a lot.

Therefore, we believe that **our empirical findings can be sufficiently explained by our theoretical analysis in Sec 4.2.** As noted by Reviewer gCqn, ***“the results would greatly help in our understanding of contrastive learning both theoretically and empirically.**”* Hope this explanation could address your concerns. Please let us know if there is more to clarify.

Reviewer comment (Experiments):
The empirical results are fairly small scale with limited evaluation of different generative models. Many evaluations are performed on CIFAR-10, where one would expect the possibility to run extensive experiments and analysis of various factors. To make the statements in the paper more convincing, it would help to extend these experiments, compute error bars, study the influence of other parameters, and extend the most promising models towards full ImageNet-scale.

Author response:
**Q2.** Another weakness are the empirical results. These results are fairly small scale with limited evaluation of different generative models. To make the statements in the paper more convincing, it would help to extend these experiments, compute error bars, study the influence of other parameters, and extend the most promising models towards full ImageNet-scale.

**A2.** In our work, we mainly explore the effect of **augmenting original data with a lot more generated data (so we call it data inflation)**. For example, we augment CIFAR-10 of 50k samples with 1M generated samples (20x inflation). It is well-known that generating high-quality samples from diffusion models is very computationally expensive. For now, the largest dataset we are able to cope with is Tiny-ImageNet (see Table 1), where training a diffusion model and generating 1M 64x64 samples take us two whole weeks. And with limit of computation, we have tried our best to evaluate the proposed method on diverse datasets (CIFAR-10, CIFAR-100, Tiny ImageNet) **with different data sizes (Tiny Imagenet is of 1/10 ImageNet size), different numbers of classes (10 & 100), and different resolutions (32x32, 64x64)**. Thus, we believe that our experiments can also justify the generality of AdaInf across different kinds of data.

**Error bars.** **We have added the error bars to the reported results following your suggestions.** From Table 1, we can see that AdaInf indeed significantly improves over the vanilla inflation.

**Generative models.**  We have also added evaluations on GANs as another SOTA generative model, also with good generation quality. We find that similar to DDPM, vanilla inflation leads to worse performance, while AdaInf can bring significant improvements. Comparably, diffusion models with better generation quality (e.g., STF with lower FID) can achieve better accuracy. The results are added to **Appendix A**.

*Inflated contrastive learning with 1M generated images on CIFAR-10.*

| Model | FID | No Inflation | Vanilla Inflation | AdaInf |
| --- | --- | --- | --- | --- |
| StyleGAN2-ADA | 2.50 | 91.33 | 90.06 | 91.93 (+1.87%) |
| STF (diffusion) | 1.94 | 91.33 | 91.35 | 93.57 (+2.02%) |

**Large-scale study.** Studying inflation on large-scale data, e.g., inflating ImageNet-1k (1M size) by 20x, would require **huge computing** (generating 1M ImageNet images (each 2s) requires 23 days, and 20M requires 460 days) which is far beyond our capability. Within the limit of rebuttal time, we propose to validate our method on ImageNet by studying two subsets: ImageNet-100 (1300 images for each class) and 10% ImageNet100 (randomly sampled 10% off from each class of ImageNet100, 130 images for each class). According to our analysis, a larger dataset (ImageNet-100) would require a stronger augmentation to attain the optimal performance.

**Results.** As shown below, fewer data (10% ImageNet-100) attain better performance with stronger augmentation (0.04 RRC min scale), while full ImageNet-100 performs better with weaker augmentation (0.08 RRC min scale). It aligns very well with our theoretical and empirical observation on a large-scale and high-resolution experiment: a larger data size (e.g., with inflation) requires a weaker augmentation to attain the optimal performance. Thus, our conclusion also holds for ImageNet-scale datasets.

| RRC min scale | 10% ImageNet100 | ImageNet100 |
| --- | --- | --- |
| 0.04 | **47.24** | 71.26 |
| 0.08 | 45.34 | **72.76** |

Reviewer comment (Evaluation):
The word "significantly" is used at different places in the paper without necessarily being backed up by a statistical test. The authors should check the usage and remove it where not supported by statistical evidence. Running multiple seeds (at least 5) for the models and reporting error bars on every CIFAR-scale experiment is strongly encouraged.

Author response:
**Q4.** "significantly" is used at different places in the paper. Please check the usage, and remove where it is not backed up by a statistical test.

**A4.** Due to the limit of time to reproduce all experiments, we run 3 times for each experiment in Table 1. It can be seen that AdaInf can indeed obtain statistically significant improvement over the baseline. We have also revised our writing accordingly following your suggestion.

Reviewer comment (Evaluation):
In Table 1, the FID scores of the used generative models on the respective dataset should be added, as this seems to have a major influence on the experiment outcome.

Author response:
**Q5.** Table 1, the FID scores of the used generative models on the respective dataset should be added, this seems to be a major influence on experiment outcome.

**A5.** We note that the default diffusion models and their FID scores used in Table 1 are introduced in the Setup paragraph in the beginning of Section 5, as quoted below. 

> *By default, we use 1M synthetic data for CIFAR-10 and CIFAR-100 generated by a high-quality diffusion model STF (Xu et al., 2023) (FIDs are 1.94 (CIFAR-10) and 3.14 (CIFAR-100)). Due to the limit of computation resource, we adopt DDPM (18.61 FID) for Tiny ImageNet. These diffusion models are unconditional since we only assume access to unlabeled data for pretraining.*
> 

We have also added it to the caption of Table 1 for better clarity in the revision.

Reviewer comment (Writing):
In Theorem 3.1, clarify the abbreviation of total variation (TV) where it first appears in the text if it was not introduced before.

Author response:
**Q6.** Theorem 3.1, was the total variation introduced before? If not, clarify the abbreviation (TV) where it first appears in the text.

**A6.** Thanks. We have added definitions here.

Reviewer comment (Writing):
The notation in Section 4.1 should be double-checked. For example, $\mathcal{\overline{X}} = {\overline{x}}$, is that correct? Or $D_{xx}$.

Author response:
Q8. The notation in 4.1 should be double checked. E.g., $\bar{\mathcal{X}}=\{\bar x\}$ is that correct? or $D_{xx}$.

**A8.** Here, we want to say that $\bar{\mathcal{X}}$ are composed of samples like $\bar x$. For $D_{xx}$, as in [1], $x$ is used as an index, so $D_{x,x}$ is the $(x,x)$-th element of $D$. We add descriptions to avoid confusion.

**Reference:**

[1] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In NeurIPS, 2021.

Reviewer comment (Writing):
In Section 4.2, "an downstream" should be corrected to "a downstream".

Author response:
**Q9.** Sec 4.2, "an downstream" -> "a downstream"

**A9.** Thank you and we have fixed it.