Paper: Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks (ICLR-2024)

Reviewer comment (Novelty):
This paper is featured with extensive empirical results. However, the core techniques in methodology (analysis and regularization of singular value spectrum) have been studied in existing works [e.g. Chen et al., 2019, Bardes et al. 2022], which may undermine the theoretical contribution of this work.

Author response:
First of all, as we discussed in the paper, the proposed method is indeed inspired by some related works [e.g. Chen et al., 2019, Bardes et al. 2022]. However, our method are not a direct application of them but designs them as specific regularizations, which are more straightforward and simpler to mitigate the noise in pre-training from our analysis.
- Second, the noisy model tuning approach is only one aspect of our contributions. Our main contribution lies more in being the first pioneering work to identify and analyze the effect of pre-training noisy datasets on downstream tasks. We think it is more important to bring up the question and analysis to the community, and more future work could be done.

Reviewer comment (Experiments):
On what scale the SVE and LSVR is computed? The entire dataset?

Author response:
The SVE and LSVR are computed on the entire test set of each downstream task.

Reviewer comment (Reproducibility):
In the loss function, are the regularizations computed per batch?

Author response:
Yes, we computed the regularization per batch, with a batch size of 32. 
We found the proposed regularization terms not sensitive to batch size, as long as it is not too small (e.g. 4). 
We provide a brief ablation of batch size in our method on JFT-300M semi-supervised pre-trained EfficientNet-B3 here:


|   Dataset   |   4   |   8   |   16  |   32  |   64  |  128  |
|:-----------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|
|  CIFAR-100  | 95.95 | 96.08 | 96.11 | 96.15 | 96.20 | 96.14 |
| Flowers-102 | 79.61 | 79.68 | 79.70 | 79.71 | 79.82 | 79.79 |

Reviewer comment (Presentation):
I find Figure 3 challenging to interpret. It consists of numerous data points for each configuration, lacking connecting lines. Consequently, I struggle to draw the conclusions reached by the authors based on this figure.

Author response:
Sorry for the confusion in Fig. 3, and thanks for your suggestions on making Fig. 3 more clear. 
In fact, we have tried to connect the points in Fig. 3, but it only makes it more difficult to visualize and interpret. 
Instead, we have provided a zoom-in version of Fig. 3 (a) and Fig. 3 (c) in Fig.10 of Appendix. 
We hope this zoom-in version is more interpretable.
In general, for the SVE of ID tasks, we first observe the increase of SVE as the noise goes up to 5%, with performance improvement for most of the tasks. 
Then we observe a further increase of SVE as noise increases to 10%, 20%, and 30%, and usually with performance drop.

Reviewer comment (Evaluation):
I've observed a potential contradiction between SVE and LSVR. For instance, in the case of two-dimensional features, [1.0, 0.0] exhibits the highest LSVR while having the lowest entropy. It would be beneficial if the authors could provide further clarification regarding this inconsistency.

Author response:
Thanks for mentioning the view of two-dimensional features. 
However, there might be a little misunderstanding here, and we would like to provide more discussion.
- First, we need to clarify that both SVE and LSVR are computed from the singular values of the features on the downstream test datasets of size $M$. When $M=1$ with a single point, the non-zero singular value would be trivial as a scalar of the magnitude of its unit basis vector (this can be simply verified by NumPy). In this case, any single point would present highest LSVR with lowest entropy, but has no relation to our observation and anlysis. 
- In general, we would require the size of the training samples and testing samples to be larger than 1 ($M > 1$), which is practical in real experiments.

Reviewer comment (Theory):
I would appreciate it if the authors could offer more detailed explanations as to why a slight amount of noise can benefit in-domain (ID) tasks. This is somewhat contradictory to the existing literature on learning with noisy labels, and additional insights would be valuable.

Author response:
Thanks for this good question. We also find it interesting and more discussions should be added. We share our insights here:

- Why a little noise can help?
    - Slight noise in pre-training makes the model learn to fit those noise, where the extra dimensions in the feature space are utilized. Thus, when applied this model to a downstream task, its extra dimensions in feature space provide better initialized and discriminative features (from higher dimenstional space), leading to better downstream accuracy. 
    - More noise in pre-training results in more dimensions to fit the noise. However, more dimensions to fit the noise is at the cost of less transferable features learned from the clean data and more nauce learned from the noise data, thus further increasing noise leads to worse performance. 
    - We do not think the better performance on downstream of slight noise in pre-training is related to better regularization in our pre-training experiments. This is similar in [1], where even heavy regularization techniques are used, the model can still fit perfectly to random noise. 

- The relation between noisy label learning and noisy model learning: This might look contradictory to learning with noisy labels, but our problem is indeed different from noisy label learning, so we give it a new name as *noisy model learning*. As shown in Fig. 4,
    - Noisy label learning: given a pre-trained model (or learn from scratch), the downstream datasets or any data we want to train the model contains noise.
    - Noisy model learning: given a pre-trained model trained on noisy datasets (which we cannot access, such as the black-box pre-training data of CLIP), we want to reduce such effect brought by the noisy pre-trained data on *any* downstream datasets.

That being said, our noisy model learning is naturally *complementary* to noisy label learning. We have an experiment in Sec. 4.3 to show that they can be used together.

Additionally, we can informally consider that noisy training data can lead to noisy weights, as in NoisyTune [2], where the authors also found introducing noise to pre-trained weights could help with downstream tasks, reaching similar conclusions as our study.

[1] Zhang et al. Understanding deep learning requires rethinking generalization. ICLR 2017.

[2] Wu et al. NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better. ACL 2022.

Reviewer comment (Evaluation):
In the paper, the authors claim that the proposed method enhances SVE and LSVR. However, Figure 5 (d) indicates that the proposed method does not yield superior LSVR compared to the LP and MLP models. Furthermore, it seems that as the noise ratio increases, LSVR does not drop significantly for all the settings.

Author response:
Thanks for mentioning this and we find Fig. 5 (d) might be difficult to interpret. 
Here, we provide more explanations. 
For Fig. 5 (d), the LSVR should be compared at the *trend* instead of the *absolute values* level. The expected LSVR across different noise ratios should be either increase or stay at the same level (it should not drop as in LP). 
Indeed, once adding the MLP, the LSVR are already improved and stays at the same level across different noise ratio. 
Adding the proposed regularization terms could further improve the LSVR of YFCC15M CLIP pre-trained as an increasing trend, while the ImageNet pre-trained stays at the same level. 
This is reflected in the OOD results in Table 1 and Table 2 too, where adding MLP already improves the OOD performance, and adding regularization terms could further boost the performance.

Reviewer comment (Evaluation):
I would like to point out that the improvements achieved by the proposed method appear to be relatively modest. According to the experimental results, the proposed method only demonstrates an approximately 1% improvement compared to the MLP model.

Author response:
The results we reported are the *average* performance of multiple datasets. 
Besides, the pre-trained backbones are frozen, where we only train the linear classifier or the MLP. 
Within this context, we do believe the improvement of approximately 1% compared to MLP tuning is significant.

Reviewer comment (Experiments):
For ResNet-50, it might be worthwhile to explore the feasibility of fine-tuning all layers rather than constraining the encoder. It would be valuable if the authors could conduct experiments to determine if the proposed loss function is effective in cases where all layers are fine-tuned.

Author response:
Thanks for your suggestion on full fine-tuning. We in general assume the pre-trained foundation models are not accessible in our study, which is mostly the case in practical large models. 
To answer your question, we further run the full fine-tuning on R50 and showed that the proposed method is still effective. 
Also note that in the studied datasets here, the plain fine-tuning (without our method) of R50 also present similar observation of ID performance to LP.
The ablation results of YFCC15M noisy pre-trained R50 are shown as follows:

| RenderedSST2 |   0%  |   5%  |  10%  |  20%  |  30%  |
|:------------:|:-----:|:-----:|:-----:|:-----:|:-----:|
|    Full FT   | 56.62 | 58.10 | 57.81 | 57.77 | 57.57 |
|     Ours     | 60.39 | 60.32 | 60.04 | 59.13 | 58.96 |
|              |       |       |       |       |       |
|  **Caltech101**  |   0%  |   5%  |  10%  |  20%  |  30%  |
|    Full FT   | 76.97 | 77.36 | 76.25 | 76.81 | 76.24 |
|     Ours     | 78.24 | 78.01 | 77.92 | 77.45 | 77.19 |
|              |       |       |       |       |       |
| **FGVCAirCraft** |   0%  |   5%  |  10%  |  20%  |  30%  |
|    Full FT   | 33.56 | 34.24 | 34.03 | 34.39 | 33.40 |
|     Ours     | 36.44 | 36.53 | 36.41 | 36.27 | 36.05 |

Reviewer comment (Writing):
Improvements are needed in the writing and presentation quality. Please check the Question section below.

Author response:
We sincerely thank you for your careful proofreading of our paper. 
We have fixed the typos and made the confusing statement more clear in the revised paper.

Reviewer comment (Evaluation):
The authors did not mention the limitations of their method and potential future work. The paper does not explore or discuss potential failure cases of the proposed methods. Understanding when and why the methods might fail is crucial for practical applications.

Author response:
Thanks for pointing this out. We briefly mentioned the limitation in the conclusion section (the last two sentences) due to the space limit. 
Now we discuss more limitation and potential failure here, which has been added in appendix C.1 and C.2. of the paper.

**Limitation**. The limitation mainly lies in our empirical study of the noise in pre-training. 
Due to the limited computing resources to extensively investigate different noise ratios by pre-training different models, we could only conduct our "motivational experiments" on ImageNet-1K and YFCC15M using ResNet-50 (which are already very expensive since we need to pre-train several ResNet-50 models on ImageNet or YFCC15M from scratch). 
Note that most of the SOTA foundation models are of much more parameters and are trained on much larger (and possibly inaccessible) datasets, which are beyond our reach to perform pre-training.
Additionally, the empirical experiments are limited to actual supervised pre-training.
While there are some existing studies showing the potential issue of noisy pre-training data, we could reveal more by scaling our experiments on larger backbones and datasets in the future.

**Potential Failure**. Yes, we agree that identifying failure cases indeed helps understand the problem better. We do observe some failure cases of the proposed methods. 
For example, from the results in Table 7, the proposed method falls short of LP on Caltech101 on almost all backbones we studied, while improving over MLP. 
Our hypothesis for the failure is that the SVD regularization term in the proposed method might need to optimize the top-$K$ singular values instead of just the largest one. The optimal value of $K$ might be different across datasets. However, setting $K=1$ can already achieve reasonable performance for most of the tasks, as shown in the results.

Reviewer comment (Theory):
Self-supervised pre-train does require external supervision. Does it mean those models will not suffer from the noise issue?

Author response:
Thanks for this wonderful question! We do believe self-supervised learning is worth further discussion:

- First of all, self-supervised learning does not require the explicit supervision of $Y$, but can be seen as supervised by the original input $X$, as we mentioned in footnote 2 of page 2 in our paper. In this context, it still requires supervision and the noise problem still exists: if $X$ is corrupted, which is usually the case in large-scale pre-training, it can be viewed as noise in self-supervision. 
- Second, despite some previous research [1] indicating that the self-supervised pre-training data contains some noise, we give more examples here. It turns out that corruption in $X$ can also have different formats in the actual data. For instance, in images, the corrupted $X$ can be poisoned or attacked images that do not contain natural content, and it will become noise for masked image modeling pre-training. Taking contrastive learning, MOCO [2], as another example. MOCO is a instance discrimination task and the negative pairs could potentially contain images that are similar to the positive anchor. This can also be viewed as noise. In auto-regressive language models such as GPT series, the noise would simply wrong or repetitive words in the sentence. 
- Third, our algorithm can naturally work for self-supervised pre-training models, as shown in Table 1 and 2, where most of these foundation models (ViT, CLIP, EfficientNet, GPT2, BERT, RoBERTa, and text-ada-002) are self-supervised pre-trained. Studying and understanding the corrupted $X$ (or noise) in self-supervised pre-training would also be very interesting and is our future direction. We believe the observation and conclusion from this work would scale and extend to the self-supervised learning scenarios because eventually, they fall into the same general formulation of noise in supervision.

In the future version of the paper, we will add more experiments and discussions about self-supervised learning.

[1] Yang et al. Xlnet: Generalized autoregressive pretraining for language understanding. NeurIPS 2019.
[2] Kaiming He, et al. Momentum Contrast for Unsupervised Visual Representation Learning

Reviewer comment (Experiments):
Does it proposed method generalize to self-supervised pre-trained models?

Author response:
Thanks for this wonderful question! We do believe self-supervised learning is worth further discussion:

- First of all, self-supervised learning does not require the explicit supervision of $Y$, but can be seen as supervised by the original input $X$, as we mentioned in footnote 2 of page 2 in our paper. In this context, it still requires supervision and the noise problem still exists: if $X$ is corrupted, which is usually the case in large-scale pre-training, it can be viewed as noise in self-supervision. 
- Second, despite some previous research [1] indicating that the self-supervised pre-training data contains some noise, we give more examples here. It turns out that corruption in $X$ can also have different formats in the actual data. For instance, in images, the corrupted $X$ can be poisoned or attacked images that do not contain natural content, and it will become noise for masked image modeling pre-training. Taking contrastive learning, MOCO [2], as another example. MOCO is a instance discrimination task and the negative pairs could potentially contain images that are similar to the positive anchor. This can also be viewed as noise. In auto-regressive language models such as GPT series, the noise would simply wrong or repetitive words in the sentence. 
- Third, our algorithm can naturally work for self-supervised pre-training models, as shown in Table 1 and 2, where most of these foundation models (ViT, CLIP, EfficientNet, GPT2, BERT, RoBERTa, and text-ada-002) are self-supervised pre-trained. Studying and understanding the corrupted $X$ (or noise) in self-supervised pre-training would also be very interesting and is our future direction. We believe the observation and conclusion from this work would scale and extend to the self-supervised learning scenarios because eventually, they fall into the same general formulation of noise in supervision.

In the future version of the paper, we will add more experiments and discussions about self-supervised learning.

[1] Yang et al. Xlnet: Generalized autoregressive pretraining for language understanding. NeurIPS 2019.
[2] Kaiming He, et al. Momentum Contrast for Unsupervised Visual Representation Learning

Reviewer comment (Reproducibility):
When you trained CLIP, did you train the text encoder together or you use a frozen text encoder?

Author response:
For CLIP pre-training, we strictly follow the training recipe of the Resnet-50 model in open clip, thus both the image encoder and text encoder are trained from scratch on synthetic noisy YFCC15M.
For CLIP fine-tuning (i.e., our noisy model tuning), we fixed all the encoders.

Reviewer comment (Writing):
"we empirically analyze the singular value spectrum of the pre-trained **the** feature space on downstream datasets" Typo: extra "the"

Author response:
Thanks for pointing this out. We have fixed this typo in the revision.

Reviewer comment (Writing):
Section 2.3, the authors should let or remind the readers what are M and D. They should be the number of samples and latent dimension size right?

Author response:
Thanks for this kind reminder. Here $M$ means the downstream dataset size (number of samples), and D denotes the feature dimensions.
We have added a footnote about this in the revised paper.

Reviewer comment (Theory):
"An initial increase in the spanning dimension of the feature space is beneficial to the discriminability on ID tasks. " The reason behind that is "the pre-trained feature extractor captures more structure in data" due to noise. But why more structure does not help OOD?

Author response:
The SVE (more structure in data) and LSVR should be looked at together to interpret this. 
As the noise ratio increases in the pre-training data, the model learns to fit more noise in pre-training. 
This will result in the pre-trained feature space with larger SVE, thus capturing more structure on downstream data.
Initially increase in SVE can help with ID downstreams because the nuance components related to the smallest singular values are not affected too much by slight noise. 
But as noise increases, the singular values of the nuance components also get increase, resulting in worse ID performance. 
Also, larger SVE comes at the cost of smaller LSVR, as the model uses more capacity/structure/dimension in feature space to fit noise, the generalizable components related to the top singular values are getting decreased, with smaller LSVR presented and worse OOD performance. 
The change in SVD spectrum is shown in the Fig.11 - Fig.14 in Appendix.