Paper: AdaFlood: Adaptive Flood Regularization (ICLR-2024)

Reviewer comment (Novelty):
The limited novelty of the proposed paper. The idea of using adaptive weighting for model training is widely used in existing machine learning areas, the more relevant ones can be curriculum learning and uncertainty learning. Moreover, as mentioned by the author in Related Work, using a held-out dataset to condition the training of the primary model is a well-known strategy in machine learning, e.g., in meta-learning, batch or data selection, neural architecture search, etc. In the present paper, the author integrates the adaptive flooding level for flooding regularization, which is just a naive combination of the existing methods. Thus the novelty of the present paper is limited.

Author response:
Indeed, adaptive weighting and held-out datasets are each commonly used techniques in machine learning, as we mentioned in the paper. We are not aware, however, of any time they have been used together for something resembling the difficulty-aware regularization setting of our work. The vast majority of research is based on applying new combinations of existing tools in new settings; our paper is no different in this regard. We think this is a strength, and not a weakness: our method is built on well-tested tools that have individually proven effective in diverse fields, suggesting that our proposal is built on a solid foundation, and with the ability to share techniques and understanding of those tools across application areas.

If we missed any closely related work in those areas, however, we are eager to compare them to our approach and cite them.

Reviewer comment (Theory):
Although the author proposes a correction function to account for the imperfections of the prediction of the auxiliary model, the proposed correction function is very heuristic, and there is no theoretical analysis to reason about the soundness of this proposed function. The efficient training for the auxiliary networks also has a similar issue.

Author response:
We agree that the correction function is heuristic, but it is a very natural and intuitive way of correcting the predictions from auxiliary networks. It is very easy to implement, and it performs well (as we demonstrated in the multiple experiments).

As the reviewer pointed out, it would be indeed better if we had solid theoretical background for every component, but in practice this is extremely rare, especially for deep learning. The influence of even the most basic regularizer, weight decay, is still a subject of very active research among deep learning theorists (e.g. [1] from earlier this year). We think that our support for AdaFlood in Proposition 1 establishes a strong theoretical baseline, and the discussion in the following paragraph gives sufficient initial intuition about the behavior of the correction function.

For the efficient training of auxiliary networks, we expanded on the motivation in highly-overparameterized regimes in Appendix E of the revised paper. In summary, in the regime where networks are well-explained by their neural tangent kernels (NTKs), held-out predictions by an auxiliary network trained from scratch perfectly agree with predictions from a network trained according to our efficient algorithm. This is because the empirical NTK of the penultimate layer is identical for the two networks. Please refer to Appendix E for further details.

[1] Joseph Shenouda, Rahul Parhi, Kangwook Lee, and Robert D. Nowak. Vector-Valued Variation Spaces and Width Bounds for DNNs: Insights on Weight Decay Regularization. arXiv:2305.16534.

Reviewer comment (Experiments):
The significance of the proposed method. Although the reviewer acknowledges the effort the author has made to validate their proposed method by the different modality of data, the data, and model chosen by the author is still not convincing. For example, though the SVHN, CIFAR10, and CIFAR100 are commonly used classification datasets, their complexity, scale, and reality are much less than ImageNet and more recent datasets like LAION. Although the review acknowledges that the author has stated in the last two lines of Page 7 that the result of the proposed method is not significant, this still gives us a feeling that the proposed method will not significantly improve the DNN training to avoid overfitting given so marginal improvement on Table 2. It is better for the author to consider a more realistic and decent scale of the dataset to conduct the experiment to show that the proposed simple method can achieve much better performance than state-of-the-art (SOTA), otherwise, it is hard to demonstrate the significance of the proposed method.

Author response:
We shared some results on larger datasets in the common response above, showing that AdaFlood still generally outperforms the other flooding methods (with even bigger gains in noisy-label settings). We are currently running experiments on the full ImageNet, and will add them to the revised paper once they are complete.

We’d like to emphasize that, while we haven’t shown results on the largest image classification datasets, we have demonstrated the versatility of our method across a variety of tasks including density estimation, regression, and classification, on settings including asynchronous time sequences, images, tabular, and text data.

We first want to highlight that in the last two lines of page 7, we were not saying that our method’s overall results were not significant – if that were the case, we wouldn’t have submitted the paper. Instead, we were saying that on _SVHN in particular_, AdaFlood performs about the same as previous flooding methods. (Performance is probably basically saturated for this class of model on SVHN.) AdaFlood does give nontrivial improvements in accuracy and calibration on CIFAR10 and CIFAR100. In the results in the common response above, we also see even larger improvements on ImageNet100 (probably because its labels are inherently noisier). Notice in particular that the improvement of AdaFlood over iFlood is larger than the improvement of iFlood over Flood, or of Flood over baseline models.

Moreover, it is far more robust to noise on these datasets than competitor methods. Outside of image classification, it also sees improvements in both original and noisy versions of many datasets across asynchronous time sequence density estimation, text classification, tabular regression.

Reviewer comment (Experiments):
The efficient training strategy is mainly randomly re-initializing the last few layers of the model and then re-training with the relevant fold held out. How to define the term "few"? What's the principle here to choose those layers? The reviewer noticed that the author said ResNet-18 is a large model and used it in the experiment. Can the proposed method still work effectively when we use ViT for image classification and Transformer for text modality? Will this efficient training strategy still work?

Author response:
Recall that the purpose of having auxiliary networks is to measure difficulty of each sample. Here, we avoid a model seeing the sample that we measure difficulty on, in training time. With this motivation, random initialization of last few layers of efficient training of auxiliary networks is to forget information about the samples we measure difficulties on.

According to the conditions provided by [1] for forgetting, randomly initializing only the last layer satisfies these conditions; initializing more layers, of course, does as well. Also, in [2] as the title implies, re-training only the last layer is sufficient to remove spurious correlations that are more specific to individual data points. These results suggest that re-initializing at least one layer should be sufficient.

Then, the question is how different predictions will be depending on the number of layers we randomly initialize. [3] provides a user guide for fine-tuning, depending on the gap between pre-training and adaptation steps. According to [3], our fine-tuning falls in the category where the required feature update is tiny (since we fine-tune on a subset of the training set), which means whether we initialize only the last layer or last few layers, it won’t change the features too much at convergence.

In addition to this guidance from prior work, we explore the question of how many layers to reinitialize in Section 4.5. We can see in those results that the performance of AdaFlood stays essentially the same (< 0.11%) if we reinitialize only the last layer, layer 4 plus the last, or layers 3 and 4 plus the last. Thus, reinitializing only the last layer gives the fastest method at essentially the same performance.
 
Based on this, we agree that it will be clearer if we are more specific when presenting the method, and so changed the description in the revised paper from “the last few layers” to “the last layer.”
 
As suggested, we also verified that the fine-tuning variant works for ViT. We took the ViT implementation from a publicly available repo (https://github.com/kentaroy47/vision-transformers-cifar10). To train multiple auxiliary networks, we split CIFAR10 into 20 folds, and trained 20 ViT models for $500$ epochs each as described in Sec 3.2. For the fine-tuning variant, we train a ViT model on the full CIFAR10, and then fine-tuned on 20 folds as described in Section 3.3. After collecting losses for $50,000$ training samples from both multiple and fine-tuned versions, we obtained a Spearman (rank) correlation between these predictions of 0.61, essentially the same as the correlation observed for ResNets. Thus, the efficient variant provides substantial signal to the “correct” adaptive flood levels with ViT as well.

Lastly, we want to emphasize that the efficient training of auxiliary networks saves computational cost in computing the adaptive flood levels, but it is not a “must-have” component for the proposed AdaFlood method.

[1] Hattie Zhou, Ankit Vani, Hugo Larochelle, Aaron Courville. “Fortuitous forgetting in connectionist networks”. ICLR 2022.

[2] Polina Kirichenko, Pavel Izmailov, Andrew Gordon Wilson. “Last layer re-training is sufficientfor robustness to spurious correlations”. ICLR, 2023.

[3] Yi Ren, Shangmin Guo, Wonho Bae, Danica J Sutherland. “How to Prepare Your Task Head for Finetuning”. ICLR 2023.

Reviewer comment (Theory):
The author states in the third paragraph of page 5 that: "This procedure is justified by recent understanding in the field that in typical settings, a single data point only loosely influence the early layers of a network." The author should provide explicit references to support this claim, e.g. what recent understanding does the author refer to? How do they relate to the proposed method?

Author response:
Followed by the sentence the reviewer mentioned, we explained that “In highly over-parameterized settings (the “kernel regime”) where neural tangent kernel (NTK) theory is a good approximation to the training of $f^{aux}$ [1], re-initializing the last layer would completely remove the effect of $x_i$ on the model.” As mentioned for a previous question, we have added a more thorough discussion of this point to Appendix E.

For further intuition about loose influence on the early layers, here is our reasoning:

According to [1], in the NTK regime where the width of a neural network is infinite, and loss function is the least-square loss, the prediction from the neural network and ridgeless kernel regression using NTK are equal in expectation. In other words, the prediction of a neural network $f$ on a test sample $\tilde{x}$, say $f(\tilde{x})$, is governed by a fixed kernel called NTK. For ridgeless kernel regression, adding data points does not change the kernel function but it does change the prediction by adding a row and column to the kernel matrix. Conversely, if we remove $x_i$ from the training set, $x_i$ will not affect on inference at all. 

Even if a neural network does not have infinite-width, it is shown that overparameterized neural networks can converge linearly to zero training loss with their parameters hardly change [2] i.e., the change in the Jacobian matrix is very small. Hence, in this regime, a single data point only loosely influence training.

In addition, [3] states that “…the gradient tends to get smaller as we move backward through the hidden layers. This means that neurons in the earlier layers learn much more slowly than neurons in later layers.” Combined with the previous arguments, we hope that our statement for “a single data point only loosely influence the early layers of a network” makes sense now.

The paragraph where the we stated “a single data point only loosely influence the early layers of a network” was about justifying that re-initializing the last (few) layers of a pre-trained neural network is enough to forget about a data point, and so, it is reasonable to compute the difficulty on the point with a fine-tuned model. In the paragraph, we divided the parameter regimes as,

- NTK regime — Re-initializing the last layer would completely remove the effect of $x_i$.
- Non-NTK (more realistic) regime — Last layer re-training retains “core” features and removes “spurious” ones that are more specific to individual data points [4]

If the reviewer thinks it is still unclear, please let us know. We are eager to discuss further and make the paper clearer.
 
 
 [1] Arthur Jacot, Franck Gabriel, and Clement Hongler. “Neural tangent kernel: Convergence and generalization in neural networks”, NeurIPS 2018.
 
 [2]Chizat, Lenaic and Oyallon, Edouard and Bach, Francis. “On lazy training in differentiable programming”, NeurIPS 2019

[3] Michael A. Nielsen, "Neural Networks and Deep Learning", Determination Press, 2015

[4] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. “Last layer re-training is sufficientfor robustness to spurious correlations”, ICLR, 2023.

Reviewer comment (Presentation):
The `Unreg.` in Figure 3 can not be seen. The author may consider another format of the line for better visualization.

Author response:
As described in the Results paragraph in Section 4.3, we visualized $\Delta Acc$, defined as the improvement in accuracy of a flood method over the accuracy of `Unreg.`  We also provided the baseline `Unreg.` accuracy in numbers, below the $\Delta Acc$ lines. Visualizing in this way is helpful since the change in inherent difficulty between noise levels is much larger than the change in performance of different methods, and a plain accuracy plot (now shown as Figure 10 in the appendix) is overwhelmed by the difference in noise levels. We will consider other visualization formats, and would be happy to hear any suggestions that would be clearer while still making the difference between methods visible.

Reviewer comment (Theory):
The method roots in a multi-view assumption, relying on auxiliary networks as different views to tailor their fitness. However, this assumption’s fragility surfaces in scenarios where auxiliary networks falter, casting doubts over the accurate estimation of flood levels.

Author response:
We first want to emphasize that we do not use a “multi-view assumption” in the sense typically used in the field. We use multiple auxiliary networks, but each one is used for a subset of the data, and “sees” the exact same features of the data – unlike multi-view learning, which uses multiple feature sets for the same underlying data points.

The motivation of using multiple auxiliary models is simply that model’s training error on a particular data point is unlikely to be a good estimate of that data point’s inherent difficulty; instead, we use a cross-validation scheme to evaluate the held-out error for each data point.

As to when “auxiliary networks falter”, we agree that utilizing auxiliary networks may harm training if auxiliary networks do not provide reasonably good predictions. Our correction function, introduced in Section 3.2, helps to mitigate this effect. As a reminder, the auxiliary network is not required to provide especially accurate predictions, but just to provide a useful-enough estimate of the difference in difficulty between samples to serve as an effective regularizer.

Reviewer comment (Experiments):
A confined exploration limited to toy datasets raises eyebrows, leaving unanswered questions regarding AdaFlood’s effectiveness when unleashed on larger, real-world, and more challenging benchmarks.

Author response:
We disagree with characterizing the datasets used in this submission as “toy datasets,” and many of them are derived from real-world problems. For example, the Stack Overflow dataset contains 6,633 sequences with 480,414 events where an event is the acquisition of badges received by users in Stack Overflow; the Reddit dataset contains 10,000 sequences with 532,026 events where an event is posting in Reddit. Perhaps these days SVHN could be considered nearly a toy dataset, but there are still many papers in machine learning and computer vision with results focusing on CIFAR-10 and -100.

As requested, however, we ran additional experiments on larger-scale datasets, with results described in the common response. Overall, we see that AdaFlood still helps over unregularized and other flooding-regularized models, with even larger improvements than before in the noisy-label setting.

Reviewer comment (Experiments):
The paper's theatrical stage seems restricted to closed-set supervised classification, leaving realms like self-supervised learning, like contrastive learning in vision and auto-regression in NLP, unnoticed.

Author response:
Our experiments have been conducted on many domains with several tasks: density estimation for asynchronous time sequences, regression for tabular datasets, classification for image and text datasets. Density estimation and regression are very different from classification tasks. Therefore, it is hard to agree that our method is restricted to closed-set supervised classification.

We agree that we have not considered contrastive learning; it’s hard to see how to apply flooding-type regularizers in that setting. Autoregressive NLP tasks, however, are also a form of density estimation, and have characteristics of both the asynchronous time sequence datasets and supervised classification settings that we consider.

Reviewer comment (Evaluation):
Does the adaptive strategy bring extra training cost? More detailed discussion is required.

Author response:
In Section 4.5, we provided detailed discussion about the wall-clock time required to find the adaptive flood levels. Once we have these levels, there is essentially no computational overhead; adding one absolute value to the loss function is negligible.

If we use $M$ auxiliary networks, the time overhead is roughly $\mathcal{O}(M T)$, where $T$ is the training time of each auxiliary model. However, the efficient fine-tuning variant can reduce this to $\mathcal{O}(T + M t)$, where $t \ll T$ is the amount of time to fine-tune a model, without losing performance. Furthermore, since these adaptive flood levels are not closely tied to the “downstream” model, we can reuse them for a variety of downstream tasks, as was done e.g. in the C-score paper [1]. We will release these adaptive flood levels as a public difficulty measure, along with the code.

Another important point is that AdaFlood models converge faster than unregularized or Flood-based models. For instance, on CIFAR10, AdaFlood’s validation loss converges after around 80 epochs, compared to 150 for unregularized models, 150 for Flood, and 100 for iFlood. Similarly, on CIFAR100, while AdaFlood and iFlood’s validation loss converges after around 60 epochs, unregularized and Flood take around 150 epochs to converge. Thus, once we have adaptive flood levels, a significant amount of training time can be saved. This effect will be amplified as it is amortized for multiple training runs, e.g. when others use our publicly-released flood values.

[1] Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. “Characterizing structural regularities of labeled data in overparameterized models”. ICML 2021.