Paper: Deep Neural Networks Tend To Extrapolate Predictably (ICLR-2024)

Reviewer comment (Novelty):
Lack of references: the norm of the learned features was already known to be a discriminant characteristic between ID and OOD data (see, e.g., [1] and [2] cited above).

Author response:
We thank the reviewer for pointing these prior works out to us; we have added them to the related works of our updated paper.

Reviewer comment (Theory):
Lack of discussion on when the OCS does not hold. Although I understand that the results on the OCS hypothesis are still preliminary, I would like some additional remarks on when it might fail (as you showed in the appendix, but also as shown in works such as [1]). I suggest to briefly address this aspect in the introduction (see also Questions).

Author response:
We thank the reviewer for this suggestion; we have added some discussion about where “reversion to the OCS” does not hold in the paper, including in the introduction.

Reviewer comment (Theory):
I don't understand the conclusions drawn from the decision-making scenario. Ideas contained in the Hendrycks baselines paper ([1] in the summary) implicitly assume the OCS: if we choose an abstain threshold on the output confidence of a standard classifier, we already expect a non-zero abstain ratio on OOD data. In general, anomaly detectors on neural networks based on the output confidence are based on these ideas. Could you explain better the intuitions provided by section 5 on exploiting the OCS? How does the OCS hypothesis possibly change the way in which we define safer AI models with rejection options, compared to simple threshold-based ones?

Author response:
We acknowledge that one limitation of our current study is that our proposed method may not improve on SOTA results in selective classification. Our main goal for section 5 is to illustrate how the choice of loss function could influence the performance of learned policies for solving general decision-making problems with OOD inputs, using selective classification as a simple example. Due to the single step nature of selective classification, there indeed exist simple “hard-coded” methods such as using an abstain threshold on the output confidence of a standard classifier whose performance might match or even surpass that of our proposed method. However, such “hard-coded” techniques do not exist for more complex decision-making problems involving multi-step reasoning. For instance, consider the problem of long-form question answering using large language models. When queried with an OOD question, we would like the model to generate more cautious responses such as “I don’t know”, or “The answer might be .. “ over highly confident but incorrect responses such as “The answer is [wrong answer]”. However, it would be challenging to eliciting this kind of linguistically calibrated behavior using “hard-coded” thresholding methods, as it requires reasoning about the uncertainty of a sequence of tokens, and some tokens may be more significant than others (e.g. “The answer is ..”/ “It is..” vs “Japan”/ “Germany”). Our work suggests that fine tuning with other forms of loss functions whose OCS corresponds to linguistically uncertain answers (e.g. reinforcement learning) may offer a promising and scalable solution to this problem. While we have not yet had a chance to fully explore this direction, we are very excited about the general principle of aligning the OCS of the loss function with cautious decisions, and we hope that our selective classification example can offer a first step towards illustrating its usage.

Reviewer comment (Theory):
Do you see the OCS hypothesis as an "average" behavior of neural networks on OODs? In other works, do you expect that high-confidence OODs have to be constructed (e.g. adversarial examples), while the other ones will mostly respect the OCS hypothesis?

Author response:
Yes, this is exactly our intuition. In our experiments, we found “reversion to the OCS” to appear in almost all the non-adversarial OOD datasets that we tried, but not for adversarial datasets.

Reviewer comment (Theory):
In the OCS hypothesis, you clearly stated the assumption that OODs are high-dimensional. Is the OCS not valid on low-dimensional toy examples?

Author response:
In our experience, we have not found the “reversion to the OCS” phenomenon to hold on low-dimensional examples. For instance, consider this simple sin prediction example: https://ibb.co/Zf70F1K. Rather than reverting to the OCS (0), the model predictions moved farther away from the OCS as inputs became more OOD.

Reviewer comment (Evaluation):
In Figure 6, the OfficeHome dataset is the only one that has lower abstain ratio for the oracle compared to the predicted rewards. Is this because of the high number of target classes, compared to the other dataset, that makes it easier to achieve low confidence outputs?

Author response:
Indeed, we have observed that MSE models sometimes tend to be a bit more prone to underfitting than CE models, especially when the number of classes are high, which may explain why they tend to predict low(er) confidence outputs even on the training distribution.

Reviewer comment (Reproducibility):
Some of the training details are opaque in the main paper, which might lead into a simpler explanation over the observed empirical performance. For instance, could the learning algorithm or the data augmentation or the normalization impact this hypothesis?

Author response:
> Some of the training details are opaque in the main paper, which might lead into a simpler explanation over the observed empirical performance. Could the learning algorithm or the data augmentation or the normalization impact this hypothesis?


Having addressed the impact of normalization in the previous section, we will focus on the learning algorithm and the data augmentation. With regards to the learning algorithm, we restrict our observations to empirical risk minimization of deep neural networks using gradient descent, which we specify in Section 3. With regards to data augmentation, our experiments show the “reversion to the OCS” phenomenon on models both trained with (ImageNet, OfficeHome, BREEDS) and without (MNIST, SkinLesionPixels, UTKFace) data augmentation. Thus, while data augmentation may have an effect on the extent of “reversion to the OCS”, we believe “reversion to the OCS” cannot solely be explained by data augmentation. 

With regards to the training details, we have updated the experiments section of Section 4 to more explicitly mention that the MNIST analysis experiments use models trained without data augmentation and without normalizations, while the CIFAR10 analysis experiments use models with data augmentations and with batch norm. Further training details for every model we trained in the paper can be found in Section B of the Appendix, including the specific neural network architecture, optimizer, optimizer parameters, and data preprocessing. Please let us know if there are any other training details you would like to see that we did not include, and we would be happy to add them.

Reviewer comment (Theory):
I am skeptical about the hypothesis formed in the following sense: even if we assume a zero input, most modern networks rely on a normalization scheme, e.g. batch or layer normalization. Then, in a trained network, the “centering” provided by the learnt means and variances of the network will not result in a zero-mean representation for the next layers. As such, I am wondering how the normalization plays into the formed hypothesis.

Author response:
> I am wondering how the normalization plays into the formed hypothesis.

To better understand the role of normalization, we trained a neural network with no normalizations (NN), batch norm (BN), and layer norm (LM) using cross entropy on the MNIST dataset (we visualize the specific neural network architectures used here: https://ibb.co/c85zgjh).  We then evaluated these models on OOD inputs with increasing levels of distribution shift (rotations of 15, 30, 45, 60, 75 degrees), and measured the distance between the models’ predictions from the OCS, which we visualize here: https://ibb.co/DMQx6D9). We can see that “reversion to the OCS” holds for all three models.

Next we analyze the representations produced by each of the models when presented with OOD inputs. We conduct the same experiments as presented in Fig. 4 of our submission on all three models (the no normalization model is the same as the one in the paper), which we visualize here: https://ibb.co/qYghdzS). We can see that the relative norm of the representations tend to decrease as distribution shift increases in the later layers of the model for all three models (left). Furthermore, we can see similar behaviors for all three models with regards to the proportion of network features which lie within the span of the following linear layer (middle), and the similarity between the accumulation of model biases to the OCS (right).

While normalizations may play a role in enhancing “reversion to the OCS”, and additional work is needed to more precisely characterize the nature of its effect, the experiments we presented here showed that normalization alone does not explain the “reversion to the OCS” phenomenon, because it still exists for models without normalizations. Furthermore, there is evidence in both models with or without normalizations for our hypothesis on the explanation behind the “reversion to the OCS” phenomenon

Reviewer comment (Evaluation):
In fig. 4, I am wondering what the norm is for ID data; is it the case that the norm is also decreasing on those networks as well?

Author response:
> In fig. 4, I am wondering what the norm is for ID data; is it the case that the norm is also decreasing on those networks as well?


In the left plot in Figure 4, the quantity we are plotting on the y-axis is the relative norm between OOD representations and ID representations (average norm of OOD representations divided by the average norm of ID representations). Thus, the ID data will correspond to a value of 1 for each layer.

Reviewer comment (Theory):
In theorem 4.1 it mentions a “shallow” network with L’ layers. Is this a contradiction?

Author response:
> In theorem 4.1 it mentions a “shallow” network with L’ layers. Is this a contradiction?


Thank you for pointing out the confusing wording of Theorem 4.1, we have updated the paper such that the theorem is stated in more clear terms. 

Note that the networks we consider can indeed have a large number of layers $L$. By “shallow”, what we meant was that if there exists a relatively shallower network (with number of layers $L' < L$) that can fit the data with some margin $\gamma$, then the network we train learns low rank weights, i.e. the learnt weights $\hat{W}$ imply the existence of low rank subspaces $\{A_i\}_{i=1}^L$ that satisfy the property (on OOD data) in Thm 4.1.

Reviewer comment (Evaluation):
[Evaluation] While it’s understandable that this work doesn’t focus on achieving SOTA results, it’s still highly desirable to see how the proposed algorithm compares to existing selective classification (or OOD detection) baselines, and/or how they can be combined to further boost performance (discussion would be fine too).

Author response:
We don't believe that our method will be better than the SOTA approach in selective classification. However, insights from SOTA methods can indeed be combined with ours to potentially boost performance. For instance, [1] found that leveraging unlabeled data to train the model via an unsupervised loss can improve the performance in selective classification by a significant margin. While our method currently does not make use of unlabeled data, adding the use of unlabeled data during training may lead to significant improvements in our method as well. We hope to investigate this (and other techniques) further in future work.

[1] Towards Better Selective Classification, ICLR 2023

Reviewer comment (Evaluation):
[Significance] While this paper is good in most aspects (as summarized in Strengths), its significance however is a bit insufficient in my opinion and can be substantially improved by addressing the following issues: 1) Evaluation as stated above. More evaluation can help strengthen the applicability of the paper.

Author response:
We don't believe that our method will be better than the SOTA approach in selective classification. However, insights from SOTA methods can indeed be combined with ours to potentially boost performance. For instance, [1] found that leveraging unlabeled data to train the model via an unsupervised loss can improve the performance in selective classification by a significant margin. While our method currently does not make use of unlabeled data, adding the use of unlabeled data during training may lead to significant improvements in our method as well. We hope to investigate this (and other techniques) further in future work.

[1] Towards Better Selective Classification, ICLR 2023

Reviewer comment (Evaluation):
The failure of the claim in some of the distribution shifts (adversarial perturbation and impulse noise in Appendix A) raises concerns about the generalizability of this work. What kinds of OOD shifts (extrapolations) are actually supported and not supported by this work? Is there a way to more formally and/or finely characterize them? More evaluation datasets as well as systematically generated OOD shifts e.g. [1] could be helpful.

Author response:
Our intuition is that distribution shifts which cause the distribution of model representations to differ from those of training inputs will lead to “reversion to the OCS”, whereas those that don’t will not. In our analysis in 4.1, we found that naturally occurring or non-adversarial shifts tend to cause the representations to shift from the training representations. If we model non-adversarial shifts as adding “noise” to the directions of the representation vectors, then we would expect the overall distribution of representations to also shift, because the training representation distribution usually lies in a low dimensional space. In contrast, because adversarial inputs are optimized to maximize loss, they may be more likely to map to representations which align with the training representations. Indeed, we found “reversion to OCS” to hold for almost all the non-adversarial shifts we tried, but not adversarial shifts. Of course, the intuition we presented above is not rigorous, and more work is needed to characterize it in a more precise and rigorous manner. We hope to tackle this important question in future work.

Reviewer comment (Evaluation):
In Fig 7, reward prediction seems to be noticeably better than standard classification on all CIFAR10 noises and particularly OfficeHome even at the training distribution (t), i.e. supposedly not OOD. Is this solely because of abstention (as shown in Fig 6), or does MSE loss somehow works better than CE loss in this case? More generally, how do their accuracies (instead of rewards) compare e.g. without considering the abstained samples?

Author response:
We visualize the accuracy of each model on all the ID/OOD datasets here: https://ibb.co/nnsfvmJ. In general, the accuracy of the CE model is about the same or a bit higher than the MSE model. Thus, the reason the MSE model has higher reward than the BC model is because it is able to abstain better, not because it has higher accuracy.

Reviewer comment (Evaluation):
In Fig 4, the normalized norm in early layers for CIFAR10 (bottom left panel) seems to be systematically increasing with noise, contrary to later layers. Is this an expected behavior?

Author response:
Not necessarily; we found the behavior of the early layers to change depending on the model and the evaluation dataset. For instance, evaluating the CIFAR10 models on Gaussian noise produces normalized norms which don't have this behavior in the early layers:  https://ibb.co/Nt9LrZR.

Reviewer comment (Experiments):
Is there a particular reason to use -4 for the incorrect results? Does any number that brings the OCS below 0 (-3.5 in the paper) work?

Author response:
Yes, any number that brings the OCS below 0 should work. For instance, we additionally include an experiment using a reward of -8 for incorrect predictions here: https://ibb.co/7CtMvJ7. Here, we can see that the difference between the reward prediction and imitation models is bigger than the one in the paper (with -4 incorrect prediction reward), because there is now a higher penalty for choosing an incorrect prediction.

Reviewer comment (Experiments):
Do behaviors of $W_i \phi_I(x)$ mentioned in Section 4.1 remain on realistic datasets, such as ImageNet and Amazon?

Author response:
To address this question, we conducted new analysis experiments on ImageNet (evaluated on ImageNet-R(endition) and ImageNet-Sketch), and our results show similar trends as those in the paper. Our results can be found here: https://ibb.co/x2gV4pY

Reviewer comment (Evaluation):
From the insight of the paper, can we say that methods that try to improve models’ performances on OOD inputs are actually moving the models’ outputs away from the OCS?

Author response:
Not necessarily. For cross entropy, a model’s performance can be improved by 1) changing a low confidence prediction to a highly confident and correct prediction, 2) changing a highly confident incorrect prediction to a highly confident correct prediction, 3) changing a highly confident but incorrect prediction to a low confidence prediction. Here, 1) will cause the model outputs to move away from the OCS, 2) will cause the model outputs to remain about the same distance to the OCS, and 3) will cause the model outputs to move towards the OCS. Thus, it is possible to improve the performance of a model on OOD inputs while moving the model’s outputs more towards the OCS.