Paper: Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift (ICLR-2024)

Reviewer comment (Experiments):
While the proposed method performs well through the simulations and empirical tests it is unclear if it is better than other methods. For example, in Table 2 and 3 accounting for the errors it does not seem to be statistically different from the other methods (for example ERM in Table 2). The authors also do not compare it in simulations with alternative methods besides simple OLS. For instance, it seems that PCR might perform well in this setting.

Author response:
Finally, we have followed your suggestion of benchmarking Principal Component Regression in our synthetic experiments. If we simply project the data onto the principal components (i.e. rotating the data) and then calculate the OLS solution, this performs similarly to OLS on the canonical basis. If we project the data onto only the leading principal component and then calculate the OLS solution, this yields performance similar to SpAR. This alludes to some important properties of SpAR's behaviour:

- Behaving like PCR: In some circumstances, projecting onto the top-k principal components leads to lower loss. This is the case with our synthetic experiment. While PCR would require one to select how many principal components to project onto, SpAR automatically chooses which eigenvectors to project out.
- Behaving like OLS: If no Spectral Inflation takes place, then the regressor produced by SpAR is equal to the OLS solution. This is optimal when $bias_{z,j}$ is greater than $var_{z,j}$. In circumstances such as this, PCR will perform worse than OLS and therefore worse than SpAR.
- Unique behaviour: Since SpAR compares each test eigenvector's bias and variance separately, the eigenvectors it projects out can be interleaved throughout the spectrum instead of grouped together at the end of the spectrum. In this circumstance, SpAR will obtain lower expected loss than OLS or PCR. Analyzing the eigenvectors projected out by SpAR, we observe that SpAR does indeed project out a complex selection of eigenvectors, rather than just a top-k. In a revision to the work, we will include an example of which eigenvectors are projected out by SpAR.

This demonstrates one of SpAR's most important properties: that it flexibly adapts to the covariate shift specified by the test data, rather than relying on the assumption that a certain adaptation will best perform OOD, as is the case with PCR. This is consistent with our other experiments where we find that adapting neural regressors using SPaR leads to consistent improvements across several real world datasets, both in the original submission and our newly added experiments from the discussion phase.

Reviewer comment (Theory):
In the decomposition results, what is the expectation being taken over? I thought that X and Z are treated as random, but in the proof in page 13 in the appendix it seems that X and Z are fixed. Are X and Z random or fixed? Are the decomposition results conditional on X and Z?

Author response:
**On the stochasticity of variables**: In our work, we only assume that the label noise on the labels Y is random. X and Z are assumed to be fixed matrices.

Reviewer comment (Theory):
Theorem 3 states that S^* is weakly better than the OLS solution, not that it is the best solution amongst all possible S. However in the paragraph above it is stated that it is the ideal set. Is it the case that S^* is the projection set that minimizes the expected loss amongst all possible S sets? It may be trivial, but it would be worth it fully explaining this in the main body of the text. If S^* is the the projection set that minimizes the expected loss then it should be stated as a theorem, if not then you should explain why you focus on S^* rather than the minimizing S.

Author response:
**On the strength of Theorem 3**: You are indeed correct that $S^*$ is the optimal set amongst all projection sets $S$ for decreasing the expected loss. This can be shown in a stronger version of theorem 3. While the projected regressor associated with $S^*$ having lower loss than the OLS solution is indeed a consequence of $S^*$ being optimal (take S equal to the empty set), it is perhaps the most important of these results as we are attempting to construct a regressor superior to OLS.

Reviewer comment (Theory):
Can you use the plug in variance estimator to estimate the variance on the same data? Why is a sample split not necessary? (this could be trivial given the assumptions)

Author response:
**About the label noise variance estimator**: The estimator for the variance of the label noise can indeed be used without a data split. We use the estimator in Section 11.2.3.6 of Kevin Murphy’s Probabilistic Machine Learning (Murphy, Kevin P. Probabilistic machine learning: an introduction. MIT press, 2022). This is defined as simply the average of the residuals of the OLS solution applied to the training data.

Reviewer comment (Theory):
What would change if Z was noisy? If we assume conditionally independent errors like for X, the conditional expectation would still be the same so it seems that most of the theory would go through with little changes (and the additional Z variance).

Author response:
**About adding noise to the test labels**: If we were to add noise to the labels for Z that was independent of the noise on the X labels and had zero mean, in expectation this would simply introduce an irreducible loss term that was dependent on the variance of the Z label noise. Since this additive term is not influenced by any parameters we control (i.e. our choice of regressor), we omit it from our calculations for clarity’s sake. This is an important distinction to make, however, as some of the loss we observe in real world experiments could be due to label noise on the test examples.

Reviewer comment (Writing):
I found it quite difficult to follow parts of Section 3, especially Section 3.4.

Author response:
Thank you for your comments about the clarity of the writing. We will give the new concepts mentioned in Section 3.4 a more clear treatment in our revision of the work.

Reviewer comment (Experiments):
The experimental evaluation could be more extensive. The proposed method is applied just to three real-world datasets, of which two are tabular datasets where small networks with a single hidden layer are used.

Author response:
## Choice of datasets
Thank you again for the pointer to Gustafsson et al 2023, which we agree is related and worth discussing. While their focus on uncertainty quantification differs from ours, the overall theme (that the reliability of neural regression models is relatively understudied and important in real world settings) aligns perfectly with the stated motivation for our method. Moreover, following your suggestion, we have incorporated one of these datasets, plus a relevant image regression benchmark from another paper, into our experiments section. The limiting factor in us adding additional benchmarks was time (speaking to your point about the lack of time during the ICLR discussion period). We did not try all eight benchmarks and pick the one that worked well. Rather, SpAR has worked well on each of the new datasets we have tried. We are happy to continue adding datasets in the camera ready version, although we believe that the evidence we have already collected (across 5 real-world datasets and 1 synthetic study) is sufficient to demonstrate the efficacy of our proposed method. We also note that Gustafsson et al 2023 was accepted to TLMR in the same week that we submitted this paper, so we only became aware of it through the discussion period here, which limited our ability to try all of of its benchmark datasets prior to submission.

Reviewer comment (Evaluation):
The experimental results are not overly impressive/convincing. The gains of ERM+SpAR compared to the ERM baseline in Figure 3, Table 2 and Table 3 seem fairly small.

Author response:
## Performance of SpAR relative to baselines
We would argue that the main practical upsides of of SpAR are its adaptiveness and consistency. As noted by many other papers, OOD generalization without tuning hyperparameters on the target domain is very challenging, and standard training (ERM) remains a difficult baseline to beat. Because we choose which eigenspaces to project by measuring spectral inflation, this means that SpAR will default to OLS for distribution shifts where OLS is well suited. Likewise, if ERM would generalize better OOD, SpAR can match ERM's performance. However, for some distribution shifts, ERM may also suffer from some amount of spectral inflation. In these cases, SpAR can exceed ERM's performance (e.g. in the 4% boost in worst-group performance on PovertyMaps, which represents SOTA on this challenging dataset). So we can think of SpAR as a "best of all worlds" approach that we believe will interest practitioners. Notably, many of the competing methods we evaluated, especially domain adaptation approaches that (like SpAR) also use unlabeled target domain data, do much worse than SpAR or ERM.

Reviewer comment (Evaluation):
The computational cost of the proposed method (Algorithm 1) is not discussed in the main paper, and only briefly mentioned in the Appendix.

Author response:
We would first like to thank you for helping to improve our work.  A number of your major concerns were shared with the other reviewers, and so please take a look at the general response to all reviewers. There, we respond to your concerns about the computational complexity of SpAR, how ERM presents a strong baseline, and how the variance in performance presented in Tables 2 and 3 is not due to SpAR but the multiple splits of the data that the models are being evaluated on.

Reviewer comment (Evaluation):
Can you please discuss the results in Figure 3, Table 2 and Table 3 a bit more, the gains of ERM+SpAR compared to ERM seem quite small? Does the proposed method actually improve the performance of ERM in a significant way?

Author response:
## Performance of SpAR relative to baselines
We would argue that the main practical upsides of of SpAR are its adaptiveness and consistency. As noted by many other papers, OOD generalization without tuning hyperparameters on the target domain is very challenging, and standard training (ERM) remains a difficult baseline to beat. Because we choose which eigenspaces to project by measuring spectral inflation, this means that SpAR will default to OLS for distribution shifts where OLS is well suited. Likewise, if ERM would generalize better OOD, SpAR can match ERM's performance. However, for some distribution shifts, ERM may also suffer from some amount of spectral inflation. In these cases, SpAR can exceed ERM's performance (e.g. in the 4% boost in worst-group performance on PovertyMaps, which represents SOTA on this challenging dataset). So we can think of SpAR as a "best of all worlds" approach that we believe will interest practitioners. Notably, many of the competing methods we evaluated, especially domain adaptation approaches that (like SpAR) also use unlabeled target domain data, do much worse than SpAR or ERM.