Paper: Maximum Likelihood Estimation is All You Need for Well-Specified Covariate Shift (ICLR-2024)

Reviewer comment (Theory):
The paper might overly exaggerate its contributions. While the title suggests that MLE is "All You Need" for well-specified covariate shift, the authors introduce a series of quite non-trivial assumptions in Assumption A, and Theorem 3.1 appears to be essentially a direct corollary of these assumptions. Technically, these assumptions seem to essentially state that the MLE converges to the optimal parameter — the very result that Theorem 3.1 claims to prove.

Author response:
We thank the reviewer for the comments.

**Regarding Assumptions:**
We highlight that we not only established the consistency of MLE (which is known before), but also critically establish that statistical rate of MLE among all algorithms. The assumptions used in this paper are essentially standard assumptions that are used in the literature to ensure that MLE is consistent in the basic setting without any distribution shift. This paper goes much further to prove that MLE is not only consistent, but also minimax-optimal even under covariate shift, with almost no additional assumption. In this sense, we consider our assumptions not strong.

Reviewer comment (Experiments):
Regarding applications, the paper limits its focus to only (generalized) linear models. This narrow scope hardly convinces that the MLE is indeed "All You Need".

Author response:
**Regarding Applications:**
Our main results hold for general model classes beyond generalized linear models. The applications section is primarily intended to illustrate the versatility of our framework. Notably, our third example deviates from the typical GLM framework. Unlike GLMs, which generally feature a convex loss function and a unique minimum, phase retrieval does not conform to these characteristics, thereby extending the scope of our theory beyond GLMs.

Reviewer comment (Presentation):
The authors should represent their results more accurately, and the reviewer recommends changing the title to "Minimax Optimal Sample Complexity for Well-Specified Covariate Shift via MLE" or something along those lines.

Author response:
Regarding comments 2&3: Thanks for your valuable comments. We will address these issues in our revision.

Reviewer comment (Theory):
Assumption A and C apply to both the model class and the source and target distributions, not only to the model class as claimed by the authors.

Author response:
Regarding Assumption A and C:
We first fix a pair of source and target distributions and then make assumptions on the model class. Our minimax optimality results are instance-specific (which is stronger than standard instance independent minimax result that allows the choice of worse case instance), i.e., given a pair of source and target distributions, when the model class satisfies the assumptions, MLE is minimax optimal.

Reviewer comment (Theory):
In Section 4, regarding the applications, it would be beneficial to also mention the lower bounds for each example.

Author response:
Regarding comments 2&3: Thanks for your valuable comments. We will address these issues in our revision.

Reviewer comment (Theory):
The finite sample guarantees that are presented hold when the number of samples $n$ is greater than some threshold, which depends on many parameters of the problem, including for example $\|\mathcal{I}_T^{1/2}\mathcal{I}_S^{-1}\mathcal{I}_T^{1/2}\|_2$, which also measures how close the source and target distributions are in some sense. Since the sample complexity bounds only hold above that threshold, it seems that an important question is what is the dependence of the optimal sample complexity on quantities like $\|\mathcal{I}_T^{1/2}\mathcal{I}_S^{-1}\mathcal{I}_T^{1/2}\|_2$. Have the authors considered this question?

Author response:
Thanks for this interesting question. Our main results state that the optimal rate is achieved when the sample size exceeds some threshold. In some cases, the threshold we obtained is optimal. For example, in linear regression (Section 4.1), when the mean shift is small, the threshold is $O(d)$, which aligns with the result of usual linear regression. However, for some other cases, e.g., logistic regression, we admit that the threshold may not be tight, as we lean on a general framework designed for a variety of models rather than a specific one. Conducting a more detailed analysis to determine the optimal threshold presents an intriguing direction for future research.

Reviewer comment (Evaluation):
The authors instantiate their general bound for the well-specified setting in the case of linear regression. It would be interesting to compare the results that they get with the guarantees of prior work on this problem, such as Lei et al., if such a comparison can be made.

Author response:
Regarding Lei et al, they give an exact minimax linear estimator under fixed design. The estimator they proposed is not MLE and is much more complicated in certain regimes. Their results rely on the strong assumption that **the covariance matrix on source and on target commute** with each other. In contrast, our results suggest that MLE is optimal among all estimators up to some constants under mild assumptions. We will add more discussions after the linear regression example (Section 4.1).

Reviewer comment (Writing):
Since there are two different distributions of source and target, I think it would help with exposition if the authors had some notation about taking expectation wrt each of the two distributions instead of hiding this dependence. For example, in the proof of Lemma A.2 the expectation is wrt the source distribution.

Author response:
Thanks for your valuable suggestions and pointing out the typo regarding the expectation. We will edit our paper accordingly.

Reviewer comment (Theory):
In page 14, in the derivation of inequality (13) in the second line, shouldn't there be an expectation above the equality sign? ($\mathbb{E}[\nabla l(\beta^*)] = 0$)

Author response:
Thanks for your valuable suggestions and pointing out the typo regarding the expectation. We will edit our paper accordingly.

Reviewer comment (Novelty):
Although the points raised by the authors are practically important, I cannot find novel findings.

Author response:
We thank the reviewer for the comments. We respectfully disagree that “MLE is all we need” is known. We will first defend our title and re-highlight our novelty and main contribution. We will then answer the remaining questions individually.

**Title, novelty of this paper.** We believe that in order to claim that MLE is *all you need* for well-specified covariate shift, we need to provably show two points:

(a) MLE can find the correct solution under well-specified covariate shifts.

(b) More importantly, no algorithm is better/more efficient than MLE in this setting (therefore MLE is truly all that you need). That is, the statistical rate of MLE is optimal among *all* algorithms.

As written in our related work section, we agree that Shimodaira (2000) provides (asymptotic only) analysis for (a), which makes (a) partially known as the reviewer pointed out. However, Shimodaira (2000) did not prove (b) even in the asymptotic sense. Specifically, Shimodaira (2000) only proves that the asymptotic rate of MLE is the best among a family of “weighted likelihood estimators”. Their work/techniques do not extend to prove the optimality of MLE among a rich variety of algorithms beyond that family. In contrast, this paper provides the first general non-asymptotic result for point (a), and gives the first proof for point (b), where lies the novelty and contribution of this paper.

Reviewer comment (Theory):
As the authors mention, it is known that the covariate shift problem is not serious when models are correctly specified (Shimodaira, 2000). Although recent studies tend to omit discussing model misspecification when discussing the covariate shift problem, classical studies motivate the use of covariate shift adaptation by considering the model misspecification setting. Those studies and classical statistics agree that correctly specified models can address the OOD generation. Therefore, when models are correctly specified, "MLE is all we need" has been known to researchers, though recent studies often omit this point.

Author response:
**It’s known OOD is not a serious problem under the correctly specified models.** We agree it’s known asymptotically. However, as we highlighted in the first bullet point, this only partially contributes to point (a), which is a fraction of our claim. The equally important (if not more important) point (b) and the non-asymptotic analysis of point (a) are not known, which are first addressed by this paper.

Reviewer comment (Novelty):
The authors insist that the contributions lie in the finite-sample minimax optimality of MLE under correctly specified parametric models. However, if we consider parametric models and minimax optimality, such a result has already been shown by existing studies with more general forms or is trivial, though I do not raise some specific related work. This is because when considering parametric models \( f_\theta(x) \) parametrized by \( \theta \), it is enough to consider the minimax optimality of estimation of \( \theta \). Once optimality is established, the result can be extended to show the minimax optimality of estimation of \( f_\theta(x) \) using the Taylor expansion for each \( x \) or uniformly over \( x \). There are various results that discuss the optimality of \( \theta \) estimation, and we can just employ them.

Author response:
**Finite sample optimality.** The reviewer raised the comment: “However, if we consider parametric models and minimax optimality, I think that such a result has already been shown by existing studies with more general forms or is trivial, though I do not raise some specific related work. ” To our best knowledge, we are not aware of any existing minimax optimality results of MLE under general well-specified covariate shift, either with more general forms, or with trivial derivations. We are also not quite able to verify the correctness of the short proof sketch provided by the reviewer. In particular, we are not sure why “it is enough to consider the minimax optimality of estimation of theta” especially in the lower bound perspective. In general, parameter estimation being hard does not imply prediction (the focus of this paper) being hard. It would be highly appreciated if the reviewer can provide concrete references or more detailed checkable proof sketch to facilitate discussion.

Reviewer comment (Theory):
Several existing studies discuss the uniform or point-wise minimax optimality of estimation of \( f_\theta(x) \), which directly implies the minimax optimality under a covariate shift. Such a result can be easily obtained if only parametric models are considered, and the results shown by the authors can be more strengthened.

Author response:
**Uniform convergence.** We agree with the reviewer that if one can provide a point-wise optimal estimator, then optimality under covariate shift is guaranteed. However, to the best of our knowledge, we haven’t seen such studies in our context. The closest one we know is the paper “Local convergence rates of the least squares estimator with applications to transfer learning”. The paper studies the local convergence rate when the regression function is Lipschitz. It would be helpful if the reviewer can provide more related work on this thread.

Reviewer comment (Theory):
Why do authors consider the minimax optimality of parametric models under a covariate shift? Is it difficult to derive corresponding results for nonparametric models? Or can the authors show uniform optimality?

Author response:
**Q1**: Thanks for raising this interesting and important question. In fact, nonparametric models behave differently under covariate shifts. For instance, nonparametric models over RKHS under covariate shift have been studied by [1]. It was shown that MLE (ERM in their language) is provably suboptimal  for addressing covariate shift under nonparametric RKHS assumptions. In contrast, for well-specified parametric models, we show that this wouldn’t be the case as MLE is minimax optimal under covariate shift for well-specified parametric models.

[1] Cong Ma, Reese Pathak, and Martin J Wainwright. Optimally tackling covariate shift in rkhs-based nonparametric regression. The Annals of Statistics, 51(2):738–761, 2023.

Reviewer comment (Novelty):
To the best of my knowledge, it has been known that correctly specified parametric models can adapt the OOD, at least in statistics. Therefore, the contributions of this study mainly lie in the finite-sample minimax optimality. Is my understanding correct?

Author response:
**Q2**: Hopefully our response in the first bullet point answers this question.

Reviewer comment (Novelty):
The title may not be appropriate. As discussed and as the authors mention in the draft, "MLE is all we need" has been known. The true contribution of this study is the finite-sample minimax optimality of MLE for correctly specified parametric models. If so, the title should express the true contributions. For example, if I were the authors, I would name the study "Minimax Optimality of Maximum Likelihood Estimation for Covariate Shifted Norm." The authors should focus on their contributions more to clarify the claims of this study.

Author response:
We thank the reviewer for the comments. We respectfully disagree that “MLE is all we need” is known. We will first defend our title and re-highlight our novelty and main contribution. We will then answer the remaining questions individually.

**Title, novelty of this paper.** We believe that in order to claim that MLE is *all you need* for well-specified covariate shift, we need to provably show two points:

(a) MLE can find the correct solution under well-specified covariate shifts.

(b) More importantly, no algorithm is better/more efficient than MLE in this setting (therefore MLE is truly all that you need). That is, the statistical rate of MLE is optimal among *all* algorithms.

As written in our related work section, we agree that Shimodaira (2000) provides (asymptotic only) analysis for (a), which makes (a) partially known as the reviewer pointed out. However, Shimodaira (2000) did not prove (b) even in the asymptotic sense. Specifically, Shimodaira (2000) only proves that the asymptotic rate of MLE is the best among a family of “weighted likelihood estimators”. Their work/techniques do not extend to prove the optimality of MLE among a rich variety of algorithms beyond that family. In contrast, this paper provides the first general non-asymptotic result for point (a), and gives the first proof for point (b), where lies the novelty and contribution of this paper.