Original: Probable Domain Generalization via Quantile Risk Minimization
Revised: Probable Domain Generalization via Quantile Risk Minimization

Revised: Author

Original: Abstract
Revised: Abstract

Original: Domain generalization (DG) seeks predictors which perform well on unseen testdistributions by leveraging labeled training data from multiple  related distributionsor domains .
Revised: Domain generalization (DG) leverages labeled training data from multiple domainswith the goal of generalizing to related test domains.

Original: To achieve this, DG is commonly formulated as a worst-case optimization problem over the set of all possible domains.
Revised: To realize this goal, the standard formulation optimizes for worst-case performance by taking an adversarialperspective over the set of all possible domains.

Original: However,with worst-case shiftsunlikely in practice, this  generally leads to overly-conservative solutions.
Revised: However, this worst-case problemis generally intractable and, with adversarial shifts extremely unlikely in practice,leads to overly-conservative solutions.

Original: In fact, arecent study found that no DG algorithm outperformed empirical risk minimizationin terms of average performance.
Revised: In fact, a recent study found that no DG algorithm outperformed empirical risk minimization in terms of average performanceover test domains.

Original: In this work, we argue that DG is neither a worstcase problem nor an average-case problem, but rather a probabilistic one. To thisend, we propose a probabilistic framework for DG, which we call Probable Domain 
Revised: To address these shortcomings, we propose a probabilisticframework for DG, which we call Probable Domain Generalization , and advocatefor predictors that perform well with high probability rather than in the worst-caseor on-average.

Original: Generalization , wherein our key idea is that distribution shifts seen during trainingshould inform us of probable shifts at test time.
Revised: Our key idea is that distribution shifts seen during training shouldinform us of probable shifts at test time.

Original: To realize this, we explicitly relatetraining and test domains as draws from the same underlying meta-distribution, andpropose a new optimization problem— Quantile Risk Minimization (QRM)—whichrequires that predictors generalize with high probability .
Revised: To achieve this, we explicitly relatetraining and test domains as draws from the same underlying meta-distribution, andpropose a new optimization problem— Quantile Risk Minimization (QRM)—whichrequires that predictors generalize with high probability.

Original: We then prove that QRM:(i) produces predictors that generalize to new domains with a desired probability,given sufficiently many domains and samples; and (ii) recovers the causal predictoras the desired probability of generalization approaches one.
Revised: We then prove that, givensufficiently many domains and samples, the empirical version(EQRM) producespredictors that generalize to new domains with the desired probability. We alsoshow that EQRM recovers the causal predictor as the desired probability of generalization approaches one.

Original: In our experiments, weintroduce a more holistic quantile-focused evaluation protocol for DG,  and showthat our algorithms outperform state-of-the-art baselines on real and synthetic data.
Revised: In our experiments, we introduce a new evaluation protocolfor DG, which underscores the importance of multiple test domains for evaluatingthe quantile performance of DG algorithms, and we show that our algorithmsoutperform strong DG baselines on real and synthetic data.

Original:  Introduction
Revised:  Introduction

Original: Despite remarkable successes in recent years [1–3], machine learning systems often fail calamitouslywhen presented with out-of-distribution (OOD) data [4–7].
Revised: Despite remarkable successes in recent years [1–3], machine learning systems often fail calamitouslywhen presented with out-of-distribution (OOD) data [4–7].

Original: In fact, evidence of state-of-the-art systemsfailing in the face of distribution shift is mounting rapidly—be it due to spurious correlations [8–10],changing sub-populations [11–13], changes in location or time [14–16] or other naturally-occurringvariations [17–23].
Revised: In fact, evidence of state-of-the-art systemsfailing in the face of distribution shift is mounting rapidly—be it due to spurious correlations [8–10],changing sub-populations [11–13], changes in location or time [14–16], or other naturally-occurringvariations [17–22].

Original: These OOD failures are particularly concerning in safety-critical applicationssuch as medical imaging [24–28] and autonomous driving [29–31], where they represent one of themost significant barriers to real-world deployment [32–35].
Revised: These OOD failures are particularly concerning in safety-critical applicationssuch as medical imaging [23–27] and autonomous driving [28–30], where they represent one of themost significant barriers to real-world deployment [31–34].

Original: Domain generalization (DG) seeks to improve a system’s
Revised: Domain generalization (DG) seeks to improve a system’s

Original: OOD performance by leveraging datasetsfrom multiple environments or domains at training time, each collected under different experimentalconditions [36–38].
Revised: OOD performance by leveraging datasetsfrom multiple environments or domains at training time, each collected under different experimentalconditions [35–37].

Original: The goal is to build a predictor which exploits invariances across the trainingdomains in the hope that these invariances also hold in related but distinct test domains [38–41].
Revised: The goal is to build a predictor which exploits invariances across the trainingdomains in the hope that these invariances also hold in related but distinct test domains [37–40].

Original: Torealize this goal,  DG is commonly formulated as a worst-case optimization problem over the set ofall possible domains.
Revised: To realize this goal, the standard DG formulation optimizes for worst-case performance by takingan adversarial perspective over the set of all possible domains.

Original: However, this problem is generally intractable [42–44] and, with worst-caseshifts unlikely in practice, leads to overly-conservative solutions—making large accuracy sacrifices inpursuit of adversarial robustness.
Revised: However, this worst-case problemis generally intractable [41–43] and, with adversarial shifts unlikely in practice, leads to overlyconservative solutions—making large accuracy sacrifices in pursuit of adversarial robustness.

Original: In fact, a recent study [38] found that, across several popular datasets,no DG algorithm outperformed empirical risk minimization (ERM)  in terms of average performance . In this work, we argue that DG is neither a worst-case problem nor an average-case problem, butrather a probabilistic one. In particular, we advocate for predictors which perform well with highprobability rather than in the worst case or on average. To this end, we propose a probabilistic
Revised: In fact, arecent study [37] found that, across several popular datasets, no DG algorithm outperformed empiricalrisk minimization (ERM) when compared in terms of average performance over test domains.

Original: [Figure]
Revised: [Figure]

Revised: In this work, we argue that DG should both seek and evaluate predictors based on their ability to perform well with high probability , rather than in the worst-case or on-average.

Original:  framework for DG which we call Probable Domain Generalization . The key idea is that distributionshifts seen during training should inform us of probable shifts at test time since they represent theways in which the underlying system tends to change (Fig. 1a).
Revised: To this end, we proposea probabilistic framework for DG, which we call Probable Domain Generalization , wherein the keyidea is that distribution shifts seen during training should inform us of probable shifts at test time.

Original: To realize this, we explicitly relatetraining and test domains as draws from the same underlying meta-distribution (Fig. 1b), and thenpropose Quantile Risk Minimization (QRM)—a new optimization problem for learning predictorsthat generalize with high probability (§ 3).
Revised: Torealize this, we explicitly relate training and test domains as draws from the same underlying metadistribution, then propose Quantile Risk Minimization (QRM)—a new optimization problem whichrequires that predictors generalize with high probability (§ 3).

Original: The goal of QRM is to minimize the α -quantile of apredictor’s risk distribution over training domains, leveraging the key insight that this α -quantileis an upper bound on the test risk which holds with probability α (Fig. 1c).
Revised: The goal of QRM is to minimize thequantiles of a predictor’s risk distribution over training domains, leveraging the fact that the α -quantileof this distribution is an upper bound on the test risk which holds with probability α (see Fig. 1c).

Original: We then prove that QRM: (i) produces predictors that generalize to new domains with probabilityα , given sufficiently many domains and samples (§ 4); and (ii) recovers the causal predictor asα → 1 , under weaker assumptions than Peters et al.
Revised: Next, we prove that, given sufficiently many domains and samples, the empirical analogue (EQRM,§ 4) produces predictors which generalize to new domains with probability α . We also show that, undersimilar assumptions to previous work [44], EQRM recovers the causal predictor as α → 1 .

Original: Thus, α is an interpretable conservativenesshyperparameter, with α = 1 corresponding to the worst-case setting.
Revised: Thus, α isan interpretable conservativeness-hyperparameter, with α = 1 corresponding to the adversarial setting.

Original: In our experiments (§ 6), weintroduce a more holistic quantile-focused evaluation protocol for DG, and demonstrate that: (i) theperformance of DG algorithms was likely never “lost” [38], but rather invisible through the lens of average performance; and (ii) our algorithms outperform strong baselines on real and synthetic datasets.
Revised: In our experiments (§ 6), we introduce a new evaluation protocol for DG which focuses on a predictor’sperformance at different test quantiles, and demonstrate that: (i) the performance of DG algorithmswas likely never “lost” [37], but rather invisible through the lens of average performance; and (ii) ourquantile-minimizing algorithms outperform state-of-the-art baselines on real and synthetic datasets.

Original: Contributions.
Revised: Contributions.

Original: To summarize our main contributions, we:
Revised: To summarize our main contributions, we:

Original: • Propose Quantile Risk Minimization for learning predictors that generalize with probability α (§ 3).• Prove that, given sufficiently many domains and samples,  QRM  does indeed produce predictorsthat generalize to new domains with probability α (Thm.
Revised: • Propose Quantile Risk Minimization for learning predictors that generalize with probability α (§ 3). • Prove that, given sufficiently many domains and samples, empirical QRM (EQRM) does indeedproduce predictors that generalize to new domains with probability α (Thm.

Original: 4.1).• Prove that QRM recovers the causal predictor as α → 1 (Prop. 4.3 and Thm. 4.4).• Demonstrate empirically that: (i) effective comparison of DG algorithms requires evaluating quantile performance; and (ii) our algorithms outperform strong baselines on real and synthetic data (§ 6).
Revised: 4.1).• Prove that EQRM recovers the causal predictor as α → 1 (similar assumptions to [44], Prop. A.4).• Demonstrate empirically that: (i) effective comparison of DG algorithms requires evaluating quan- tile performance; and (ii) our algorithms outperform strong baselines on real and synthetic data (§ 6).

Original:  Background
Revised:  Background

Original: Domain generalization (DG).
Revised: Domain generalization (DG).

Original: In DG, predictors are trained on data drawn from multiple relatedtraining distributions or domains and then evaluated on related but unseen test domains.
Revised: In DG, predictors are trained on data drawn from multiple relatedtraining distributions or domains and then evaluated on related but unseen test domains.

Original: For example,in the Camelyon17 dataset [46], the task is to predict if a given image of cells contains tumortissue, and domains correspond to different hospitals in which these images were captured.
Revised: For example,in the Camelyon17 dataset [45], the task is to predict if a given image of cells contains canceroustumors, and domains correspond to different hospitals in which these images were captured.

Original: Moreformally, we consider datasets D e
Revised: Moreformally, we consider datasets D e

Original: = { ( x ei , y ei ) } n e i = 1 collected from m different training domains orenvironments E tr : = { e 1 , . . .
Revised: = { ( x ei , y ei ) } n e i = 1 collected from m different training domains orenvironments E tr : = { e 1 , . . .

Original: , e m } , with each dataset D e containing data pairs ( x ei , y ei ) sampled i.i.d.from P ( X e , Y e ) .
Revised: , e m } , with each dataset D e containing data pairs ( x ei , y ei ) sampled i.i.d.from P ( X e , Y e ) .

Original: Then, given a suitable function class F and loss function ℓ , the goal of DG is to learna predictor f ∈ F that generalizes to data drawn from a larger set of all possible domains E all ⊃ E tr .
Revised: Then, given a suitable function class F and loss function ℓ , the goal of DG is tolearn a predictor f ∈ F that generalizes to data drawn from a larger set of all possible environments E all ⊃ E tr .

Original: By letting R e ( f ) denote the statistical risk of f in domain e , this is commonly [9, 22, 41, 47–49]formalized as the following minimax optimization problem:
Revised: By letting R e ( f ) denote the statistical risk of f in domain e , this can be formalized as thefollowing minimax optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original:  As we only have access to data from a finite subset of E all during training, solving (DG) is not justchallenging but in fact impossible [41, 50, 51] without restrictions on how the domains may differ
Revised: X e ) , Y [Equation] [Equation] As we only have access to data from a finite subset of E all during training, solving (DG) is not justchallenging but in fact impossible [40, 46, 47] without restrictions on how the domains may differ.

Original: Causality and invariance in DG.
Revised: Causality and invariance in DG.

Original: Causal works on DG [9, 41, 45, 51, 52] describe domaindifferences using the language of causality and the notion of interventions [53, 54].
Revised: Causal works on DG [9, 40, 44, 47, 48] describe domain or environmental differences using the language of causality and the notion of interventions [49, 50].

Original: In particular, they assume all domains share the same underlying structural causal model (SCM) [53], with differentdomains corresponding to different interventions (see Appendix A.1 for formal definitions and asimple example).
Revised: In particular, they assume all environments share the same underlying structural causal model (SCM) [49],with different environments corresponding to different interventions (see Appendix A.1 for formaldefinitions and a simple example).

Original: Early work studied the problem of learning from multiple cause-effect datasets thatshare a functional mechanism but differ in noise distributions [39].
Revised: Early work studied the problem of learning from multiple causeeffect datasets that share a functional mechanism but differ in noise distributions [38].

Original: More generally, given (data from)multiple distributions, one can try to identify components which are stable, robust, or invariant , andfind means to transfer them across problems [55–59].
Revised: More generally,given (data from) multiple distributions, one can try to identify components which are stable, robust,or invariant , and find means to transfer them across problems [51–55].

Original: Assuming that the mechanism of Y remainsfixed or invariant 1 but all X s may be intervened upon, recent works have shown that only the causalpredictor: (i) has invariant predictive distributions [45], coefficients [9], or risks [41] across domains;and (ii) generalizes to arbitrary interventions on the X s [9, 45, 52].
Revised: Assuming that the mechanismof Y remains fixed or invariant 1 but all X s may be intervened upon, recent works have shown that onlythe causal predictor: (i) has invariant predictive distributions [44], coefficients [9], or risks [40] acrossenvironments; and (ii) generalizes to arbitrary interventions on the X s [9, 44, 48].

Original: These works then exploit suchinsights by leveraging some form of invariance across domains to discover causal relationshipswhich, through the invariant mechanism assumption, generalize to new domains.
Revised: These works thenexploit such insights by leveraging some form of invariance across environments to discover causalrelationships which, through the invariant mechanism assumption, generalize to new environments.

Original:  Quantile Risk Minimization
Revised:  Quantile Risk Minimization

Original: In this section we introduce Quantile Risk Minimization (QRM) for achieving probable domaingeneralization.
Revised: In this section we introduce Quantile Risk Minimization (QRM) for achieving probable domaingeneralization.

Original: The core idea is to replace the worst-case perspective of (DG) with a probabilistic one.
Revised: The core idea is to replace the worst-case perspective of (DG) with a probabilistic one.

Original: This approach is founded on a great deal of work in classical fields such as control theory [60, 61] andsmoothed analysis [62], wherein approaches that yield high-probability guarantees are used in placeof worst-case approaches in an effort to mitigate conservatism and computational limitations.
Revised: This approach is founded on a great deal of work in classical fields such as control theory [56, 57] andsmoothed analysis [58], wherein approaches that yield high-probability guarantees are used in placeof worst-case approaches in an effort to mitigate conservatism and computational limitations.

Original: Thismitigation is of particular interest in domain generalization since generalizing to arbitrary domains isimpossible [41, 50, 51].
Revised: Thismitigation is of particular interest in domain generalization since generalizing to arbitrary domains isimpossible [40, 46, 47].

Original: Thus, motivated by this classical literature, our goal is to obtain predictorsthat are robust with high probability over domains drawn from E all , rather than in the worst-case.
Revised: Thus, motivated by this classical literature, our goal is to obtain predictorsthat are robust with high probability over domains drawn from E all , rather than in the worst-case.

Original: A distribution over environments.
Revised: A distribution over environments.

Original: We start by assuming the existence of a probability distribution 98 Q ( e ) over the set of all environments E all .
Revised: We start by assuming the existence of a probability distribution 101 Q ( e ) over the set of all environments E all .

Original: For instance, in the context of medical imaging, Qcould represent a distribution over potential changes to a hospital’s setup (see Fig. 1a) or simply adistribution over candidate hospitals.
Revised: For instance, in the context of medical imaging, Qcould represent a distribution over potential changes to a hospital’s setup (see Fig. 1a) or simply adistribution over candidate hospitals.

Original: Given that such a distribution Q exists 2 , we can think of the risk
Revised: Given that such a distribution Q exists 2 , we can think of the risk

Original: R e ( f ) as a random variable for each f ∈ F , where the randomness is engendered by the draw ofe ∼ Q .
Revised: R e ( f ) as a random variable for each f ∈ F , where the randomness is engendered by the draw ofe ∼ Q .

Original: This perspective gives rise to the following analogue of the optimization problem in (DG):
Revised: This perspective gives rise to the following analogue of the optimization problem in (DG):

Original: [Equation]
Revised: [Equation]

Original: whereess sup
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Here, ess sup denotes the essential-supremum operator from measure theory, meaning that for eachf ∈ F , ess sup Q R e ( f ) is the least upper bound on R e ( f ) that holds for almost every e ∼ Q .
Revised: Here, ess sup denotes the essential-supremum operator from measure theory, meaning that for eachf ∈ F , ess sup Q R e ( f ) is the least upper bound on R e ( f ) that holds for almost every e ∼ Q .

Original: Inthis way, the ess sup in (3.1) is the measure-theoretic analogue of the max operator in (DG), withthe subtle but critical difference being that the ess sup in (3.1) can neglect domains ofmeasure zerounder Q . For example, for discrete Q , (3.1) ignores domains which are impossible (zero probability)while (DG) does not, laying the foundation for ignoring domains which are improbable .
Revised: Inthis way, the ess sup in (3.1) is the measure-theoretic analogue to the max operator in (DG), thecaveat being that the ess sup in (3.1) can neglect domains of Q -measure zero.

Original: High-probability generalization.
Revised: High-probability generalization.

Original: Although the minimax problem in (3.1) explicitly incorporates thedistribution Q over environments, this formulation is no less conservative than (DG).
Revised: Although the minimax problem in (3.1) explicitly incorporatesthe distribution Q over environments, this formulation is no less conservative than (DG).

Original: Indeed, in manycases, (3.1) is equivalent to (DG); see Appendix B for details.
Revised: Indeed, inmany cases of interest, (3.1) is equivalent to (DG); see Appendix E for details.

Original: Therefore, rather than considering theworst-case problem in (3.1), we propose the following generalization of (3.1) which requires thatpredictors generalize with probability α rather than in the worst-case:
Revised: Therefore, rather thanconsidering the worst-case problem in (3.1), we propose the following generalization of (3.1) whichrequires that predictors generalize with probability α rather than in the worst-case:

Original: [Equation]
Revised: [Equation]

Original: The optimization problem of (3.2) formally defines what we mean by probable domain generalization.
Revised: The optimization problem of (3.2) formally defines what we mean by probable domain generalization.

Original: In particular, we say that a predictor f generalizes with risk t at level α if f has risk at most t withprobability at least α over domains sampled from Q .
Revised: In particular, we say that a predictor f generalizes with risk t at level α if f has risk at most t withprobability at least α over domains sampled from Q .

Original: In this way, the conservativeness parameter αcontrols the strictness of generalizing to unseen domains.
Revised: In this way, the conservativeness parameter αcontrols the strictness of generalizing to unseen domains.

Original: A distribution over risks.
Revised: A distribution over risks.

Original: The optimization problem presented in (3.2) offers a principled formulationfor generalizing to unseen distributional shifts governed by Q .
Revised: The optimization problem presented in (3.2) offers a principled formulationfor generalizing to unseen distributional shifts governed by Q .

Original: However, Q is often unknown inpractice and its support E all may be high-dimensional or challenging to define [22].
Revised: However, Q is often unknown inpractice and its support E all may be high-dimensional or challenging to define [22].

Original: While manyprevious works have made progress by limiting the scope of possible shift types over domains [19, 22,48], in practice, such structural assumptions are often difficult to justify and impossible to test.
Revised: While manyprevious works have made progress by limiting the scope of possible shifts over domains [19, 59, 60],in practice, such structural assumptions are often difficult to justify and impossible to test.

Original: Forthis reason, we start our exposition of QRM by offering an alternative view of (3.2) which elucidateshow a predictor’s risk distribution plays a central role in achieving probable domain generalization.
Revised: For thisreason, we start our exposition of QRM by offering an alternative view of (3.2) which elucidates howa predictor’s risk distribution plays a central role in achieving probable DG.

Original: As Q is often unknown,

Original: To begin, note that for each f ∈ F , the distribution over domains Q naturally induces 3 adistribution T f over the risks in each domain R e ( f ) .
Revised: To begin, note that for each f ∈ F , the distribution over domains Q naturally induces 3 adistribution T f over the risks in each domain R e ( f ) .

Original: In this way, rather than considering therandomness of Q in the often-unknown and (potentially) high-dimensional space of possible shiftsor interventions (Figs.
Revised: In this way, rather than considering therandomness of Q in the often-unknown and (potentially) high-dimensional space of possibledistribution shifts or interventions (Figs.

Original: 1a and 1b), one can consider it in the real-valued space of risks (Fig. 1c).
Revised: 1a and 1b), one can consider it in the real-valued space ofrisks (Fig. 1c).

Original: This is analogous to statistical learning theory, where the analysis of convergence of empirical riskminimizers (i.e., of functions) is substituted by that of a weaker form of convergence, namely that ofscalar risk functionals—a crucial step for VC theory [63].
Revised: This is analogous to statistical learning theory, where the analysis of convergence ofempirical risk minimizers (i.e., of functions) is substituted by that of a weaker form of convergence,namely that of scalar risk functionals—a crucial step for VC theory [61].

Original: From this perspective, the statistics of T fcan be thought of as capturing the sensitivity of f to different environmental shifts, summarizing theeffect of different intervention types, strengths, and frequencies.
Revised: From this perspective,the statistics of T f can be thought of as capturing the sensitivity of f to different environmentalshifts, summarizing the effect of different intervention types, strengths, and frequencies.

Original: To this end, (3.2) can be equivalentlyrewritten in terms of the risk distribution T f as follows:
Revised: To thisend, (3.2) can be equivalently rewritten in terms of the risk distribution T f as follows:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Here, F − 1 T f ( α ) denotes the inverse CDF (or quantile) function of the risk distribution T f .
Revised: Here, F − 1 T f ( α ) denotes the inverse CDF (or quantile) function of the risk distribution T f .

Original: By meansof this reformulation, we elucidate how solving (QRM) amounts to finding a predictor with minimalα -quantile risk.
Revised: By means ofthis reformulation, we elucidate how solving (QRM) amounts to finding a predictor with minimal αquantile risk.

Original: That is, (QRM) requires that a predictor f satisfy the probabilistic constraint forat least an α -fraction of the risks R ∼ T f , or, equivalently, for an α -fraction of the environmentse ∼ Q .

Original: In this way, by varying α , (QRM) recovers several notable objectives for DG as special cases,namely a worst-case problem with α = 1 and an average-case problem with α = 0.5 (median).
Revised: Also, by varying α , (QRM) recovers several notable objectives for DG as special cases.

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: For α = 1 , (QRM) is equivalent to the minimax problem of (3.1) .
Revised: For α = 1 , (QRM) is equivalent to the minimax problem of (3.1) .

Original: Furthermore,if the mean and  median of T f coincide for all f ∈ F , then for α = 1 / 2 , (QRM) is equivalent to(expected) risk minimization (RM, [63]) in which T f is the uniform distribution
Revised: Furthermore, ifthe mean and the median of T f coincide for all f ∈ F , then for α = 1 / 2 , (QRM) is equivalent to theaverage-case problem of risk minimization (RM, [61]) in which T f is the uniform distribution

Original: [Equation]
Revised: [Equation]

Original: Algorithms for Quantile Risk Minimization
Revised: Algorithms for Quantile Risk Minimization

Original: In this section we introduce two algorithms for empirically solving (QRM): empirical QRM (EQRM)and empirical superquantile RM (ESQRM).
Revised: In this section we introduce two algorithms for empirically solving (QRM): empirical QRM (EQRM)and empirical superquantile RM (ESQRM).

Original: 4.1 Empirical QRM
Revised: 4.1 Empirical QRM

Original: Estimated risk distribution.
Revised: Estimated risk distribution (cid:98) T f .

Original: In practice, given a predictor f and its empirical risksˆ R e 1 ( f ) , . . .
Revised: In practice, given a predictor f and its empirical risksˆ R e 1 ( f ) , . . .

Original: , ˆ R e m ( f ) on the m training domains, we must form an estimated risk distribution (cid:98) T f .
Revised: , ˆ R e m ( f ) on the m training domains, we must form an estimated risk distribution (cid:98) T f .

Original: Ifwe have prior knowledge about the form of T f (e.g. Gaussian), we can simply estimate the parametersof this distribution (e.g. via the sample mean and variance).
Revised: Ifwe have prior knowledge about the form of T f (e.g. Gaussian), we can simply estimate the parametersof this distribution (e.g. via the sample mean and variance).

Original: In the absence of such knowledge, wecan use nonparametric methods like kernel density estimation (KDE, [64, 65]).
Revised: In the absence of such knowledge, we canuse nonparametric methods like kernel density estimation (KDE, [62, 63]).

Original: Fig.
Revised: Fig.

Original: 1c depicts the PDF and CDF for 10 training risks when using KDE with Gaussian kernels, while Fig.
Revised: 1c depicts the PDFand CDF for 10 training risks when using KDE with Gaussian kernels, while Fig.

Original: 2 comparesthe smoothed KDE CDF (black) to the unsmoothed empirical CDF (gray).[Figure] [Figure] [Figure]
Revised: 4 of Appendix Bcompares the smoothed KDE CDF (black) to the unsmoothed empirical CDF (gray).

Original: P r ob a b ilit y α

Original: F − 1 ( α )

Original: KDEEmpirical

Original: Importance of smoothing.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, the KDE smoothing permits risk “extrapolation” [41] beyond our largest training risk, with the estimated α -quantile risk going to infinity as α → 1 (when kernels have fullsupport).
Revised: 4, the KDE smoothing permits risk “extrapolation” [40] beyond our largest training risk, withthe estimated α -quantile risk going to infinity as α goes to one (when kernels have full support).

Original: Critically, as detailed in Prop. 4.3 below, this unlocks invariantprediction for QRM. In Appendix C, we discuss the suitability of differentdata-dependent bandwidth-selection methods for our purposes, with different methods encoding different assumptions about right-tail heavinessand thus about projected OOD risk.
Revised: Importantly, different bandwidth-selection methods encode different assumptions about right-tailheaviness and thus about projected OOD risk. In Appendix B, we discuss the suitability of differentdata-dependent bandwidth-selection methods for our purposes.

Original: In practice, we use Gaussian kernelswith  the Gaussian-optimal rule [66] or Silverman’s rule-of-thumb [66].
Revised: In practice, we use Gaussian kernelswith either the Gaussian-optimal rule [64] or Silverman’s rule-of-thumb [64] for bandwidth selection.

Original: Empirical problem.
Revised: Empirical problem.

Original: Armed with a predictor’s estimated risk distribution (cid:98) T f , we can approximatelysolve (QRM) using the following empirical analogue:[Equation] [Equation]
Revised: Armed with a predictor’s estimated risk distribution (cid:98) T f , we can approximatelysolve (QRM) using the following empirical analogue:

Original: [Equation]
Revised: [Equation]

Original: Note that (4.1) depends only on known quantities, so we can compute and minimize it in practice.
Revised: Note that (4.1) depends only on known quantities, so we can compute and minimize it in practice.

Original: Alsonote that in finance, when concerned with a distribution over potential losses, the α -quantile value isknown as the value at risk  (VaR) at level α [67].
Revised: Also note that in finance, when concerned with a distribution over potential investment losses,the α -quantile value is known as the value at risk 4 (VaR) at level α and represents the maximumpossible loss after excluding all worst outcomes whose combined probability is at most 1 − α [65].

Original: Alg. 1 of Appendix E.1 details the EQRM algorithm.
Revised: Appendix D.1 gives further algorithmic details, including pseudocode in Alg. 1.

Original: Generalization bound.
Revised: Generalization bound.

Original: We now give a simplified version of our main generalization bound—
Revised: We now give a simplified version of our main generalization bound—

Original: Thm.
Revised: Thm.

Original: D.1.
Revised: G.1.

Original: The full version, stated and proved in Appendix D, provides specific finite-sample boundson ϵ 1 and ϵ 2 below, depending on the hypothesis class F , the empirical estimator (cid:91) QRM α , and theassumptions made on the possible risk profiles of hypotheses f ∈ F .
Revised: The full version, stated and proved in Appendix G, provides specific finite-sample boundson ϵ 1 and ϵ 2 below, depending on the hypothesis class F , the empirical estimator (cid:91) QRM α , and theassumptions made on the possible risk profiles of hypotheses f ∈ F .

Original: Theorem 4.1 (Simplified form of Thm.
Revised: Theorem 4.1 (Simplified form of Thm.

Original: D.1, uniform convergence) .
Revised: G.1, uniform convergence) .

Original: Given m domains and n samplesin each, then with high probability over the training data,sup
Revised: Given m domains and n samplesin each, then with high probability over the training data,sup

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where ϵ 1 → 0 as n → ∞ and ϵ 2 → 0 as m → ∞ .
Revised: where ϵ 1 → 0 as n → ∞ and ϵ 2 → 0 as m → ∞ .

Original: This  states that, for m , n sufficiently large, the empirical α -quantile risk  is a good estimate of thepopulation α -quantile risk.
Revised: This theorem states that, for m , n sufficiently large, the empirical α -quantile risk F − 1 (cid:98) T f ( α ) is a goodestimate of the population α -quantile risk F − 1 T f ( α ) .

Original: Thus, a predictor f which minimizes the empirical problem in (4.1) shouldapproximately minimize the population problem in (QRM). While many domains are required forthis generalization bound to be tight, our empirical results in § 6 demonstrate that (E)QRM performswell in practice given only a few domains.
Revised: In particular, a predictor f which minimizesthe empirical problem in (4.1) should approximately minimize the population problem in (QRM),meaning f will have risk ⪅ F − 1 (cid:98) T f ( α ) with probability ⪆ α . That is, such an f will have risk below

Original: In other words, while many domains are required for α toprecisely estimate the true quantile, α can still be used as an interpretable conservativeness-hyperparameter given only a few domains, albeit with a less precise interpretation.
Revised: F − 1 (cid:98) T f ( α ) on approximately 100 α % of test domains.

Original: Recovering the causal predictor.
Revised: Recovering the causal predictor.

Original: We now state our causal recovery results, namely that: (i) QRMlearns a minimal invariant-risk predictor as α → 1 ; and (ii) learning a minimal invariant-risk predictoris sufficient to recover the causal predictor under assumptions which are both weaker and more generalthan those of Peters et al.
Revised: We now give an informal version of the results in Appendix A.2,which show that QRM can recover the causal mechanism of Y as α goes to one.

Original: [45, Thm 2] and Krueger et al. [41, Thm 1] (Appendix A.2.3).
Revised: Proposition 4.2 (QRM can recover the causal predictor) .

Original: For intuitionon (i), we refer the reader to Appendices A.2.1 and A.2.2 where we illustrate how setting α → 1 eventually forces the risk variance over domains to 0. Together, these results provide the conditions underwhich QRM recovers the causal mechanism of Y as α → 1 , as well as new insights into the fundamental connection between invariance (in particular, invariant risks , see Appendix G.1.2) and causality. Definition 4.2. A predictor is said to be an invariant-risk predictor if its risk is equal almost surelyacross domains (i.e., Var e ∼ Q
Revised: Assume: (i) data is generated from a linear

Original: [ R e ( f )] =0 ).
Revised: SEM Y =

Original: A predictor is said to be a minimal invariant-riskpredictor if it achieves the minimal possible risk over all possible invariant-risk predictors. Proposition 4.3 (QRM learns a minimal invariant-risk predictor as α → 1 , informal version of Props.
Revised: β X + N , with coefficient β ∈ R d and noise distribution N constant across environments,

Original: A.4 and A.5) . Assume: (i) F contains an invariant-risk predictor with finite training risks;and (ii) no arbitrarily-negative training risks.
Revised: E [ N ] = 0 , and N independent of X ; (ii) F is the class of linear predictors, indexed by ˆ β ∈ R d ; (iii)all causes of Y are observed; (iv) the loss ℓ is squared-error; and (v) the distribution of E X ∼ e

Original: Then, as α → 1 , Gaussian and kernel QRM predictors(the latter with certain bandwidth-selection methods) converge tominimal invariant-risk predictors. Theorem 4.4 (The causal predictor is (often) the only minimal invariant-risk predictor) .
Revised: [ X ](over environments e ∼ Q ) has a density with respect to the Lebesgue measure on R d .

Original: Assume that:(i)
Revised: Then, for both

Original: Y is generated from a linear SEM, Y = β ⊺ X + N , with coefficients β ∈ R d and noise N havingconstant variance and E [ N ] = 0 across domains; (ii) F is the class of linear predictors, indexed byˆ β ∈ R d ; (iii) all components of X are observed; (iv) the loss ℓ is squared-error; and (v) the systemof equations [Equation] has the unique solution x = 0 .
Revised: Gaussian estimators of T f and kernel-density estimators of T f with certain bandwidth-selectionmethods, for m sufficiently large, the QRM predictor

Original: If ˆ β is a minimal invariant-risk predictor, then ˆ β = β  .
Revised: ˆ β α → β as α → 1 , with probability 1 .

Original: In Appendix A.2, we prove the formal versions of Prop. 4.3 (Props. A.4 and A.5) along with Thm.
Revised: In Appendix A.2 we prove Proposition 4.2 and discuss both bandwidth-selection methods andconnections to similar results like that of Krueger et al.

Original: 4.4,provide
Revised: [40, Thm 1].

Original: some intuition and concrete examples regarding Assumption (v) of Thm. 4.4, and relate our αparameter to the β parameter of VREx [41, Eq. 8] for the special case of Gaussian estimators of T f .
Revised: In § 6.1, we demonstrate thatthis result holds empirically. Finally, in Appendix A.2.1, we relate our α parameter to the β parameterof VREx [40, Eq. 8] for the special case of Gaussian estimators of T f .

Original: 4.2 Empirical SQRM
Revised: 4.2 Empirical SQRM

Original: Although the formulation presented in (4.1) is intuitive and interpretable, the quantile function isnotoriously difficult to optimize [68].
Revised: Although the formulation presented in (4.1) is intuitive and interpretable, the quantile function isnotoriously difficult to optimize [66].

Original: Indeed, in many cases of interest, F − 1 can be nonconvex anddiscontinuous, which induces an unfavorable optimization landscape [69, 70].
Revised: Indeed, in many cases of interest, F − 1 can be nonconvex anddiscontinuous, which induces an unfavorable optimization landscape [67, 68].

Original: To circumvent this,we propose an alternative scheme in which we optimize the superquantile (SQ)—also known as theconditional value at risk (CVaR) or expected tail loss —at level α , which is defined 4 as follows:
Revised: To circumvent this,we propose an alternative scheme in which we optimize the superquantile (SQ)—also known as theconditional value at risk (CVaR) or expected tail loss —at level α , which is defined 5 as follows:

Original: [Equation]
Revised: [Equation]

Original: SQ α can be seen as the conditional expectation of a random variable R subject to R being larger thanthe α -quantile F − 1 ( α ) .
Revised: SQ α can be seen as the conditional expectation of a random variable R subject to R being larger thanthe α -quantile F − 1 ( α ) .

Original: In our case, where R represents the statistical risk on a randomly-sampledenvironment, SQ α can be seen as the expected risk in the worst 100 · ( 1 − α ) % of cases/domains.
Revised: In our case, where R represents the statistical risk on a randomly-sampledenvironment, SQ α can be seen as the expected risk in the worst 100 · ( 1 − α ) % of cases/environments.

Original: Importantly, unlike the quantile, the superquantile has many desirable mathematical properties: (i) itis continuous in α ; (ii) it is jointly convex in R and α ; and (iii) it is the tightest convex surrogate forthe inverse CDF [71, § 2].
Revised: Importantly, unlike the quantile, the superquantile has many desirable mathematical properties: (i) itis continuous in α ; (ii) it is jointly convex in R and α ; and (iii) it is the tightest convex surrogate forthe inverse CDF [69, § 2].

Original: Therefore, we also consider the following optimization problem:
Revised: Therefore, we also consider the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: We call the empirical analogue, which uses the empirical density rather than T f , empirical SQRM.
Revised: We call the empirical analogue, which uses the empirical density rather than T f , empirical SQRM.

Original: Alg. 2 of Appendix E.1 details the ESQRM algorithm.
Revised: Appendix D.1 gives further algorithmic details, including pseudocode in Alg. 2.

Original: Connections to existing problems.
Revised: Connections to existing problems.

Original: It is straightforward to show that (SQRM) recovers the RMproblem in (3.3) and the minimax problem in (3.2) as special cases with α = 0 and α = 1 respectively.
Revised: It is straightforward to show that (SQRM) recovers the RMproblem in (3.3) and the minimax problem in (3.2) as special cases with α = 0 and α = 1 respectively.

Original: Moreover, in Appendix F, we exploit the well-known duality properties of CVaR to showan alternative, view of (SQRM) as a distributionally-robust optimization problem, admitting closeconnections to previous work, including GroupDRO [48]; see Prop. F.1 and F.2 for details.
Revised: Moreover, in Appendix F, we exploit the well-known duality properties of CVaR to show an alternativeview of (SQRM) as a distributionally-robust optimization problem, admitting close connections toprevious work [60]; see Propositions F.1 and F.2 for details.

Original:  Related work
Revised:  Related work

Original: Robust optimization in DG.
Revised: Robust optimization in DG.

Original: Throughout this paper, we follow an established line of work (seee.g.
Revised: Throughout this paper, we follow an established line of work (seee.g.

Original: , [9, 41, 49]) which formulates the DG problem through the lens of robust optimization [47].
Revised: , [9, 40, 70]) which formulates the DG problem through the lens of robust optimization [71].

Original: Tothis end, various algorithms have been proposed for solving constrained [22] and distributionallyrobust [48] variants of the worst-case problem in (DG).
Revised: Tothis end, various algorithms have been proposed for solving constrained [59] and distributionallyrobust [60] variants of the worst-case problem in (DG).

Original: Indeed, this robust formulation has a firmfoundation in the broader machine learning literature, with notable works in adversarial robustness [72–76 ] and fair learning [77, 78] employing similar formulations.
Revised: Indeed, this robust formulation has a firmfoundation in the broader machine learning literature, with notable works in adversarial robustness [72–76] and fair learning [77, 78] employing similar formulations.

Original: Unlike these past works, we considera robust but non-adversarial formulation for DG, where predictors are trained to generalize with highprobability rather than in the worst case.
Revised: Unlike these past works, we considera robust but non-adversarial formulation for DG, where predictors are trained to generalize withhigh probability rather than in the worst case.

Original: Moreover, the majority of this literature—both within andoutside of DG—relies on specific structural assumptions (e.g. covariate shift) on the types of possibleinterventions or perturbations.
Revised: Moreover, the majority of this literature—both withinand outside of DG—relies on specific structural assumptions (e.g. covariate shift) on the types ofpossible interventions or perturbations.

Original: In contrast, we make the weaker and more flexible assumption ofiid-sampled domains, which ultimately makes use of the data/domains to determine the types of shiftsthat are possible or probable . We further discuss this important difference in § 7.
Revised: In contrast, QRM and SQRM make an assumption on theenvironment-generating process (we further discuss this important difference in § 7).

Original: Other approaches to DG.
Revised: Other approaches to DG.

Original: Outside of robust optimization, many algorithms have been proposed forthe DG setting which draw on insights from a diverse array of fields, including approaches based ontools from meta-learning [40, 79–82], kernel methods [83, 84], and information theory [49].
Revised: Outside of robust optimization, many algorithms have been proposed forthe DG setting which draw on insights from a diverse array of fields, including approaches basedon tools from meta-learning [39, 79–82], kernel methods [83, 84], and information theory [85, 86].

Original: Alsoprominent are works that design regularizers to generalize OOD [85–87] and works that seek domaininvariant representations [88–90].
Revised: Also prominent are works that design regularizers to generalize OOD [87–89] and works that seekdomain-invariant representations [90–93].

Original: However, the majority of these works do not explicitly relate thedomains seen during training and test time, which is a key feature of our approach.
Revised: However, the majority of these works do not explicitlyrelate the environments seen during training and test time, which is a key feature of our approach.

Original: Moreover, many ofthese works employ hyperparameters which are difficult to interpret, which has no doubt contributedto the well-established model-selection problem in DG [38].
Revised: Moreover, many of these works employ hyperparameters which are difficult to interpret, which hasno doubt contributed to the well-established model-selection problem in DG [37].

Original: In contrast, in our framework, α can beeasily interpreted in terms of the quantiles of the risk distribution.
Revised: In contrast, in ourframework, α can be clearly interpreted in terms of the quantiles of the risk distribution T f .

Original: Finally, while many of the bestperforming DG methods provide no theoretical results in the nonlinear setting (e.g.[9, 41, 82, 91]),

Original: QRM has both strong theoretical and empirical results in the nonlinear setting (see § 6).

Original: High-probability generalization.
Revised: High-probability generalization.

Original: As noted in § 3, relaxing worst-case problems in favor of probabilistic ones has a long history in control theory [42, 60, 61, 92, 93], operations research [94],and smoothed analysis [62].
Revised: As noted in § 3, relaxing worst-case problems in favor of probabilistic ones has a long history in control theory [41, 56, 57, 94, 95], operations research [96],and smoothed analysis [58].

Original: Recently, this paradigm has been applied to several areas of machinelearning, including perturbation-based robustness [95, 96], fairness [97], active learning [98], andreinforcement learning [99, 100].
Revised: Recently, this paradigm has been applied to several areas of machinelearning, including perturbation-based robustness [97, 98], fairness [99], active learning [100], andreinforcement learning [101, 102].

Original: However, it has not yet been applied to domain generalization.[Figure]
Revised: However, it has not yet been applied to domain generalization.

Original: Quantile minimization.
Revised: Quantile minimization.

Original: In financial optimization, VaR and CVaR [67, 69, 101] are central to theliterature surrounding portfolio risk management, with numerous applications spanning bankingregulations and insurance policies [68, 102].
Revised: In financial optimization, VaR and CVaR [65, 67, 103] are central to theliterature surrounding portfolio risk management, with numerous applications spanning bankingregulations and insurance policies [66, 104].

Original: In statistical learning theory, several recent papers havederived uniform convergence guarantees in terms of alternative risk functionals besides expectedrisk [98, 103–105].
Revised: In statistical learning theory, several recent papers havederived uniform convergence guarantees in terms of alternative risk functionals besides expectedrisk [100, 105–107].

Original: These results focus on CVaR and closely related functionals that can be writtenin terms of expectations over the loss distribution.
Revised: These results focus on CVaR and closely related functionals that can bewritten in terms of expectations over the loss distribution.

Original: In contrast, our uniform convergence guarantee(Theorem D.1) shows uniform convergence of VaR , which cannot be written assuch an expectation; this necessitates stronger conditions to obtain uniform convergence, which ultimately suggestregularizing the estimated risk distribution (e.g. by kernel smoothing).
Revised: In contrast, our uniform convergenceguarantee (Theorem G.1) shows uniform convergence of VaR , which cannot be written as an suchexpectation; this necessitates stronger conditions to obtain uniform convergence, which ultimatelysuggest regularizing the estimated risk distribution (e.g., by kernel smoothing).

Original: Invariant prediction and causality.
Revised: Invariant prediction and causality.

Original: As discussed in § 2,  recent works have leveraged different formsof invariance across domains to discover causal relationships which, under the invariant mechanismassumption [54], generalize to new domains [9, 41, 45, 52, 106–108]. In particular, like QRM (asα → 1 ), Krueger et al. [41] leverage invariant risks , while Arjovsky et al. [9] leverage invariantfunctions or regression coefficients.
Revised: As discussed in § 2, under the invariant mechanism assumption [50], recent works have leveraged different forms of invariance across environments to discovercausal relationships which, by assumption, generalize to new environments [9, 40, 44, 48, 108–110].

Original: We further discuss this important distinction in Appendix G.1.2. More recently, noting that causal parameters are often too conservative and thus that performingwell out-of-distribution can require non-causal relationships, Rothenhäusler et al. [109] proposed Anchor regression for interpolating between causal and predictive parameters.
Revised: More recently, noting that causal parameters are often too conservative and thus that performingwell out-of-distribution often requires leveraging some non-causal relationships, Rothenhäusler et al.[111] proposed Anchor regression for interpolating between causal and predictive parameters usingan interpretable parameter.

Original: However, the assumedshift types are even more restrictive than in the works above, despite nonlinear extensions [51].
Revised: However, the assumed shift types are even more restrictive than theaforementioned methods, despite nonlinear extensions [47].

Original:  Experiments
Revised:  Experiments

Original: We now evaluate our algorithms on synthetic  and real-world datasets. These evaluations illustratethe importance of multiple test domains in DG benchmarks, as well as the benefits of our algorithmsin practice.
Revised: We now evaluate our algorithms on: synthetic datasets requiring OOD generalization; and real-worlddatasets from the WILDS [12] package, illustrating the practical benefits of our algorithms andalso the importance of multiple test domains in DG benchmarks.

Original: Appendices E and G report further results and experimental details.
Revised: Further results can be found in Appendix C.3, while full experimental details are given in Appendix D. Code will be made available. [Figure]

Original: 6.1 Synthetic datasets
Revised: 6.1 Synthetic datasets

Original: Linear regression.
Revised: Linear regression.

Original: We first consider a  linear regression dataset based on thefollowing linear SCM:
Revised: We first consider a toy linear regression dataset based on the linear SCM of

Original: [Equation]
Revised: Ex.

Original: with N j ∼ N ( 0, σ 2 j ) .
Revised: A.3.

Original: Here we have two features: one cause X 1 = X cause and one effect X 2 =
Revised: Here we have two features—one cause X 1 = X cause ∈

Original: X effectof Y  .
Revised: Pa ( Y ) and one effect X 2 = X effect .

Original: By fixing σ 21 = 1 and σ 2 Y = 2 across domains but sampling σ 2 ∼ LogNormal ( 0, 0.5 ) , we create [Table] [Table] a dataset in which X 2 is more predictive of Y than X 1 but less stable.
Revised: By fixing σ 21 = 1 and σ 2 Y = 2 across environments but sampling σ 2 ∼ LogNormal ( 0, 0.5 ) , we createa dataset in which X 2 is more predictive of Y than X 1 but less stable.

Original: Importantly, as we knowthe true distribution over domains Q ( e ) =
Revised: Importantly, as we know thetrue distribution over environments Q ( e )=

Original: LogNormal ( σ [Table] 2 ; 0, 0.5 ) , we know the true risk quantiles.
Revised: LogNormal ( σ e 2 ; 0, 0.5 ) , we know the true risk quantiles.

Original: Fig.
Revised: Fig.

Original: 3 depicts results for different α ’s with m = 1000 domains and n = 100000 samples in each,using the mean-squared-error(MSE) loss.
Revised: 2 depicts results for different α ’s with m = 1000 environments and n = 100000 samples in each,using the mean-squared-error loss.

Original: Here we see that: A:for eachtrue quantile (x-axis),the corresponding α has the lowest risk (y-axis), confirming that the empirical α -quantile risk isa good estimate of the population α -quantile risk; B: As α → 1 , the estimated risk distribution off α approaches an invariant (or Dirac delta) distribution centered on the risk of the causal predictor; C: the regression coefficients approach those of the causal predictor as α → 1 , trading predictiveperformance for robustness; and D: reducing the number of domains m reduces the accuracy of theestimated α -quantile risks.
Revised: Here we see that: A: each α is optimal at the correspondingquantile, confirming that the empirical α -quantile risk is a good estimate of the population α -quantilerisk; B: As α → 1 , the estimated risk distribution of f α approaches an invariant (or Dirac delta)distribution centered on the risk of the causal predictor; C: the regression coefficients approach thoseof the causal predictor as α → 1 , trading predictive performance for robustness; and D: reducing thenumber of domains m reduces the accuracy of the estimated α -quantile risks.

Original: In Appendix G.1, we: (i) depict the risk CDFs corresponding to plot B above, and discuss how they depict the predictors’ risk-robustness curves (G.1.1); and (ii) discussthe solutions of QRM on datasets in which σ 21 , σ 22 and/or σ 2 Y change over domains, compared toexisting invariance-seeking algorithms like IRM [9] and VREx [41] (G.1.2).
Revised: In Appendix C.1, weadditionally: (i) depict the risk CDFs corresponding to plot B above, and discuss how they depicta predictor’s risk-robustness curve (see Fig. 5); and (ii) compare QRM to existing algorithms ondatasets in which σ 21 , σ 22 or σ 2 Y change over environments. [Table]

Original: ColoredMNIST.
Revised: ColorMNIST.

Original: We next consider the ColoredMNIST or CMNIST dataset [9].
Revised: We next consider the ColorMNIST or CMNIST dataset of

Revised: Arjovsky et al.

Original: Here, the MNISTdataset is used to construct a binary classification task (0–4 or 5–9) in which digit color (red or green)is a highly-informative but spurious feature.
Revised: [9], where the MNIST dataset is used to construct a binaryclassification task (0–4 or 5–9) in which digit color (red or green) is ahighly-informative but spurious feature.

Original: In particular, the two training domains are constructedsuch that red digits have an 80% and 90% chance of belonging to class 0, while the single test domainis constructed such that they only have a 10% chance.
Revised: In particular, the two trainingdomains are constructed such that red digits have an 80% and 90% chanceof belonging to class 0, while the single test domain is constructed suchthat red digits only have a 10% chance (labeled -90%).

Original: The goal is to learn an invariant predictorwhich uses only digit shape—a stable feature having a 75% chance of correctly determining the classin all 3 domains.
Revised: The goal is to learnan invariant predictor which uses only digit shape—a stable feature havinga 75% chance of correctly determining the class in all 3 domains.

Original: We compare with IRM [9],GroupDRO [48], SD [110], IGA [111] and VREx [41]using: (i) random initialization (Xavier method [112]); and (ii) random initialization followed byseveral iterations of ERM.
Revised: Table 1compares the performance of QRM to IRM [9], VREx [40] and GroupDRO [60] when using thestandard hyperparameters and test-domain model selection of DomainBed [37].

Original: The ERM initialization or pretraining directly corresponds to the delicatepenalty “annealing” or warm-up periods used by most penalty-based methods [9, 41, 110, 111]. Forall methods, we use a 2-hidden-layer MLP with 390 hidden units, the Adam optimizer, a learningrate of 0.0001 , and dropout with p = 0.2 . We sweep over five penalty weights for the baselines andfive α ’s for QRM/SQRM. See Appendix E.2 for more details.
Revised: Here we see that

Original: Table 1 shows that: (i) all methodsstruggle without ERM pretraining, explaining the need for penalty-annealing strategies in previousworks and corroborating the results of [113, Table 1]; (ii) with ERM pretraining, QRMmatchesor outperforms IRM andVREx, even approaching oracle performance (that of ERM trained ongrayscale digits).These results suggest ERM pretraining as an effective strategy for DG methods—astrategy we employ in the next section.
Revised: QRM outperforms existing methods and even approaches oracle performance (ERM trained ongrayscale digits).

Original: Fig.
Revised: Fig.

Original: 3 depicts the behavior of QRM with different α s. 
Revised: 2 then depicts the behavior of QRM with different α s. In particular, we see

Original: Here we see that: E: increasing α leadsto more consistent performance across domains, eventually forcing the model to ignore color andfocus on invariant, shaped-based prediction; and F: a predictor’s estimated accuracy-CDF depictsits accuracy-robustness curve, just as its estimated risk-CDF depicts its risk-robustness curve. Notethat α = 0.5 gives the best worst-case risk over the two training domains—the preferred solutionof DRO [48]—while α → 1 sacrifices risk for increased invariance or robustness. 6.2 DomainBed datasets: between synthetic and real-world shifts
Revised: [Table]

Original: We next consider two large-scale datasets from the DomainBed [38] suite: PACS and VLCS . Whileboth of these datasets encode real-world shifts, each contains only four domains – three for trainingand one for testing – and therefore it is not possible to analyze the tail performance of our algorithms. However, in line with a litany of recent work, we offer a comparison of QRM to a range of standardbaselines in Table 2 in terms of the average performance across each choice of test domain; we
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure] [Table] [Table] [Table]
Revised: that: E: increasing α leads to more consistent performance across domains, eventually forcing themodel to ignore color and focus on invariant, shaped-based prediction; and F: a predictor’s estimatedaccuracy-CDF depicts its accuracy-robustness curve, just as its estimated risk-CDF F (cid:98) T f depicts its riskrobustness curve.

Original: [Figure] include a full set of results with additional baselines in Appendix G.2. Observe that on PACS , we beatstate-of-the-art algorithms across all three model-selection methods, with a notable 2-percentage-pointimprovement for the cross-validation selection method. On VLCS , we match the state-of-the-art, witha slight improvement of nearly 1-percentage-point for the oracle selection method.
Revised: Note that α = 0.5 gives the best worst-case risk over the two training domains—thepreferred solution of DRO [60]—while α → 1 sacrifices risk for increased robustness or invariance.

Original: 6.3 Real-world datasets
Revised: 6.2 Real-world datasets

Original: We now evaluate our methods on the real-world or in-the-wild distribution shifts of WILDS [12].
Revised: We now evaluate our methods on the real-world or in-the-wild distribution shifts of WILDS [12].

Original: We focus our evaluation on iWildCam [114] and OGB-MolPCBA [115, 116]—two large-scale classification datasets which have numerous test domains and thus facilitate a comparison of the riskdistributions and their quantiles.
Revised: We focus our evaluation on two large-scale classification datasets—iWildCam [112] and OGBMolPCBA [113, 114]—both of which have numerous test domains, facilitating a comparison of therisk quantiles and distributions.

Original: Additional comparisons (e.g. using average accuracy) can be foundin Appendix G.3.
Revised: Additional comparisons (e.g. using average accuracy ) can be foundin Appendix C.2.

Original: Our results demonstrate that, across two distinct data types (images and moleculargraphs), QRM and SQRM offer superior quantile and superquantile performance respectively. iWildCam.
Revised: Our results demonstrate that, across two distinct data types (images and moleculargraphs), QRM and SQRM offer superior quantile and superquantile performance respectively.iWildCam.

Original: We first consider the iWildCam image-classification dataset which has 243 trainingdomains and 48 test domains. Here, the label Y is 1 of 182 different animal species  and the domaine is the camera trap which took the photo.
Revised: We first consider the iWildCam dataset, where the task is to classify animal species fromcamera-trap images, given 243 training domains and 48 test domains.

Original: In Table 3, we observe that QRM α does indeed tendto optimize the α -risk quantile, with larger α s during training resulting in lower test-time risksat the corresponding quantiles.
Revised: In Table 2, we observe that QRM α does indeed tend to optimize the α -risk quantile, with larger α s during training resulting inlower test-time risks at the corresponding quantiles.

Original: Similarly, Table 5 shows that SQRM α effectively optimizes thesuperquantile function, with α = 0.75 yielding particularly low values relative to ERM.
Revised: Similarly, Table 4 shows that SQRM α effectivelyoptimizes the superquantile function, with α = 0.75 yielding particularly low values relative to ERM.

Original: In the twoleftmost panes of Fig. 4, we plot the (smoothed) test-time risk distributions.
Revised: In the two leftmost panes of Figure 3, we plot the (smoothed) test-time risk distributions.

Original: Here we see a cleartrend: as α increases, the tails of the risk distribution tend to drop below ERM, which corroboratesthe superior quantile and superquantile performance reported in Tables 3 and 5.
Revised: Here wesee a clear trend: as α increases, the tails of the risk distribution tend to drop below ERM, whichcorroborates the superior quantile and superquantile performance reported in Tables 2 and 4.

Original: Note that, in both Tables 3 and 5, QRM and SQRM tend to record lower average risks than ERM.
Revised: Note that, in both Tables 2 and 4, QRM and SQRM tend to record lower average risks than ERM.

Original: This has several plausible explanations.
Revised: This has several plausible explanations.

Original: First, the number of testing domains (48) is relatively small, which could result in a biased sample with respect to the training domains.
Revised: First, the number of testing domains (48) is relatively small,which could result in a biased sample with respect to the training domains.

Original: Second, the test domainsmay not represent i.i.d.
Revised: Second, the test domainsmay not represent i.i.d.

Original: draws from Q , with WILDS [12] test domains tending to be more challenging.
Revised: draws from Q , with WILDS [12] test domains tending to be more challenging.

Original: OGB-MolPCBA.
Revised: OGB-MolPCBA.

Original: We next consider the OGB-MolPCBA (or OGB ) dataset, which is a molecular graphclassification benchmark containing 44,930 training domains and 43,793 test domains with an averageof 3.6 samples per domain.
Revised: We next consider the OGB-MolPCBA (or OGB) dataset, which is a moleculargraph-classification benchmark containing 44,930 training domains and 43,793 test domains.

Original: Tables 4 and 6 show that ERM achieves the lowest average test risk on OGB ,in contrast to the iWildCam results, while QRM α and SQRM α still achieve stronger quantile and superquantile performance.
Revised: In contrast to our results on iWildCam, Table 3 and 5 show that ERM achieves the lowest averagetest risk on OGB, which supports the hypothesis that having access to a large sample of domains isnecessary to generalize OOD. Both of these tables also indicate that QRM α and SQRM α achievestronger quantile and superquantile performance relative to ERM.

Original: Of particular note is the fact that our methods significantly outperform ERMwith respect to worst-case performance (columns/quantiles labeled 1.0); when QRM α and SQRM αare run with large values of α , we reduce the worst-case risk by more than a factor of two.
Revised: Of particular note is the fact that ourmethods significantly outperform ERM with respect to worst-case performance (columns/quantileslabeled 1.0); when QRM α and SQRM α are run with large values of α , we reduce the worst-caserisk by more than a factor of two.

Original: In Fig. 4,we again see that the risk distributions of QRM α and SQRM α have lighter tails than that of ERM.
Revised: In Figure 3, we again see that the risk distributions of QRM α and SQRM α have lighter tails than that of ERM.

Original: A new evaluation protocol for DG.
Revised: The importance of multiple test domains.

Original: The analysis provided in Tables 3-6 and Fig. 4 diverges from thestandard evaluation protocol in  DG  [12, 38].
Revised: The analysis provided in Tables 2-5 and Figure 3diverges from the standard evaluation protocol in the DG literature [12, 37].

Original: Rather than evaluating an algorithm’s performance onaverage across test domains, we seek to understand the distribution of its performance —particularly inthe tails by means of the quantile and superquantile functions.
Revised: Rather than evaluatingan algorithm’s performance on average across test domains, we seek to understand the distribution ofits performance, in particular in the tails by means of the quantile and superquantile functions.

Original: This new evaluation protocol lays barethe importance of  multiple test domains in DG benchmarks, allowing predictors’ risk distributions tobe analyzed and compared.
Revised: Thisnew evaluation protocol lays bare the importance of having access to multiple test domains, which isnecessary for analyzing the tail behavior of a predictor’s risk distribution.

Original: Indeed, as DG is neither a worst-case nor an average-case problem, solelyreporting a predictor’s worst or average risk  over test domains can be misleading when assessing itsability to generalize OOD (as shown in Tables 3-6).
Revised: Indeed, as (DG) is not anaverage-case problem, solely reporting a predictor’s test risk (or accuracy) averaged over multiple testdomains can be misleading when assessing its ability to generalize OOD (as shown in Tables 2-5).

Original: This underscores the necessity of incorporatingtail risk measures into amore holistic evaluation protocol for DG,ultimately providing a morenuanced and complete picture of a predictor’s ability to generalize OOD.
Revised: This underscores the necessity of incorporating tail risk measures into a holistic evaluation protocolfor DG, providing a more nuanced and complete picture of a predictor’s ability to generalize OOD.

Original: In practice, which measureis preferred will depend on the application-of-interest, e.g. via a human-specified quantile-of-interest.

Original:  Discussion
Revised:  Discussion

Original: On the assumption of i.i.d.
Revised: On the assumption of i.i.d.

Original: domains.
Revised: environments.

Original: For α to approximate the probability of generalizing, trainingand test domains must be i.i.d.-sampled.
Revised: For α to approximate the probability of generalizing,training and test environments must be i.i.d.-sampled.

Original: While this is rarely true in practice—e.g. hospitals haveshared funders, service providers, foundation dates, etc.—we can better satisfy this assumption bysubscribing to a new data collection process in which we collect training data from domains whichare representative of how the underlying system tends to change.
Revised: While this is rarely true in practice—e.g.hospitals have shared funders, service providers, foundation dates, etc.—we can better satisfy thisassumption by prescribing to a new data collection process in which we collect training environmentswhich are representative of how the underlying system can change.

Original: For example: (i) randomly select100 US hospitals; (ii) gather and label data from these hospitals; (iii) train our system with thedesired α ; (iv) deploy our system to all US hospitals where it will be successful with probability≈ α .
Revised: For example: (i) randomlyselect 100 US hospitals; (ii) gather and label data from these hospitals; (iii) train our system withthe desired α ; (iv) deploy our system to all US hospitals where it will be successful with probability≈ α .

Original: While this process may seem expensive, time-consuming and vulnerable (e.g. new hospitals),it offers a promising path to machine learning systems which generalize with high probability .
Revised: While this process may seem expensive, time-consuming and vulnerable (e.g. new hospitals),it offers a promising path to machine learning systems which generalize with high probability .

Original: Moreover, it is worth noting the alternative: prior works achieve generalization by assuming thatonly particular types of shifts can occur, e.g. covariate shifts [22, 117, 118], label shifts [118, 119],concept shifts [120], measurement shifts [19], mean shifts [109], shifts which leave the mechanism of
Revised: Moreover, it is worth noting the alternative: prior works achieve generalization by assuming thatonly particular types of shifts can occur, e.g. covariate shifts [59, 115, 116], label shifts [116, 117],concept shifts [118], measurement shifts [19], mean shifts [111], shifts which leave the mechanism of

Original: Y invariant [9, 39, 41, 45], etc.
Revised: Y invariant [9, 38, 40, 44], etc.

Original: In real-world settings, where the shift mechanisms are often unknown,such assumptions are both difficult to justify and impossible to test.
Revised: In real-world settings, where the shift mechanisms are often unknown,such assumptions are both difficult to justify and impossible to test.

Original: Future work could look to relaxthe i.i.d.-domains assumption by leveraging knowledge of domain dependencies (e.g. time). Interpretable model selection.
Revised: Future work could look to relaxthe i.i.d. assumption by leveraging knowledge of environment dependencies (e.g. time).α controls the risk-robustness trade-off.

Original: α approximates the probability with which our predictor willgeneralize with risk below the associated α -quantile value.
Revised: α approximates the probability with which our predictorwill generalize with risk below the associated α -quantile value.

Original: Thus, α represents an interpretableparameterization of the  risk-robustness trade-off.
Revised: Thus, α represents an interpretableparameterization of the often-inherent risk-robustness trade-off.

Original: Such interpretability is critical for model selectionin DG, and for practitioners with application-specific requirements on performance and/or robustness.
Revised: Such interpretability may aidpractitioners meet application-specific requirements on performance and/or robustness.

Original: The wider value of risk distributions.
Revised: The wider value of risk distributions.

Original: As demonstrated in § 6, a predictor’s risk distribution hasvalue beyond quantile-minimization—it estimates the probability associated with each level of risk.
Revised: As demonstrated in § 6, a predictor’s risk distribution hasvalue beyond quantile-minimization—it estimates the probability associated with each level of risk.

Original: Thus, regardless of the algorithm used, risk distributions can be used to analyze trained predictors.
Revised: Thus, regardless of the algorithm used, risk distributions can be used to analyze trained predictors.

Revised: A new evaluation protocol for DG.

Revised: We have argued that the goal of DG should be to learn predictorswhich perform well with high probability , rather than in the worst-case or on-average.

Revised: Evaluating thisnewly-sought ability requires new evaluation protocols which provide a more complete picture ofpredictor performance, going beyond the average or worst-case.

Revised: To this end, we have proposed usingmultiple test domains to evaluate predictor performance at different quantiles.

Original: Conclusion
Revised: Conclusion.

Original: We have presented QRM for achieving probable domain generalization, motivated by the argumentthat DG should seek predictors which generalize with high probability rather than in the worst-case oron-average.
Revised: We have presented QRM for achieving probable domain generalization, motivated bythe argument that DG should seek predictors which generalize with high probability rather than in theworst-case or on-average.

Original: By explicitly relating training and test domains as draws from the same underlying metadistribution, we proposed to learn predictors with minimal α -quantile risk under the training domains.
Revised: In particular, by relating training and test domains as draws from the sameunderlying meta-distribution, we proposed to learn predictors with minimal α -quantile risk underthe training domains.

Original: Theoretically, we proved that α does indeed approximate the probability of generalizing and that QRMrecovers the causal predictor as α → 1 .
Revised: Theoretically, we proved that α does indeed approximate the probability ofgeneralizing when using the empirical version EQRM, and that EQRM α recovers the causal predictoras α → 1 .

Original: Empirically, we demonstrated the importance of  multiple testdomains and tail performance in DG benchmarks, and that our algorithms perform well in practice.
Revised: Empirically, we demonstrated the importance of having multiple test domains to evaluatethe quantile performance of DG algorithms, and that our algorithms perform well in practice.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section X.•
Revised: [Yes] See Section X.•

Original: Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.•
Revised: [No] The code and the data are proprietary.•

Original: Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes] See Appendix H.(c)
Revised: [Yes] See Appendix H.(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [N/A](d)
Revised: [N/A](d)

Original: Have you read the ethics review guidelines and ensured that your paper conforms to them?[Yes]2.
Revised: Have you read the ethics review guidelines and ensured that your paper conforms to them?[Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes](b) Did you include complete proofs of all theoretical results?
Revised: [Yes](b) Did you include complete proofs of all theoretical results?

Original: [Yes]3.
Revised: [Yes]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?

Original: [Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?
Revised: [Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?

Original: [Yes](c)
Revised: [Yes](c)

Original: Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
Revised: Did you report error bars (e.g., with respect to the random seed after running experiments [Table] multiple times)?

Original: [No]
Revised: [No]

Original: [Table]
Revised: [Table]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type ofGPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type ofGPUs, internal cluster, or cloud provider)?

Original: [Yes] [Table]
Revised: [Yes]4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a)
Revised: (a)

Original: If your work uses existing assets, did you cite the creators?
Revised: If your work uses existing assets, did you cite the creators?

Original: [Yes](b) Did you mention the license of the assets?
Revised: [Yes](b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: [Table]
Revised: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table]
Revised: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]
Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applica- [Table] ble?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] [Table]
Revised: [N/A](b)

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board
Revised: Did you describe any potential participant risks, with links to Institutional Review Board

Original: [Table]
Revised: [Table]

Original: (IRB) approvals, if applicable?
Revised: (IRB) approvals, if applicable?

Original: [N/A](c)
Revised: [N/A]

Original:  Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?
Revised: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on [Table] participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]