Original: Probable Domain Generalization via Quantile Risk Minimization
Revised: Probable Domain Generalization via Quantile Risk Minimization

Original: Abstract
Revised: Abstract

Original: Domain generalization (DG) seeks predictors which perform well on unseen testdistributions by leveraging labeled training data from multiple related distributionsor domains .
Revised: Domain generalization (DG) seeks predictors which perform well on unseen testdistributions by leveraging labeled training data from multiple related distributionsor domains .

Original: To achieve this, DG is commonly formulated as a worst-case optimization problem over the set of all possible domains.
Revised: To achieve this, DG is commonly formulated as a worst-case optimization problem over the set of all possible domains.

Original: However, with worst-case shiftsunlikely in practice, this generally leads to overly-conservative solutions.
Revised: However, with worst-case shiftsunlikely in practice, this generally leads to overly-conservative solutions.

Original: In fact, arecent study found that no DG algorithm outperformed empirical risk minimizationin terms of average performance.
Revised: In fact, arecent study found that no DG algorithm outperformed empirical risk minimizationin terms of average performance.

Original: In this work, we argue that DG is neither a worstcase problem nor an average-case problem, but rather a probabilistic one.
Revised: In this work, we argue that DG is neither a worstcase problem nor an average-case problem, but rather a probabilistic one.

Original: To thisend, we propose a probabilistic framework for DG, which we call Probable Domain
Revised: To thisend, we propose a probabilistic framework for DG, which we call Probable Domain

Original: Generalization , wherein our key idea is that distribution shifts seen during trainingshould inform us of probable shifts at test time.
Revised: Generalization , wherein our key idea is that distribution shifts seen during trainingshould inform us of probable shifts at test time.

Original: To realize this, we explicitly relatetraining and test domains as draws from the same underlying meta-distribution, andpropose a new optimization problem— Quantile Risk Minimization (QRM)—whichrequires that predictors generalize with high probability .
Revised: To realize this, we explicitly relatetraining and test domains as draws from the same underlying meta-distribution, andpropose a new optimization problem— Quantile Risk Minimization (QRM)—whichrequires that predictors generalize with high probability .

Original: We then prove that QRM:(i) produces predictors that generalize to new domains with a desired probability,given sufficiently many domains and samples; and (ii) recovers the causal predictoras the desired probability of generalization approaches one.
Revised: We then prove that QRM:(i) produces predictors that generalize to new domains with a desired probability,given sufficiently many domains and samples; and (ii) recovers the causal predictoras the desired probability of generalization approaches one.

Original: In our experiments, weintroduce a more holistic quantile-focused evaluation protocol for DG, and showthat our algorithms outperform state-of-the-art baselines on real and synthetic data.
Revised: In our experiments, weintroduce a more holistic quantile-focused evaluation protocol for DG, and showthat our algorithms outperform state-of-the-art baselines on real and synthetic data.

Original:  Introduction
Revised:  Introduction

Original: Despite remarkable successes in recent years [1–3], machine learning systems often fail calamitouslywhen presented with out-of-distribution (OOD) data [4–7].
Revised: Despite remarkable successes in recent years [1–3], machine learning systems often fail calamitouslywhen presented with out-of-distribution (OOD) data [4–7].

Original: In fact, evidence of state-of-the-art systemsfailing in the face of distribution shift is mounting rapidly—be it due to spurious correlations [8–10],changing sub-populations [11–13], changes in location or time [14–16] or other naturally-occurringvariations [17–23].
Revised: In fact, evidence of state-of-the-art systemsfailing in the face of distribution shift is mounting rapidly—be it due to spurious correlations [8–10],changing sub-populations [11–13], changes in location or time [14–16] or other naturally-occurringvariations [17–23].

Original: These OOD failures are particularly concerning in safety-critical applicationssuch as medical imaging [24–28] and autonomous driving [29–31], where they represent one of themost significant barriers to real-world deployment [32–35].
Revised: These OOD failures are particularly concerning in safety-critical applicationssuch as medical imaging [24–28] and autonomous driving [29–31], where they represent one of themost significant barriers to real-world deployment [32–35].

Original: Domain generalization (DG) seeks to improve a system’s
Revised: Domain generalization (DG) seeks to improve a system’s

Original: OOD performance by leveraging datasetsfrom multiple environments or domains at training time, each collected under different experimentalconditions [36–38].
Revised: OOD performance by leveraging datasetsfrom multiple environments or domains at training time, each collected under different experimentalconditions [36–38].

Original: The goal is to build a predictor which exploits invariances across the trainingdomains in the hope that these invariances also hold in related but distinct test domains [38–41].
Revised: The goal is to build a predictor which exploits invariances across the trainingdomains in the hope that these invariances also hold in related but distinct test domains [38–41].

Original: Torealize this goal, DG is commonly formulated as a worst-case optimization problem over the set ofall possible domains.
Revised: Torealize this goal, DG is commonly formulated as a worst-case optimization problem over the set ofall possible domains.

Original: However, this problem is generally intractable [42–44] and, with worst-caseshifts unlikely in practice, leads to overly-conservative solutions—making large accuracy sacrifices inpursuit of adversarial robustness.
Revised: However, this problem is generally intractable [42–44] and, with worst-caseshifts unlikely in practice, leads to overly-conservative solutions—making large accuracy sacrifices inpursuit of adversarial robustness.

Original: In fact, a recent study [38] found that, across several popular datasets,no DG algorithm outperformed empirical risk minimization (ERM) in terms of average performance .
Revised: In fact, a recent study [38] found that, across several popular datasets,no DG algorithm outperformed empirical risk minimization (ERM) in terms of average performance .

Original: In this work, we argue that DG is neither a worst-case problem nor an average-case problem, butrather a probabilistic one.
Revised: In this work, we argue that DG is neither a worst-case problem nor an average-case problem, butrather a probabilistic one.

Original: In particular, we advocate for predictors which perform well with highprobability rather than in the worst case or on average.
Revised: In particular, we advocate for predictors which perform well with highprobability rather than in the worst case or on average.

Original: To this end, we propose a probabilistic
Revised: To this end, we propose a probabilistic

Original: [Figure]
Revised: [Figure]

Original: framework for DG which we call Probable Domain Generalization .
Revised: framework for DG which we call Probable Domain Generalization .

Original: The key idea is that distributionshifts seen during training should inform us of probable shifts at test time since they represent theways in which the underlying system tends to change (Fig. 1a).
Revised: The key idea is that distributionshifts seen during training should inform us of probable shifts at test time since they represent theways in which the underlying system tends to change (Fig. 1a).

Original: To realize this, we explicitly relatetraining and test domains as draws from the same underlying meta-distribution (Fig. 1b), and thenpropose Quantile Risk Minimization (QRM)—a new optimization problem for learning predictorsthat generalize with high probability (§ 3).
Revised: To realize this, we explicitly relatetraining and test domains as draws from the same underlying meta-distribution (Fig. 1b), and thenpropose Quantile Risk Minimization (QRM)—a new optimization problem for learning predictorsthat generalize with high probability (§ 3).

Original: The goal of QRM is to minimize the α -quantile of apredictor’s risk distribution over training domains, leveraging the key insight that this α -quantileis an upper bound on the test risk which holds with probability α (Fig. 1c).
Revised: The goal of QRM is to minimize the α -quantile of apredictor’s risk distribution over training domains, leveraging the key insight that this α -quantileis an upper bound on the test risk which holds with probability α (Fig. 1c).

Original: We then prove that QRM: (i) produces predictors that generalize to new domains with probabilityα , given sufficiently many domains and samples (§ 4); and (ii) recovers the causal predictor asα → 1 , under weaker assumptions than Peters et al.
Revised: We then prove that QRM: (i) produces predictors that generalize to new domains with probabilityα , given sufficiently many domains and samples (§ 4); and (ii) recovers the causal predictor asα → 1 , under weaker assumptions than Peters et al.

Original: Thus, α is an interpretable conservativenesshyperparameter, with α = 1 corresponding to the worst-case setting.
Revised: Thus, α is an interpretable conservativenesshyperparameter, with α = 1 corresponding to the worst-case setting.

Original: In our experiments (§ 6), weintroduce a more holistic quantile-focused evaluation protocol for DG, and demonstrate that: (i) theperformance of DG algorithms was likely never “lost” [38], but rather invisible through the lens of average performance; and (ii) our algorithms outperform strong baselines on real and synthetic datasets.
Revised: In our experiments (§ 6), weintroduce a more holistic quantile-focused evaluation protocol for DG, and demonstrate that: (i) theperformance of DG algorithms was likely never “lost” [38], but rather invisible through the lens of average performance; and (ii) our algorithms outperform strong baselines on real and synthetic datasets.

Original: Contributions.
Revised: Contributions.

Original: To summarize our main contributions, we:
Revised: To summarize our main contributions, we:

Original: • Propose Quantile Risk Minimization for learning predictors that generalize with probability α (§ 3).• Prove that, given sufficiently many domains and samples, QRM does indeed produce predictorsthat generalize to new domains with probability α (Thm.
Revised: • Propose Quantile Risk Minimization for learning predictors that generalize with probability α (§ 3).• Prove that, given sufficiently many domains and samples, QRM does indeed produce predictorsthat generalize to new domains with probability α (Thm.

Original: 4.1).• Prove that QRM recovers the causal predictor as α → 1 (Prop. 4.3 and Thm.
Revised: 4.1).• Prove that QRM recovers the causal predictor as α → 1 (Prop. 4.3 and Thm.

Original: 4.4).• Demonstrate empirically that: (i) effective comparison of DG algorithms requires evaluating quantile performance; and (ii) our algorithms outperform strong baselines on real and synthetic data (§ 6).
Revised: 4.4).• Demonstrate empirically that: (i) effective comparison of DG algorithms requires evaluating quantile performance; and (ii) our algorithms outperform strong baselines on real and synthetic data (§ 6).

Original:  Background
Revised:  Background

Original: Domain generalization (DG).
Revised: Domain generalization (DG).

Original: In DG, predictors are trained on data drawn from multiple relatedtraining distributions or domains and then evaluated on related but unseen test domains.
Revised: In DG, predictors are trained on data drawn from multiple relatedtraining distributions or domains and then evaluated on related but unseen test domains.

Original: For example,in the Camelyon17 dataset [46], the task is to predict if a given image of cells contains tumortissue, and domains correspond to different hospitals in which these images were captured.
Revised: For example,in the Camelyon17 dataset [46], the task is to predict if a given image of cells contains tumortissue, and domains correspond to different hospitals in which these images were captured.

Original: Moreformally, we consider datasets D e
Revised: Moreformally, we consider datasets D e

Original: = { ( x ei , y ei ) } n e i = 1 collected from m different training domains orenvironments E tr : = { e 1 , . . .
Revised: = { ( x ei , y ei ) } n e i = 1 collected from m different training domains orenvironments E tr : = { e 1 , . . .

Original: , e m } , with each dataset D e containing data pairs ( x ei , y ei ) sampled i.i.d.from P ( X e , Y e ) .
Revised: , e m } , with each dataset D e containing data pairs ( x ei , y ei ) sampled i.i.d.from P ( X e , Y e ) .

Original: Then, given a suitable function class F and loss function ℓ , the goal of DG is to learna predictor f ∈ F that generalizes to data drawn from a larger set of all possible domains E all ⊃ E tr .
Revised: Then, given a suitable function class F and loss function ℓ , the goal of DG is to learna predictor f ∈ F that generalizes to data drawn from a larger set of all possible domains E all ⊃ E tr .

Original: By letting R e ( f ) denote the statistical risk of f in domain e , this is commonly [9, 22, 41, 47–49]formalized as the following minimax optimization problem:
Revised: By letting R e ( f ) denote the statistical risk of f in domain e , this is commonly [9, 22, 41, 47–49]formalized as the following minimax optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: As we only have access to data from a finite subset of E all during training, solving (DG) is not justchallenging but in fact impossible [41, 50, 51] without restrictions on how the domains may differ
Revised: As we only have access to data from a finite subset of E all during training, solving (DG) is not justchallenging but in fact impossible [41, 50, 51] without restrictions on how the domains may differ

Original: Causality and invariance in DG.
Revised: Causality and invariance in DG.

Original: Causal works on DG [9, 41, 45, 51, 52] describe domaindifferences using the language of causality and the notion of interventions [53, 54].
Revised: Causal works on DG [9, 41, 45, 51, 52] describe domaindifferences using the language of causality and the notion of interventions [53, 54].

Original: In particular, they assume all domains share the same underlying structural causal model (SCM) [53], with differentdomains corresponding to different interventions (see Appendix A.1 for formal definitions and asimple example).
Revised: In particular, they assume all domains share the same underlying structural causal model (SCM) [53], with differentdomains corresponding to different interventions (see Appendix A.1 for formal definitions and asimple example).

Original: Early work studied the problem of learning from multiple cause-effect datasets thatshare a functional mechanism but differ in noise distributions [39].
Revised: Early work studied the problem of learning from multiple cause-effect datasets thatshare a functional mechanism but differ in noise distributions [39].

Original: More generally, given (data from)multiple distributions, one can try to identify components which are stable, robust, or invariant , andfind means to transfer them across problems [55–59].
Revised: More generally, given (data from)multiple distributions, one can try to identify components which are stable, robust, or invariant , andfind means to transfer them across problems [55–59].

Original: Assuming that the mechanism of Y remainsfixed or invariant 1 but all X s may be intervened upon, recent works have shown that only the causalpredictor: (i) has invariant predictive distributions [45], coefficients [9], or risks [41] across domains;and (ii) generalizes to arbitrary interventions on the X s [9, 45, 52].
Revised: Assuming that the mechanism of Y remainsfixed or invariant 1 but all X s may be intervened upon, recent works have shown that only the causalpredictor: (i) has invariant predictive distributions [45], coefficients [9], or risks [41] across domains;and (ii) generalizes to arbitrary interventions on the X s [9, 45, 52].

Original: These works then exploit suchinsights by leveraging some form of invariance across domains to discover causal relationshipswhich, through the invariant mechanism assumption, generalize to new domains.
Revised: These works then exploit suchinsights by leveraging some form of invariance across domains to discover causal relationshipswhich, through the invariant mechanism assumption, generalize to new domains.

Original:  Quantile Risk Minimization
Revised:  Quantile Risk Minimization

Original: In this section we introduce Quantile Risk Minimization (QRM) for achieving probable domaingeneralization.
Revised: In this section we introduce Quantile Risk Minimization (QRM) for achieving probable domaingeneralization.

Original: The core idea is to replace the worst-case perspective of (DG) with a probabilistic one.
Revised: The core idea is to replace the worst-case perspective of (DG) with a probabilistic one.

Original: This approach is founded on a great deal of work in classical fields such as control theory [60, 61] andsmoothed analysis [62], wherein approaches that yield high-probability guarantees are used in placeof worst-case approaches in an effort to mitigate conservatism and computational limitations.
Revised: This approach is founded on a great deal of work in classical fields such as control theory [60, 61] andsmoothed analysis [62], wherein approaches that yield high-probability guarantees are used in placeof worst-case approaches in an effort to mitigate conservatism and computational limitations.

Original: Thismitigation is of particular interest in domain generalization since generalizing to arbitrary domains isimpossible [41, 50, 51].
Revised: Thismitigation is of particular interest in domain generalization since generalizing to arbitrary domains isimpossible [41, 50, 51].

Original: Thus, motivated by this classical literature, our goal is to obtain predictorsthat are robust with high probability over domains drawn from E all , rather than in the worst-case.
Revised: Thus, motivated by this classical literature, our goal is to obtain predictorsthat are robust with high probability over domains drawn from E all , rather than in the worst-case.

Original: A distribution over environments.
Revised: A distribution over environments.

Original: We start by assuming the existence of a probability distribution 98 Q ( e ) over the set of all environments E all .
Revised: We start by assuming the existence of a probability distribution 98 Q ( e ) over the set of all environments E all .

Original: For instance, in the context of medical imaging, Qcould represent a distribution over potential changes to a hospital’s setup (see Fig. 1a) or simply adistribution over candidate hospitals.
Revised: For instance, in the context of medical imaging, Qcould represent a distribution over potential changes to a hospital’s setup (see Fig. 1a) or simply adistribution over candidate hospitals.

Original: Given that such a distribution Q exists 2 , we can think of the risk
Revised: Given that such a distribution Q exists 2 , we can think of the risk

Original: R e ( f ) as a random variable for each f ∈ F , where the randomness is engendered by the draw ofe ∼ Q .
Revised: R e ( f ) as a random variable for each f ∈ F , where the randomness is engendered by the draw ofe ∼ Q .

Original: This perspective gives rise to the following analogue of the optimization problem in (DG):
Revised: This perspective gives rise to the following analogue of the optimization problem in (DG):

Original: [Equation]
Revised: [Equation]

Original: where ess sup
Revised: where ess sup

Original: [Equation]
Revised: [Equation]

Original: Here, ess sup denotes the essential-supremum operator from measure theory, meaning that for eachf ∈ F , ess sup Q R e ( f ) is the least upper bound on R e ( f ) that holds for almost every e ∼ Q .
Revised: Here, ess sup denotes the essential-supremum operator from measure theory, meaning that for eachf ∈ F , ess sup Q R e ( f ) is the least upper bound on R e ( f ) that holds for almost every e ∼ Q .

Original: Inthis way, the ess sup in (3.1) is the measure-theoretic analogue of the max operator in (DG), withthe subtle but critical difference being that the ess sup in (3.1) can neglect domains of measure zerounder Q .
Revised: Inthis way, the ess sup in (3.1) is the measure-theoretic analogue of the max operator in (DG), withthe subtle but critical difference being that the ess sup in (3.1) can neglect domains of measure zerounder Q .

Original: For example, for discrete Q , (3.1) ignores domains which are impossible (zero probability)while (DG) does not, laying the foundation for ignoring domains which are improbable .
Revised: For example, for discrete Q , (3.1) ignores domains which are impossible (zero probability)while (DG) does not, laying the foundation for ignoring domains which are improbable .

Original: High-probability generalization.
Revised: High-probability generalization.

Original: Although the minimax problem in (3.1) explicitly incorporates thedistribution Q over environments, this formulation is no less conservative than (DG).
Revised: Although the minimax problem in (3.1) explicitly incorporates thedistribution Q over environments, this formulation is no less conservative than (DG).

Original: Indeed, in manycases, (3.1) is equivalent to (DG); see Appendix B for details.
Revised: Indeed, in manycases, (3.1) is equivalent to (DG); see Appendix B for details.

Original: Therefore, rather than considering theworst-case problem in (3.1), we propose the following generalization of (3.1) which requires thatpredictors generalize with probability α rather than in the worst-case:
Revised: Therefore, rather than considering theworst-case problem in (3.1), we propose the following generalization of (3.1) which requires thatpredictors generalize with probability α rather than in the worst-case:

Original: [Equation]
Revised: [Equation]

Original: The optimization problem of (3.2) formally defines what we mean by probable domain generalization.
Revised: The optimization problem of (3.2) formally defines what we mean by probable domain generalization.

Original: In particular, we say that a predictor f generalizes with risk t at level α if f has risk at most t withprobability at least α over domains sampled from Q .
Revised: In particular, we say that a predictor f generalizes with risk t at level α if f has risk at most t withprobability at least α over domains sampled from Q .

Original: In this way, the conservativeness parameter αcontrols the strictness of generalizing to unseen domains.
Revised: In this way, the conservativeness parameter αcontrols the strictness of generalizing to unseen domains.

Original: A distribution over risks.
Revised: A distribution over risks.

Original: The optimization problem presented in (3.2) offers a principled formulationfor generalizing to unseen distributional shifts governed by Q .
Revised: The optimization problem presented in (3.2) offers a principled formulationfor generalizing to unseen distributional shifts governed by Q .

Original: However, Q is often unknown inpractice and its support E all may be high-dimensional or challenging to define [22].
Revised: However, Q is often unknown inpractice and its support E all may be high-dimensional or challenging to define [22].

Original: While manyprevious works have made progress by limiting the scope of possible shift types over domains [19, 22,48], in practice, such structural assumptions are often difficult to justify and impossible to test.
Revised: While manyprevious works have made progress by limiting the scope of possible shift types over domains [19, 22,48], in practice, such structural assumptions are often difficult to justify and impossible to test.

Original: Forthis reason, we start our exposition of QRM by offering an alternative view of (3.2) which elucidateshow a predictor’s risk distribution plays a central role in achieving probable domain generalization.
Revised: Forthis reason, we start our exposition of QRM by offering an alternative view of (3.2) which elucidateshow a predictor’s risk distribution plays a central role in achieving probable domain generalization.

Original: As Q is often unknown,
Revised: As Q is often unknown,

Original: To begin, note that for each f ∈ F , the distribution over domains Q naturally induces 3 adistribution T f over the risks in each domain R e ( f ) .
Revised: To begin, note that for each f ∈ F , the distribution over domains Q naturally induces 3 adistribution T f over the risks in each domain R e ( f ) .

Original: In this way, rather than considering therandomness of Q in the often-unknown and (potentially) high-dimensional space of possible shiftsor interventions (Figs.
Revised: In this way, rather than considering therandomness of Q in the often-unknown and (potentially) high-dimensional space of possible shiftsor interventions (Figs.

Original: 1a and 1b), one can consider it in the real-valued space of risks (Fig. 1c).
Revised: 1a and 1b), one can consider it in the real-valued space of risks (Fig. 1c).

Original: This is analogous to statistical learning theory, where the analysis of convergence of empirical riskminimizers (i.e., of functions) is substituted by that of a weaker form of convergence, namely that ofscalar risk functionals—a crucial step for VC theory [63].
Revised: This is analogous to statistical learning theory, where the analysis of convergence of empirical riskminimizers (i.e., of functions) is substituted by that of a weaker form of convergence, namely that ofscalar risk functionals—a crucial step for VC theory [63].

Original: From this perspective, the statistics of T fcan be thought of as capturing the sensitivity of f to different environmental shifts, summarizing theeffect of different intervention types, strengths, and frequencies.
Revised: From this perspective, the statistics of T fcan be thought of as capturing the sensitivity of f to different environmental shifts, summarizing theeffect of different intervention types, strengths, and frequencies.

Original: To this end, (3.2) can be equivalentlyrewritten in terms of the risk distribution T f as follows:
Revised: To this end, (3.2) can be equivalentlyrewritten in terms of the risk distribution T f as follows:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Here, F − 1 T f ( α ) denotes the inverse CDF (or quantile) function of the risk distribution T f .
Revised: Here, F − 1 T f ( α ) denotes the inverse CDF (or quantile) function of the risk distribution T f .

Original: By meansof this reformulation, we elucidate how solving (QRM) amounts to finding a predictor with minimalα -quantile risk.
Revised: By meansof this reformulation, we elucidate how solving (QRM) amounts to finding a predictor with minimalα -quantile risk.

Original: That is, (QRM) requires that a predictor f satisfy the probabilistic constraint forat least an α -fraction of the risks R ∼ T f , or, equivalently, for an α -fraction of the environmentse ∼ Q .
Revised: That is, (QRM) requires that a predictor f satisfy the probabilistic constraint forat least an α -fraction of the risks R ∼ T f , or, equivalently, for an α -fraction of the environmentse ∼ Q .

Original: In this way, by varying α , (QRM) recovers several notable objectives for DG as special cases,namely a worst-case problem with α = 1 and an average-case problem with α = 0.5 (median).
Revised: In this way, by varying α , (QRM) recovers several notable objectives for DG as special cases,namely a worst-case problem with α = 1 and an average-case problem with α = 0.5 (median).

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: For α = 1 , (QRM) is equivalent to the minimax problem of (3.1) .
Revised: For α = 1 , (QRM) is equivalent to the minimax problem of (3.1) .

Original: Furthermore,if the mean and median of T f coincide for all f ∈ F , then for α = 1 / 2 , (QRM) is equivalent to(expected) risk minimization (RM, [63]) in which T f is the uniform distribution
Revised: Furthermore,if the mean and median of T f coincide for all f ∈ F , then for α = 1 / 2 , (QRM) is equivalent to(expected) risk minimization (RM, [63]) in which T f is the uniform distribution

Original: [Equation]
Revised: [Equation]

Original:  Algorithms for Quantile Risk Minimization
Revised:  Algorithms for Quantile Risk Minimization

Original: In this section we introduce two algorithms for empirically solving (QRM): empirical QRM (EQRM)and empirical superquantile RM (ESQRM).
Revised: In this section we introduce two algorithms for empirically solving (QRM): empirical QRM (EQRM)and empirical superquantile RM (ESQRM).

Original: 4.1 Empirical QRM
Revised: 4.1 Empirical QRM

Original: Estimated risk distribution.
Revised: Estimated risk distribution.

Original: In practice, given a predictor f and its empirical risksˆ R e 1 ( f ) , . . .
Revised: In practice, given a predictor f and its empirical risksˆ R e 1 ( f ) , . . .

Original: , ˆ R e m ( f ) on the m training domains, we must form an estimated risk distribution (cid:98) T f .
Revised: , ˆ R e m ( f ) on the m training domains, we must form an estimated risk distribution (cid:98) T f .

Original: Ifwe have prior knowledge about the form of T f (e.g. Gaussian), we can simply estimate the parametersof this distribution (e.g. via the sample mean and variance).
Revised: Ifwe have prior knowledge about the form of T f (e.g. Gaussian), we can simply estimate the parametersof this distribution (e.g. via the sample mean and variance).

Original: In the absence of such knowledge, wecan use nonparametric methods like kernel density estimation (KDE, [64, 65]).
Revised: In the absence of such knowledge, wecan use nonparametric methods like kernel density estimation (KDE, [64, 65]).

Original: Fig.
Revised: Fig.

Original: 1c depicts the
Revised: 1c depicts the

Original: PDF and CDF for 10 training risks when using KDE with Gaussian kernels, while Fig.
Revised: PDF and CDF for 10 training risks when using KDE with Gaussian kernels, while Fig.

Original: 2 comparesthe smoothed KDE CDF (black) to the unsmoothed empirical CDF (gray).
Revised: 2 comparesthe smoothed KDE CDF (black) to the unsmoothed empirical CDF (gray).

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: P r ob a b ilit y α
Revised: P r ob a b ilit y α

Original: F − 1 ( α )
Revised: F − 1 ( α )

Original: KDEEmpirical
Revised: KDEEmpirical

Original: Importance of smoothing.
Revised: Importance of smoothing.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, the KDE smoothing permits risk “extrapolation” [41] beyond our largest training risk, with the estimated α -quantile risk going to infinity as α → 1 (when kernels have fullsupport).
Revised: 2, the KDE smoothing permits risk “extrapolation” [41] beyond our largest training risk, with the estimated α -quantile risk going to infinity as α → 1 (when kernels have fullsupport).

Original: Critically, as detailed in Prop. 4.3 below, this unlocks invariantprediction for QRM.
Revised: Critically, as detailed in Prop. 4.3 below, this unlocks invariantprediction for QRM.

Original: In Appendix C, we discuss the suitability of differentdata-dependent bandwidth-selection methods for our purposes, with different methods encoding different assumptions about right-tail heavinessand thus about projected OOD risk.
Revised: In Appendix C, we discuss the suitability of differentdata-dependent bandwidth-selection methods for our purposes, with different methods encoding different assumptions about right-tail heavinessand thus about projected OOD risk.

Original: In practice, we use Gaussian kernelswith the Gaussian-optimal rule [66] or Silverman’s rule-of-thumb [66].
Revised: In practice, we use Gaussian kernelswith the Gaussian-optimal rule [66] or Silverman’s rule-of-thumb [66].

Original: Empirical problem.
Revised: Empirical problem.

Original: Armed with a predictor’s estimated risk distribution (cid:98) T f , we can approximatelysolve (QRM) using the following empirical analogue:
Revised: Armed with a predictor’s estimated risk distribution (cid:98) T f , we can approximatelysolve (QRM) using the following empirical analogue:

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that (4.1) depends only on known quantities, so we can compute and minimize it in practice.
Revised: Note that (4.1) depends only on known quantities, so we can compute and minimize it in practice.

Original: Alsonote that in finance, when concerned with a distribution over potential losses, the α -quantile value isknown as the value at risk (VaR) at level α [67].
Revised: Alsonote that in finance, when concerned with a distribution over potential losses, the α -quantile value isknown as the value at risk (VaR) at level α [67].

Original: Alg.
Revised: Alg.

Original: 1 of Appendix E.1 details the EQRM algorithm.
Revised: 1 of Appendix E.1 details the EQRM algorithm.

Original: Generalization bound.
Revised: Generalization bound.

Original: We now give a simplified version of our main generalization bound—
Revised: We now give a simplified version of our main generalization bound—

Original: Thm.
Revised: Thm.

Original: D.1.
Revised: D.1.

Original: The full version, stated and proved in Appendix D, provides specific finite-sample boundson ϵ 1 and ϵ 2 below, depending on the hypothesis class F , the empirical estimator (cid:91) QRM α , and theassumptions made on the possible risk profiles of hypotheses f ∈ F .
Revised: The full version, stated and proved in Appendix D, provides specific finite-sample boundson ϵ 1 and ϵ 2 below, depending on the hypothesis class F , the empirical estimator (cid:91) QRM α , and theassumptions made on the possible risk profiles of hypotheses f ∈ F .

Original: Theorem 4.1 (Simplified form of Thm.
Revised: Theorem 4.1 (Simplified form of Thm.

Original: D.1, uniform convergence) .
Revised: D.1, uniform convergence) .

Original: Given m domains and n samplesin each, then with high probability over the training data,sup
Revised: Given m domains and n samplesin each, then with high probability over the training data,sup

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ϵ 1 → 0 as n → ∞ and ϵ 2 → 0 as m → ∞ .
Revised: where ϵ 1 → 0 as n → ∞ and ϵ 2 → 0 as m → ∞ .

Original: This states that, for m , n sufficiently large, the empirical α -quantile risk is a good estimate of thepopulation α -quantile risk.
Revised: This states that, for m , n sufficiently large, the empirical α -quantile risk is a good estimate of thepopulation α -quantile risk.

Original: Thus, a predictor f which minimizes the empirical problem in (4.1) shouldapproximately minimize the population problem in (QRM).
Revised: Thus, a predictor f which minimizes the empirical problem in (4.1) shouldapproximately minimize the population problem in (QRM).

Original: While many domains are required forthis generalization bound to be tight, our empirical results in § 6 demonstrate that (E)QRM performswell in practice given only a few domains.
Revised: While many domains are required forthis generalization bound to be tight, our empirical results in § 6 demonstrate that (E)QRM performswell in practice given only a few domains.

Original: In other words, while many domains are required for α toprecisely estimate the true quantile, α can still be used as an interpretable conservativeness-hyperparameter given only a few domains, albeit with a less precise interpretation.
Revised: In other words, while many domains are required for α toprecisely estimate the true quantile, α can still be used as an interpretable conservativeness-hyperparameter given only a few domains, albeit with a less precise interpretation.

Original: Recovering the causal predictor.
Revised: Recovering the causal predictor.

Original: We now state our causal recovery results, namely that: (i) QRMlearns a minimal invariant-risk predictor as α → 1 ; and (ii) learning a minimal invariant-risk predictoris sufficient to recover the causal predictor under assumptions which are both weaker and more generalthan those of Peters et al.
Revised: We now state our causal recovery results, namely that: (i) QRMlearns a minimal invariant-risk predictor as α → 1 ; and (ii) learning a minimal invariant-risk predictoris sufficient to recover the causal predictor under assumptions which are both weaker and more generalthan those of Peters et al.

Original: [45, Thm 2] and Krueger et al.
Revised: [45, Thm 2] and Krueger et al.

Original: [41, Thm 1] (Appendix A.2.3).
Revised: [41, Thm 1] (Appendix A.2.3).

Original: For intuitionon (i), we refer the reader to Appendices A.2.1 and A.2.2 where we illustrate how setting α → 1 eventually forces the risk variance over domains to 0.
Revised: For intuitionon (i), we refer the reader to Appendices A.2.1 and A.2.2 where we illustrate how setting α → 1 eventually forces the risk variance over domains to 0.

Original: Together, these results provide the conditions underwhich QRM recovers the causal mechanism of Y as α → 1 , as well as new insights into the fundamental connection between invariance (in particular, invariant risks , see Appendix G.1.2) and causality.
Revised: Together, these results provide the conditions underwhich QRM recovers the causal mechanism of Y as α → 1 , as well as new insights into the fundamental connection between invariance (in particular, invariant risks , see Appendix G.1.2) and causality.

Original: Definition 4.2.
Revised: Definition 4.2.

Original: A predictor is said to be an invariant-risk predictor if its risk is equal almost surelyacross domains (i.e., Var e ∼ Q
Revised: A predictor is said to be an invariant-risk predictor if its risk is equal almost surelyacross domains (i.e., Var e ∼ Q

Original: [ R e ( f )]
Revised: [ R e ( f )]

Original: = 0 ).
Revised: = 0 ).

Original: A predictor is said to be a minimal invariant-riskpredictor if it achieves the minimal possible risk over all possible invariant-risk predictors.
Revised: A predictor is said to be a minimal invariant-riskpredictor if it achieves the minimal possible risk over all possible invariant-risk predictors.

Original: Proposition 4.3 (QRM learns a minimal invariant-risk predictor as α → 1 , informal version of
Revised: Proposition 4.3 (QRM learns a minimal invariant-risk predictor as α → 1 , informal version of

Original: Props.
Revised: Props.

Original: A.4 and A.5) .
Revised: A.4 and A.5) .

Original: Assume: (i) F contains an invariant-risk predictor with finite training risks;and (ii) no arbitrarily-negative training risks.
Revised: Assume: (i) F contains an invariant-risk predictor with finite training risks;and (ii) no arbitrarily-negative training risks.

Original: Then, as α → 1 , Gaussian and kernel QRM predictors(the latter with certain bandwidth-selection methods) converge to minimal invariant-risk predictors.
Revised: Then, as α → 1 , Gaussian and kernel QRM predictors(the latter with certain bandwidth-selection methods) converge to minimal invariant-risk predictors.

Original: Theorem 4.4 (The causal predictor is (often) the only minimal invariant-risk predictor) .
Revised: Theorem 4.4 (The causal predictor is (often) the only minimal invariant-risk predictor) .

Original: Assume that:(i)
Revised: Assume that:(i)

Original: Y is generated from a linear SEM, Y = β ⊺ X + N , with coefficients β ∈ R d and noise N havingconstant variance and E [ N ] = 0 across domains; (ii) F is the class of linear predictors, indexed byˆ β ∈ R d ; (iii) all components of X are observed; (iv) the loss ℓ is squared-error; and (v) the systemof equations
Revised: Y is generated from a linear SEM, Y = β ⊺ X + N , with coefficients β ∈ R d and noise N havingconstant variance and E [ N ] = 0 across domains; (ii) F is the class of linear predictors, indexed byˆ β ∈ R d ; (iii) all components of X are observed; (iv) the loss ℓ is squared-error; and (v) the systemof equations

Original: [Equation]
Revised: [Equation]

Original: has the unique solution x = 0 .
Revised: has the unique solution x = 0 .

Original: If ˆ β is a minimal invariant-risk predictor, then ˆ β = β .
Revised: If ˆ β is a minimal invariant-risk predictor, then ˆ β = β .

Original: In Appendix A.2, we prove the formal versions of Prop. 4.3 (Props.
Revised: In Appendix A.2, we prove the formal versions of Prop. 4.3 (Props.

Original: A.4 and A.5) along with Thm.
Revised: A.4 and A.5) along with Thm.

Original: 4.4,provide
Revised: 4.4,provide

Original: some intuition and concrete examples regarding Assumption (v) of Thm. 4.4, and relate our αparameter to the β parameter of VREx [41, Eq. 8] for the special case of Gaussian estimators of T f .
Revised: some intuition and concrete examples regarding Assumption (v) of Thm. 4.4, and relate our αparameter to the β parameter of VREx [41, Eq. 8] for the special case of Gaussian estimators of T f .

Original: 4.2 Empirical SQRM
Revised: 4.2 Empirical SQRM

Original: Although the formulation presented in (4.1) is intuitive and interpretable, the quantile function isnotoriously difficult to optimize [68].
Revised: Although the formulation presented in (4.1) is intuitive and interpretable, the quantile function isnotoriously difficult to optimize [68].

Original: Indeed, in many cases of interest, F − 1 can be nonconvex anddiscontinuous, which induces an unfavorable optimization landscape [69, 70].
Revised: Indeed, in many cases of interest, F − 1 can be nonconvex anddiscontinuous, which induces an unfavorable optimization landscape [69, 70].

Original: To circumvent this,we propose an alternative scheme in which we optimize the superquantile (SQ)—also known as theconditional value at risk (CVaR) or expected tail loss —at level α , which is defined 4 as follows:
Revised: To circumvent this,we propose an alternative scheme in which we optimize the superquantile (SQ)—also known as theconditional value at risk (CVaR) or expected tail loss —at level α , which is defined 4 as follows:

Original: [Equation]
Revised: [Equation]

Original: SQ α can be seen as the conditional expectation of a random variable R subject to R being larger thanthe α -quantile F − 1 ( α ) .
Revised: SQ α can be seen as the conditional expectation of a random variable R subject to R being larger thanthe α -quantile F − 1 ( α ) .

Original: In our case, where R represents the statistical risk on a randomly-sampledenvironment, SQ α can be seen as the expected risk in the worst 100 · ( 1 − α ) % of cases/domains.
Revised: In our case, where R represents the statistical risk on a randomly-sampledenvironment, SQ α can be seen as the expected risk in the worst 100 · ( 1 − α ) % of cases/domains.

Original: Importantly, unlike the quantile, the superquantile has many desirable mathematical properties: (i) itis continuous in α ; (ii) it is jointly convex in R and α ; and (iii) it is the tightest convex surrogate forthe inverse CDF [71, § 2].
Revised: Importantly, unlike the quantile, the superquantile has many desirable mathematical properties: (i) itis continuous in α ; (ii) it is jointly convex in R and α ; and (iii) it is the tightest convex surrogate forthe inverse CDF [71, § 2].

Original: Therefore, we also consider the following optimization problem:
Revised: Therefore, we also consider the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: We call the empirical analogue, which uses the empirical density rather than T f , empirical SQRM.
Revised: We call the empirical analogue, which uses the empirical density rather than T f , empirical SQRM.

Original: Alg. 2 of Appendix E.1 details the ESQRM algorithm.
Revised: Alg. 2 of Appendix E.1 details the ESQRM algorithm.

Original: Connections to existing problems.
Revised: Connections to existing problems.

Original: It is straightforward to show that (SQRM) recovers the RMproblem in (3.3) and the minimax problem in (3.2) as special cases with α = 0 and α = 1 respectively.
Revised: It is straightforward to show that (SQRM) recovers the RMproblem in (3.3) and the minimax problem in (3.2) as special cases with α = 0 and α = 1 respectively.

Original: Moreover, in Appendix F, we exploit the well-known duality properties of CVaR to showan alternative, view of (SQRM) as a distributionally-robust optimization problem, admitting closeconnections to previous work, including GroupDRO [48]; see Prop. F.1 and F.2 for details.
Revised: Moreover, in Appendix F, we exploit the well-known duality properties of CVaR to showan alternative, view of (SQRM) as a distributionally-robust optimization problem, admitting closeconnections to previous work, including GroupDRO [48]; see Prop. F.1 and F.2 for details.

Original:  Related work
Revised:  Related work

Original: Robust optimization in DG.
Revised: Robust optimization in DG.

Original: Throughout this paper, we follow an established line of work (seee.g.
Revised: Throughout this paper, we follow an established line of work (seee.g.

Original: , [9, 41, 49]) which formulates the DG problem through the lens of robust optimization [47].
Revised: , [9, 41, 49]) which formulates the DG problem through the lens of robust optimization [47].

Original: Tothis end, various algorithms have been proposed for solving constrained [22] and distributionallyrobust [48] variants of the worst-case problem in (DG).
Revised: Tothis end, various algorithms have been proposed for solving constrained [22] and distributionallyrobust [48] variants of the worst-case problem in (DG).

Original: Indeed, this robust formulation has a firmfoundation in the broader machine learning literature, with notable works in adversarial robustness [72–76 ] and fair learning [77, 78] employing similar formulations.
Revised: Indeed, this robust formulation has a firmfoundation in the broader machine learning literature, with notable works in adversarial robustness [72–76 ] and fair learning [77, 78] employing similar formulations.

Original: Unlike these past works, we considera robust but non-adversarial formulation for DG, where predictors are trained to generalize with highprobability rather than in the worst case.
Revised: Unlike these past works, we considera robust but non-adversarial formulation for DG, where predictors are trained to generalize with highprobability rather than in the worst case.

Original: Moreover, the majority of this literature—both within andoutside of DG—relies on specific structural assumptions (e.g. covariate shift) on the types of possibleinterventions or perturbations.
Revised: Moreover, the majority of this literature—both within andoutside of DG—relies on specific structural assumptions (e.g. covariate shift) on the types of possibleinterventions or perturbations.

Original: In contrast, we make the weaker and more flexible assumption ofiid-sampled domains, which ultimately makes use of the data/domains to determine the types of shiftsthat are possible or probable .
Revised: In contrast, we make the weaker and more flexible assumption ofiid-sampled domains, which ultimately makes use of the data/domains to determine the types of shiftsthat are possible or probable .

Original: We further discuss this important difference in § 7.
Revised: We further discuss this important difference in § 7.

Original: Other approaches to DG.
Revised: Other approaches to DG.

Original: Outside of robust optimization, many algorithms have been proposed forthe DG setting which draw on insights from a diverse array of fields, including approaches based ontools from meta-learning [40, 79–82], kernel methods [83, 84], and information theory [49].
Revised: Outside of robust optimization, many algorithms have been proposed forthe DG setting which draw on insights from a diverse array of fields, including approaches based ontools from meta-learning [40, 79–82], kernel methods [83, 84], and information theory [49].

Original: Alsoprominent are works that design regularizers to generalize OOD [85–87] and works that seek domaininvariant representations [88–90].
Revised: Alsoprominent are works that design regularizers to generalize OOD [85–87] and works that seek domaininvariant representations [88–90].

Original: However, the majority of these works do not explicitly relate thedomains seen during training and test time, which is a key feature of our approach.
Revised: However, the majority of these works do not explicitly relate thedomains seen during training and test time, which is a key feature of our approach.

Original: Moreover, many ofthese works employ hyperparameters which are difficult to interpret, which has no doubt contributedto the well-established model-selection problem in DG [38].
Revised: Moreover, many ofthese works employ hyperparameters which are difficult to interpret, which has no doubt contributedto the well-established model-selection problem in DG [38].

Original: In contrast, in our framework, α can beeasily interpreted in terms of the quantiles of the risk distribution.
Revised: In contrast, in our framework, α can beeasily interpreted in terms of the quantiles of the risk distribution.

Original: Finally, while many of the bestperforming DG methods provide no theoretical results in the nonlinear setting (e.g.[9, 41, 82, 91]),
Revised: Finally, while many of the bestperforming DG methods provide no theoretical results in the nonlinear setting (e.g.[9, 41, 82, 91]),

Original: QRM has both strong theoretical and empirical results in the nonlinear setting (see § 6).
Revised: QRM has both strong theoretical and empirical results in the nonlinear setting (see § 6).

Original: High-probability generalization.
Revised: High-probability generalization.

Original: As noted in § 3, relaxing worst-case problems in favor of probabilistic ones has a long history in control theory [42, 60, 61, 92, 93], operations research [94],and smoothed analysis [62].
Revised: As noted in § 3, relaxing worst-case problems in favor of probabilistic ones has a long history in control theory [42, 60, 61, 92, 93], operations research [94],and smoothed analysis [62].

Original: Recently, this paradigm has been applied to several areas of machinelearning, including perturbation-based robustness [95, 96], fairness [97], active learning [98], andreinforcement learning [99, 100].
Revised: Recently, this paradigm has been applied to several areas of machinelearning, including perturbation-based robustness [95, 96], fairness [97], active learning [98], andreinforcement learning [99, 100].

Original: However, it has not yet been applied to domain generalization.
Revised: However, it has not yet been applied to domain generalization.

Original: [Figure]
Revised: [Figure]

Original: Quantile minimization.
Revised: Quantile minimization.

Original: In financial optimization, VaR and CVaR [67, 69, 101] are central to theliterature surrounding portfolio risk management, with numerous applications spanning bankingregulations and insurance policies [68, 102].
Revised: In financial optimization, VaR and CVaR [67, 69, 101] are central to theliterature surrounding portfolio risk management, with numerous applications spanning bankingregulations and insurance policies [68, 102].

Original: In statistical learning theory, several recent papers havederived uniform convergence guarantees in terms of alternative risk functionals besides expectedrisk [98, 103–105].
Revised: In statistical learning theory, several recent papers havederived uniform convergence guarantees in terms of alternative risk functionals besides expectedrisk [98, 103–105].

Original: These results focus on CVaR and closely related functionals that can be writtenin terms of expectations over the loss distribution.
Revised: These results focus on CVaR and closely related functionals that can be writtenin terms of expectations over the loss distribution.

Original: In contrast, our uniform convergence guarantee(Theorem D.1) shows uniform convergence of VaR , which cannot be written as such an expectation; this necessitates stronger conditions to obtain uniform convergence, which ultimately suggestregularizing the estimated risk distribution (e.g. by kernel smoothing).
Revised: In contrast, our uniform convergence guarantee(Theorem D.1) shows uniform convergence of VaR , which cannot be written as such an expectation; this necessitates stronger conditions to obtain uniform convergence, which ultimately suggestregularizing the estimated risk distribution (e.g. by kernel smoothing).

Original: Invariant prediction and causality.
Revised: Invariant prediction and causality.

Original: As discussed in § 2, recent works have leveraged different formsof invariance across domains to discover causal relationships which, under the invariant mechanismassumption [54], generalize to new domains [9, 41, 45, 52, 106–108].
Revised: As discussed in § 2, recent works have leveraged different formsof invariance across domains to discover causal relationships which, under the invariant mechanismassumption [54], generalize to new domains [9, 41, 45, 52, 106–108].

Original: In particular, like QRM (asα → 1 ), Krueger et al.
Revised: In particular, like QRM (asα → 1 ), Krueger et al.

Original: [41] leverage invariant risks , while Arjovsky et al.
Revised: [41] leverage invariant risks , while Arjovsky et al.

Original: [9] leverage invariantfunctions or regression coefficients.
Revised: [9] leverage invariantfunctions or regression coefficients.

Original: We further discuss this important distinction in Appendix G.1.2.
Revised: We further discuss this important distinction in Appendix G.1.2.

Original: More recently, noting that causal parameters are often too conservative and thus that performingwell out-of-distribution can require non-causal relationships, Rothenhäusler et al.
Revised: More recently, noting that causal parameters are often too conservative and thus that performingwell out-of-distribution can require non-causal relationships, Rothenhäusler et al.

Original: [109] proposed
Revised: [109] proposed

Original: Anchor regression for interpolating between causal and predictive parameters.
Revised: Anchor regression for interpolating between causal and predictive parameters.

Original: However, the assumedshift types are even more restrictive than in the works above, despite nonlinear extensions [51].
Revised: However, the assumedshift types are even more restrictive than in the works above, despite nonlinear extensions [51].

Original:  Experiments
Revised:  Experiments

Original: We now evaluate our algorithms on synthetic and real-world datasets.
Revised: We now evaluate our algorithms on synthetic and real-world datasets.

Original: These evaluations illustratethe importance of multiple test domains in DG benchmarks, as well as the benefits of our algorithmsin practice.
Revised: These evaluations illustratethe importance of multiple test domains in DG benchmarks, as well as the benefits of our algorithmsin practice.

Original: Appendices E and G report further results and experimental details.
Revised: Appendices E and G report further results and experimental details.

Original: 6.1 Synthetic datasets
Revised: 6.1 Synthetic datasets

Original: Linear regression.
Revised: Linear regression.

Original: We first consider a linear regression dataset based on the following linear SCM:
Revised: We first consider a linear regression dataset based on the following linear SCM:

Original: [Equation]
Revised: [Equation]

Original: with N j ∼ N ( 0, σ 2 j ) .
Revised: with N j ∼ N ( 0, σ 2 j ) .

Original: Here we have two features: one cause X 1 = X cause and one effect X 2 =
Revised: Here we have two features: one cause X 1 = X cause and one effect X 2 =

Original: X effectof Y .
Revised: X effectof Y .

Original: By fixing σ 21 = 1 and σ 2 Y = 2 across domains but sampling σ 2 ∼ LogNormal ( 0, 0.5 ) , we create
Revised: By fixing σ 21 = 1 and σ 2 Y = 2 across domains but sampling σ 2 ∼ LogNormal ( 0, 0.5 ) , we create

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: a dataset in which X 2 is more predictive of Y than X 1 but less stable.
Revised: a dataset in which X 2 is more predictive of Y than X 1 but less stable.

Original: Importantly, as we knowthe true distribution over domains Q ( e ) =
Revised: Importantly, as we knowthe true distribution over domains Q ( e ) =

Original: LogNormal ( σ
Revised: LogNormal ( σ

Original: [Table]
Revised: [Table]

Original: 2 ; 0, 0.5 ) , we know the true risk quantiles.
Revised: 2 ; 0, 0.5 ) , we know the true risk quantiles.

Original: Fig.
Revised: Fig.

Original: 3 depicts results for different α ’s with m = 1000 domains and n = 100000 samples in each,using the mean-squared-error (MSE) loss.
Revised: 3 depicts results for different α ’s with m = 1000 domains and n = 100000 samples in each,using the mean-squared-error (MSE) loss.

Original: Here we see that: A: for each true quantile (x-axis),the corresponding α has the lowest risk (y-axis), confirming that the empirical α -quantile risk isa good estimate of the population α -quantile risk; B: As α → 1 , the estimated risk distribution off α approaches an invariant (or Dirac delta) distribution centered on the risk of the causal predictor;
Revised: Here we see that: A: for each true quantile (x-axis),the corresponding α has the lowest risk (y-axis), confirming that the empirical α -quantile risk isa good estimate of the population α -quantile risk; B: As α → 1 , the estimated risk distribution off α approaches an invariant (or Dirac delta) distribution centered on the risk of the causal predictor;

Original: C: the regression coefficients approach those of the causal predictor as α → 1 , trading predictiveperformance for robustness; and D: reducing the number of domains m reduces the accuracy of theestimated α -quantile risks.
Revised: C: the regression coefficients approach those of the causal predictor as α → 1 , trading predictiveperformance for robustness; and D: reducing the number of domains m reduces the accuracy of theestimated α -quantile risks.

Original: In Appendix G.1, we: (i) depict the risk CDFs corresponding to plot
Revised: In Appendix G.1, we: (i) depict the risk CDFs corresponding to plot

Original: B above, and discuss how they depict the predictors’ risk-robustness curves (G.1.1); and (ii) discussthe solutions of QRM on datasets in which σ 21 , σ 22 and/or σ 2 Y change over domains, compared toexisting invariance-seeking algorithms like IRM [9] and VREx [41] (G.1.2).
Revised: B above, and discuss how they depict the predictors’ risk-robustness curves (G.1.1); and (ii) discussthe solutions of QRM on datasets in which σ 21 , σ 22 and/or σ 2 Y change over domains, compared toexisting invariance-seeking algorithms like IRM [9] and VREx [41] (G.1.2).

Original: ColoredMNIST.
Revised: ColoredMNIST.

Original: We next consider the ColoredMNIST or CMNIST dataset [9].
Revised: We next consider the ColoredMNIST or CMNIST dataset [9].

Original: Here, the MNISTdataset is used to construct a binary classification task (0–4 or 5–9) in which digit color (red or green)is a highly-informative but spurious feature.
Revised: Here, the MNISTdataset is used to construct a binary classification task (0–4 or 5–9) in which digit color (red or green)is a highly-informative but spurious feature.

Original: In particular, the two training domains are constructedsuch that red digits have an 80% and 90% chance of belonging to class 0, while the single test domainis constructed such that they only have a 10% chance.
Revised: In particular, the two training domains are constructedsuch that red digits have an 80% and 90% chance of belonging to class 0, while the single test domainis constructed such that they only have a 10% chance.

Original: The goal is to learn an invariant predictorwhich uses only digit shape—a stable feature having a 75% chance of correctly determining the classin all 3 domains.
Revised: The goal is to learn an invariant predictorwhich uses only digit shape—a stable feature having a 75% chance of correctly determining the classin all 3 domains.

Original: We compare with IRM [9], GroupDRO [48], SD [110], IGA [111] and VREx [41]using: (i) random initialization (Xavier method [112]); and (ii) random initialization followed byseveral iterations of ERM.
Revised: We compare with IRM [9], GroupDRO [48], SD [110], IGA [111] and VREx [41]using: (i) random initialization (Xavier method [112]); and (ii) random initialization followed byseveral iterations of ERM.

Original: The ERM initialization or pretraining directly corresponds to the delicatepenalty “annealing” or warm-up periods used by most penalty-based methods [9, 41, 110, 111].
Revised: The ERM initialization or pretraining directly corresponds to the delicatepenalty “annealing” or warm-up periods used by most penalty-based methods [9, 41, 110, 111].

Original: Forall methods, we use a 2-hidden-layer MLP with 390 hidden units, the Adam optimizer, a learningrate of 0.0001 , and dropout with p = 0.2 .
Revised: Forall methods, we use a 2-hidden-layer MLP with 390 hidden units, the Adam optimizer, a learningrate of 0.0001 , and dropout with p = 0.2 .

Original: We sweep over five penalty weights for the baselines andfive α ’s for QRM/SQRM.
Revised: We sweep over five penalty weights for the baselines andfive α ’s for QRM/SQRM.

Original: See Appendix E.2 for more details.
Revised: See Appendix E.2 for more details.

Original: Table 1 shows that: (i) all methodsstruggle without ERM pretraining, explaining the need for penalty-annealing strategies in previousworks and corroborating the results of [113, Table 1]; (ii) with ERM pretraining, QRM matchesor outperforms IRM and VREx, even approaching oracle performance (that of ERM trained ongrayscale digits).
Revised: Table 1 shows that: (i) all methodsstruggle without ERM pretraining, explaining the need for penalty-annealing strategies in previousworks and corroborating the results of [113, Table 1]; (ii) with ERM pretraining, QRM matchesor outperforms IRM and VREx, even approaching oracle performance (that of ERM trained ongrayscale digits).

Original: These results suggest ERM pretraining as an effective strategy for DG methods—astrategy we employ in the next section.
Revised: These results suggest ERM pretraining as an effective strategy for DG methods—astrategy we employ in the next section.

Original: Fig.
Revised: Fig.

Original: 3 depicts the behavior of QRM with different α s.
Revised: 3 depicts the behavior of QRM with different α s.

Original: Here we see that: E: increasing α leadsto more consistent performance across domains, eventually forcing the model to ignore color andfocus on invariant, shaped-based prediction; and F: a predictor’s estimated accuracy-CDF depictsits accuracy-robustness curve, just as its estimated risk-CDF depicts its risk-robustness curve.
Revised: Here we see that: E: increasing α leadsto more consistent performance across domains, eventually forcing the model to ignore color andfocus on invariant, shaped-based prediction; and F: a predictor’s estimated accuracy-CDF depictsits accuracy-robustness curve, just as its estimated risk-CDF depicts its risk-robustness curve.

Original: Notethat α = 0.5 gives the best worst-case risk over the two training domains—the preferred solutionof DRO [48]—while α → 1 sacrifices risk for increased invariance or robustness.
Revised: Notethat α = 0.5 gives the best worst-case risk over the two training domains—the preferred solutionof DRO [48]—while α → 1 sacrifices risk for increased invariance or robustness.

Original: 6.2 DomainBed datasets: between synthetic and real-world shifts
Revised: 6.2 DomainBed datasets: between synthetic and real-world shifts

Original: We next consider two large-scale datasets from the DomainBed [38] suite: PACS and VLCS .
Revised: We next consider two large-scale datasets from the DomainBed [38] suite: PACS and VLCS .

Original: Whileboth of these datasets encode real-world shifts, each contains only four domains – three for trainingand one for testing – and therefore it is not possible to analyze the tail performance of our algorithms.
Revised: Whileboth of these datasets encode real-world shifts, each contains only four domains – three for trainingand one for testing – and therefore it is not possible to analyze the tail performance of our algorithms.

Original: However, in line with a litany of recent work, we offer a comparison of QRM to a range of standardbaselines in Table 2 in terms of the average performance across each choice of test domain;
Revised: However, in line with a litany of recent work, we offer a comparison of QRM to a range of standardbaselines in Table 2 in terms of the average performance across each choice of test domain;

Original: we
Revised: we

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: include a full set of results with additional baselines in Appendix G.2.
Revised: include a full set of results with additional baselines in Appendix G.2.

Original: Observe that on PACS , we beatstate-of-the-art algorithms across all three model-selection methods, with a notable 2-percentage-pointimprovement for the cross-validation selection method.
Revised: Observe that on PACS , we beatstate-of-the-art algorithms across all three model-selection methods, with a notable 2-percentage-pointimprovement for the cross-validation selection method.

Original: On VLCS , we match the state-of-the-art, witha slight improvement of nearly 1-percentage-point for the oracle selection method.
Revised: On VLCS , we match the state-of-the-art, witha slight improvement of nearly 1-percentage-point for the oracle selection method.

Original: 6.3 Real-world datasets
Revised: 6.3 Real-world datasets

Original: We now evaluate our methods on the real-world or in-the-wild distribution shifts of WILDS [12].
Revised: We now evaluate our methods on the real-world or in-the-wild distribution shifts of WILDS [12].

Original: We focus our evaluation on iWildCam [114] and OGB-MolPCBA [115, 116]—two large-scale classification datasets which have numerous test domains and thus facilitate a comparison of the riskdistributions and their quantiles.
Revised: We focus our evaluation on iWildCam [114] and OGB-MolPCBA [115, 116]—two large-scale classification datasets which have numerous test domains and thus facilitate a comparison of the riskdistributions and their quantiles.

Original: Additional comparisons (e.g. using average accuracy) can be foundin Appendix G.3.
Revised: Additional comparisons (e.g. using average accuracy) can be foundin Appendix G.3.

Original: Our results demonstrate that, across two distinct data types (images and moleculargraphs), QRM and SQRM offer superior quantile and superquantile performance respectively.
Revised: Our results demonstrate that, across two distinct data types (images and moleculargraphs), QRM and SQRM offer superior quantile and superquantile performance respectively.

Original: iWildCam.
Revised: iWildCam.

Original: We first consider the iWildCam image-classification dataset which has 243 trainingdomains and 48 test domains.
Revised: We first consider the iWildCam image-classification dataset which has 243 trainingdomains and 48 test domains.

Original: Here, the label Y is 1 of 182 different animal species and the domaine is the camera trap which took the photo.
Revised: Here, the label Y is 1 of 182 different animal species and the domaine is the camera trap which took the photo.

Original: In Table 3, we observe that QRM α does indeed tendto optimize the α -risk quantile, with larger α s during training resulting in lower test-time risksat the corresponding quantiles.
Revised: In Table 3, we observe that QRM α does indeed tendto optimize the α -risk quantile, with larger α s during training resulting in lower test-time risksat the corresponding quantiles.

Original: Similarly, Table 5 shows that SQRM α effectively optimizes thesuperquantile function, with α = 0.75 yielding particularly low values relative to ERM.
Revised: Similarly, Table 5 shows that SQRM α effectively optimizes thesuperquantile function, with α = 0.75 yielding particularly low values relative to ERM.

Original: In the twoleftmost panes of Fig. 4, we plot the (smoothed) test-time risk distributions.
Revised: In the twoleftmost panes of Fig. 4, we plot the (smoothed) test-time risk distributions.

Original: Here we see a cleartrend: as α increases, the tails of the risk distribution tend to drop below ERM, which corroboratesthe superior quantile and superquantile performance reported in Tables 3 and 5.
Revised: Here we see a cleartrend: as α increases, the tails of the risk distribution tend to drop below ERM, which corroboratesthe superior quantile and superquantile performance reported in Tables 3 and 5.

Original: Note that, in both Tables 3 and 5, QRM and SQRM tend to record lower average risks than ERM.
Revised: Note that, in both Tables 3 and 5, QRM and SQRM tend to record lower average risks than ERM.

Original: This has several plausible explanations.
Revised: This has several plausible explanations.

Original: First, the number of testing domains (48) is relatively small, which could result in a biased sample with respect to the training domains.
Revised: First, the number of testing domains (48) is relatively small, which could result in a biased sample with respect to the training domains.

Original: Second, the test domainsmay not represent i.i.d.
Revised: Second, the test domainsmay not represent i.i.d.

Original: draws from Q , with WILDS [12] test domains tending to be more challenging.
Revised: draws from Q , with WILDS [12] test domains tending to be more challenging.

Original: OGB-MolPCBA.
Revised: OGB-MolPCBA.

Original: We next consider the OGB-MolPCBA (or OGB ) dataset, which is a molecular graphclassification benchmark containing 44,930 training domains and 43,793 test domains with an averageof 3.6 samples per domain.
Revised: We next consider the OGB-MolPCBA (or OGB ) dataset, which is a molecular graphclassification benchmark containing 44,930 training domains and 43,793 test domains with an averageof 3.6 samples per domain.

Original: Tables 4 and 6 show that ERM achieves the lowest average test risk on OGB ,in contrast to the iWildCam results, while QRM α and SQRM α still achieve stronger quantile and superquantile performance.
Revised: Tables 4 and 6 show that ERM achieves the lowest average test risk on OGB ,in contrast to the iWildCam results, while QRM α and SQRM α still achieve stronger quantile and superquantile performance.

Original: Of particular note is the fact that our methods significantly outperform ERMwith respect to worst-case performance (columns/quantiles labeled 1.0); when QRM α and SQRM αare run with large values of α , we reduce the worst-case risk by more than a factor of two.
Revised: Of particular note is the fact that our methods significantly outperform ERMwith respect to worst-case performance (columns/quantiles labeled 1.0); when QRM α and SQRM αare run with large values of α , we reduce the worst-case risk by more than a factor of two.

Original: In Fig.
Revised: In Fig.

Original: 4,we again see that the risk distributions of QRM α and SQRM α have lighter tails than that of ERM.
Revised: 4,we again see that the risk distributions of QRM α and SQRM α have lighter tails than that of ERM.

Original: A new evaluation protocol for DG.
Revised: A new evaluation protocol for DG.

Original: The analysis provided in Tables 3-6 and Fig.
Revised: The analysis provided in Tables 3-6 and Fig.

Original: 4 diverges from thestandard evaluation protocol in DG [12, 38].
Revised: 4 diverges from thestandard evaluation protocol in DG [12, 38].

Original: Rather than evaluating an algorithm’s performance onaverage across test domains, we seek to understand the distribution of its performance —particularly inthe tails by means of the quantile and superquantile functions.
Revised: Rather than evaluating an algorithm’s performance onaverage across test domains, we seek to understand the distribution of its performance —particularly inthe tails by means of the quantile and superquantile functions.

Original: This new evaluation protocol lays barethe importance of multiple test domains in DG benchmarks, allowing predictors’ risk distributions tobe analyzed and compared.
Revised: This new evaluation protocol lays barethe importance of multiple test domains in DG benchmarks, allowing predictors’ risk distributions tobe analyzed and compared.

Original: Indeed, as DG is neither a worst-case nor an average-case problem, solelyreporting a predictor’s worst or average risk over test domains can be misleading when assessing itsability to generalize OOD (as shown in Tables 3-6).
Revised: Indeed, as DG is neither a worst-case nor an average-case problem, solelyreporting a predictor’s worst or average risk over test domains can be misleading when assessing itsability to generalize OOD (as shown in Tables 3-6).

Original: This underscores the necessity of incorporatingtail risk measures into a more holistic evaluation protocol for DG, ultimately providing a morenuanced and complete picture of a predictor’s ability to generalize OOD.
Revised: This underscores the necessity of incorporatingtail risk measures into a more holistic evaluation protocol for DG, ultimately providing a morenuanced and complete picture of a predictor’s ability to generalize OOD.

Original: In practice, which measureis preferred will depend on the application-of-interest, e.g. via a human-specified quantile-of-interest.
Revised: In practice, which measureis preferred will depend on the application-of-interest, e.g. via a human-specified quantile-of-interest.

Original:  Discussion
Revised:  Discussion

Original: On the assumption of i.i.d.
Revised: On the assumption of i.i.d.

Original: domains.
Revised: domains.

Original: For α to approximate the probability of generalizing, trainingand test domains must be i.i.d.-sampled.
Revised: For α to approximate the probability of generalizing, trainingand test domains must be i.i.d.-sampled.

Original: While this is rarely true in practice—e.g. hospitals haveshared funders, service providers, foundation dates, etc.—we can better satisfy this assumption bysubscribing to a new data collection process in which we collect training data from domains whichare representative of how the underlying system tends to change.
Revised: While this is rarely true in practice—e.g. hospitals haveshared funders, service providers, foundation dates, etc.—we can better satisfy this assumption bysubscribing to a new data collection process in which we collect training data from domains whichare representative of how the underlying system tends to change.

Original: For example: (i) randomly select100 US hospitals; (ii) gather and label data from these hospitals; (iii) train our system with thedesired α ; (iv) deploy our system to all US hospitals where it will be successful with probability≈ α .
Revised: For example: (i) randomly select100 US hospitals; (ii) gather and label data from these hospitals; (iii) train our system with thedesired α ; (iv) deploy our system to all US hospitals where it will be successful with probability≈ α .

Original: While this process may seem expensive, time-consuming and vulnerable (e.g. new hospitals),it offers a promising path to machine learning systems which generalize with high probability .
Revised: While this process may seem expensive, time-consuming and vulnerable (e.g. new hospitals),it offers a promising path to machine learning systems which generalize with high probability .

Original: Moreover, it is worth noting the alternative: prior works achieve generalization by assuming thatonly particular types of shifts can occur, e.g. covariate shifts [22, 117, 118], label shifts [118, 119],concept shifts [120], measurement shifts [19], mean shifts [109], shifts which leave the mechanism of
Revised: Moreover, it is worth noting the alternative: prior works achieve generalization by assuming thatonly particular types of shifts can occur, e.g. covariate shifts [22, 117, 118], label shifts [118, 119],concept shifts [120], measurement shifts [19], mean shifts [109], shifts which leave the mechanism of

Original: Y invariant [9, 39, 41, 45], etc.
Revised: Y invariant [9, 39, 41, 45], etc.

Original: In real-world settings, where the shift mechanisms are often unknown,such assumptions are both difficult to justify and impossible to test.
Revised: In real-world settings, where the shift mechanisms are often unknown,such assumptions are both difficult to justify and impossible to test.

Original: Future work could look to relaxthe i.i.d.-domains assumption by leveraging knowledge of domain dependencies (e.g. time).
Revised: Future work could look to relaxthe i.i.d.-domains assumption by leveraging knowledge of domain dependencies (e.g. time).

Original: Interpretable model selection.
Revised: Interpretable model selection.

Original: α approximates the probability with which our predictor willgeneralize with risk below the associated α -quantile value.
Revised: α approximates the probability with which our predictor willgeneralize with risk below the associated α -quantile value.

Original: Thus, α represents an interpretableparameterization of the risk-robustness trade-off.
Revised: Thus, α represents an interpretableparameterization of the risk-robustness trade-off.

Original: Such interpretability is critical for model selectionin DG, and for practitioners with application-specific requirements on performance and/or robustness.
Revised: Such interpretability is critical for model selectionin DG, and for practitioners with application-specific requirements on performance and/or robustness.

Original: The wider value of risk distributions.
Revised: The wider value of risk distributions.

Original: As demonstrated in § 6, a predictor’s risk distribution hasvalue beyond quantile-minimization—it estimates the probability associated with each level of risk.
Revised: As demonstrated in § 6, a predictor’s risk distribution hasvalue beyond quantile-minimization—it estimates the probability associated with each level of risk.

Original: Thus, regardless of the algorithm used, risk distributions can be used to analyze trained predictors.
Revised: Thus, regardless of the algorithm used, risk distributions can be used to analyze trained predictors.

Original:  Conclusion
Revised:  Conclusion

Original: We have presented QRM for achieving probable domain generalization, motivated by the argumentthat DG should seek predictors which generalize with high probability rather than in the worst-case oron-average.
Revised: We have presented QRM for achieving probable domain generalization, motivated by the argumentthat DG should seek predictors which generalize with high probability rather than in the worst-case oron-average.

Original: By explicitly relating training and test domains as draws from the same underlying metadistribution, we proposed to learn predictors with minimal α -quantile risk under the training domains.
Revised: By explicitly relating training and test domains as draws from the same underlying metadistribution, we proposed to learn predictors with minimal α -quantile risk under the training domains.

Original: Theoretically, we proved that α does indeed approximate the probability of generalizing and that QRMrecovers the causal predictor as α → 1 .
Revised: Theoretically, we proved that α does indeed approximate the probability of generalizing and that QRMrecovers the causal predictor as α → 1 .

Original: Empirically, we demonstrated the importance of multiple testdomains and tail performance in DG benchmarks, and that our algorithms perform well in practice. 
Revised: Empirically, we demonstrated the importance of multiple testdomains and tail performance in DG benchmarks, and that our algorithms perform well in practice. Checklist

Revised: The checklist follows the references.

Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Revised: For example:

Revised: • Did you include the license to the code and datasets?

Revised: [Yes] See Section X.•

Revised: Did you include the license to the code and datasets?

Revised: [No] The code and the data are proprietary.•

Revised: Did you include the license to the code and datasets?

Revised: [N/A]

Revised: Please do not modify the questions and only use the provided macros for your answers.

Revised: Note that the

Revised: Checklist section does not count towards the page limit.

Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Revised: For all authors...

Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Revised: [Yes](b) Did you describe the limitations of your work?

Revised: [Yes] See Appendix H.(c)

Revised: Did you discuss any potential negative societal impacts of your work?

Revised: [N/A](d)

Revised: Have you read the ethics review guidelines and ensured that your paper conforms to them?[Yes]2.

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes](b) Did you include complete proofs of all theoretical results?

Revised: [Yes]3.

Revised: If you ran experiments...

Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?

Revised: [Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?

Revised: [Yes](c)

Revised: Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?

Revised: [No]

Revised: [Table]

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type ofGPUs, internal cluster, or cloud provider)?

Revised: [Yes]

Revised: [Table]

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Revised: (a)

Revised: If your work uses existing assets, did you cite the creators?

Revised: [Yes](b) Did you mention the license of the assets?

Revised: [N/A]

Revised: [Table]

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [Yes]

Revised: [Table]

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Revised: [Table]

Revised: using/curating?

Revised: [N/A]

Revised: [Table]

Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Revised: [Table]

Revised: information or offensive content?

Revised: [N/A]

Revised: [Table]

Revised: If you used crowdsourcing or conducted research with human subjects...

Revised: [Table]

Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applica-

Revised: [Table]

Revised: ble?

Revised: [N/A]

Revised: [Table]

Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board

Revised: [Table]

Revised: (IRB) approvals, if applicable?

Revised: [N/A](c)

Revised: Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?

Revised: [N/A]

Revised: [Table]