Original: B ITRATE -C ONSTRAINED DRO: B EYOND W ORST C ASE R OBUSTNESS T O U NKNOWN G ROUP S HIFTS
Revised: B ITRATE -C ONSTRAINED DRO: B EYOND W ORST C ASE R OBUSTNESS T O U NKNOWN G ROUP S HIFTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Although training machine learning models for robustness is critical for real-world adoption, determining how to best ensure robustness remains an open problem.
Revised: Although training machine learning models for robustness is critical for real-world adoption, determining how to best ensure robustness remains an open problem.

Original: Some methods ( e.g., DRO) are overly conservative, while others ( e.g., Group
Revised: Some methods ( e.g., DRO) are overly conservative, while others ( e.g., Group

Original: DRO) require domain knowledge that may be hard to obtain.
Revised: DRO) require domain knowledge that may be hard to obtain.

Original: In this work, we address limitations in prior approaches by assuming a more nuanced form of group shift: conditioned on the label, we assume that the true group function is simple.
Revised: In this work, we address limitations in prior approaches by assuming a more nuanced form of group shift: conditioned on the label, we assume that the true group function is simple.

Original: For example, we may expect that group shifts occur along high-level features ( e.g., image background, lighting).
Revised: For example, we may expect that group shifts occur along high-level features ( e.g., image background, lighting).

Original: Thus, we aim to learn a model that maintains high accuracy on simple group functions realized by these features, but need not spend valuable model capacity achieving high accuracy on contrived groups of examples.
Revised: Thus, we aim to learn a model that maintains high accuracy on simple group functions realized by these features, but need not spend valuable model capacity achieving high accuracy on contrived groups of examples.

Original: Based on this idea, we formulate a two-player game where conditioned on the label the adversary can only separate datapoints into potential groups using simple features, which corresponds to a bitrate constraint on the adversary’s capacity.
Revised: Based on this idea, we formulate a two-player game where conditioned on the label the adversary can only separate datapoints into potential groups using simple features, which corresponds to a bitrate constraint on the adversary’s capacity.

Original: Our resulting practical algorithm, Bitrate-Constrained DRO ( BR-DRO ), does not require group information on training samples yet matches the performance of Group DRO on datasets that have known groups or are long-tailed.
Revised: Our resulting practical algorithm, Bitrate-Constrained DRO ( BR-DRO ), does not require group information on training samples yet matches the performance of Group DRO on datasets that have known groups or are long-tailed.

Original: Our theoretical analysis reveals that in some settings BR-DRO objective can provably yield statistically efficient and less pessimistic solutions than unconstrained DRO.
Revised: Our theoretical analysis reveals that in some settings BR-DRO objective can provably yield statistically efficient and less pessimistic solutions than unconstrained DRO.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learned models may perform poorly when tested on distributions that differ from the training distribution.
Revised: Learned models may perform poorly when tested on distributions that differ from the training distribution.

Original: A common form of distribution shift is group shift, where the source and target differ only in the marginal distribution over finite groups or sub-populations, with no change in group conditionals (Oren et al., 2019; Duchi et al., 2019; Sagawa et al., 2019).
Revised: A common form of distribution shift is group shift, where the source and target differ only in the marginal distribution over finite groups or sub-populations, with no change in group conditionals (Oren et al., 2019; Duchi et al., 2019; Sagawa et al., 2019).

Original: Prior works consider various approaches to address group shift.
Revised: Prior works consider various approaches to address group shift.

Original: One solution is to ensure robustness to worst case shifts using distributionally robust optimization (DRO) (Bagnell, 2005; Ben-Tal et al., 2013; Duchi et al., 2016), which considers a two-player game where a learner minimizes risk on distributions chosen by an adversary from a predefined uncertainty set.
Revised: One solution is to ensure robustness to worst case shifts using distributionally robust optimization (DRO) (Bagnell, 2005; Ben-Tal et al., 2013; Duchi et al., 2016), which considers a two-player game where a learner minimizes risk on distributions chosen by an adversary from a predefined uncertainty set.

Original: As the adversary is unconstrained in proposing distributions, DRO often yields overly pessimistic solutions (Hu et al., 2018) and can suffer from statistical challenges (Duchi et al., 2019).
Revised: As the adversary is unconstrained in proposing distributions, DRO often yields overly pessimistic solutions (Hu et al., 2018) and can suffer from statistical challenges (Duchi et al., 2019).

Original: Methods like Group DRO (Sagawa et al., 2019) avoid overly pessimistic solutions by assuming knowledge of group membership for each training example.
Revised: Methods like Group DRO (Sagawa et al., 2019) avoid overly pessimistic solutions by assuming knowledge of group membership for each training example.

Original: However, these group-based methods provide no guarantees on shifts that deviate from the predefined groups, and are not applicable to problems that lack group knowledge.
Revised: However, these group-based methods provide no guarantees on shifts that deviate from the predefined groups, and are not applicable to problems that lack group knowledge.

Original: In this work, we therefore ask: Can we train non-pessimistic robust models without access to group information?
Revised: In this work, we therefore ask: Can we train non-pessimistic robust models without access to group information?

Original: We address this question by considering a more nuanced assumption on the structure of the underlying groups.
Revised: We address this question by considering a more nuanced assumption on the structure of the underlying groups.

Original: We assume that, conditioned on the label, group boundaries are realized by high-level features that depend on a small set of underlying factors.
Revised: We assume that, conditioned on the label, group boundaries are realized by high-level features that depend on a small set of underlying factors.

Original: This leads to simpler group functions with large margins and simple decision boundaries (Figure 1 (left) ).
Revised: This leads to simpler group functions with large margins and simple decision boundaries (Figure 1 (left) ).

Original: Invoking the principle of minimum description length (Gr ¨ unwald, 2007), restricting our adversary to functions that satisfy this assumption corresponds to a bitrate constraint.
Revised: Invoking the principle of minimum description length (Gr ¨ unwald, 2007), restricting our adversary to functions that satisfy this assumption corresponds to a bitrate constraint.

Original: In DRO, the adversary upweights points with higher losses under the current learner, which in practice often correspond to examples that belong to a rare group, contain complex patterns, or are mislabeled (Carlini et al., 2019; Toneva et al., 2018).
Revised: In DRO, the adversary upweights points with higher losses under the current learner, which in practice often correspond to examples that belong to a rare group, contain complex patterns, or are mislabeled (Carlini et al., 2019; Toneva et al., 2018).

Original: Restricting the adversary’s capacity prevents it from upweighting individual hard or mislabeled examples (as they cannot be identified with simple features), and biases it towards identifying erroneous data points misclassified by simple features.
Revised: Restricting the adversary’s capacity prevents it from upweighting individual hard or mislabeled examples (as they cannot be identified with simple features), and biases it towards identifying erroneous data points misclassified by simple features.

Original: This also complements the failure mode of neural networks trained
Revised: This also complements the failure mode of neural networks trained

Original: [Figure]
Revised: [Figure]

Original: with stochastic gradient descent (SGD) that rely on simple spurious features that correctly classify points in the majority group but may fail on minority groups (Blodgett et al., 2016).
Revised: with stochastic gradient descent (SGD) that rely on simple spurious features that correctly classify points in the majority group but may fail on minority groups (Blodgett et al., 2016).

Original: The main contribution of this paper is Bitrate-Constrained DRO ( BR-DRO ), a supervised learning procedure that provides robustness to distribution shifts along groups realized by simple functions.
Revised: The main contribution of this paper is Bitrate-Constrained DRO ( BR-DRO ), a supervised learning procedure that provides robustness to distribution shifts along groups realized by simple functions.

Original: Despite not using group information on training examples, we demonstrate that BR-DRO can match the performance of methods requiring them.
Revised: Despite not using group information on training examples, we demonstrate that BR-DRO can match the performance of methods requiring them.

Original: We also find that BR-DRO correctly identifies true minority points, whereas DRO without group information does not.
Revised: We also find that BR-DRO correctly identifies true minority points, whereas DRO without group information does not.

Original: This indicates that not optimizing for performance on contrived worst-case shifts can reduce the pessimism inherent in DRO.
Revised: This indicates that not optimizing for performance on contrived worst-case shifts can reduce the pessimism inherent in DRO.

Original: It further validates: (i) our assumption on the simple nature of group shift; and (ii) that our method of capacity control meaningfully structures the uncertainty set to be robust to such shifts.
Revised: It further validates: (i) our assumption on the simple nature of group shift; and (ii) that our method of capacity control meaningfully structures the uncertainty set to be robust to such shifts.

Original: As a consequence of the constraint, we also find that BR-DRO is robust to random noise in the training data (Song et al., 2022), since it cannot form “groups” entirely based on randomly mislabeled points with low bitrate features.
Revised: As a consequence of the constraint, we also find that BR-DRO is robust to random noise in the training data (Song et al., 2022), since it cannot form “groups” entirely based on randomly mislabeled points with low bitrate features.

Original: This is in contrast with existing methods that use the learner’s training error to up-weight arbitrary sets of difficult training points ( e.g., Liu et al., 2021; Levy et al., 2020), which we show are highly susceptible to label noise (see Figure 1).
Revised: This is in contrast with existing methods that use the learner’s training error to up-weight arbitrary sets of difficult training points ( e.g., Liu et al., 2021; Levy et al., 2020), which we show are highly susceptible to label noise (see Figure 1).

Original: Finally, we theoretically analyze our approach— characterizing how the degree of constraint on the adversary can effect worst risk estimation and excess risk (pessimism) bounds, as well as convergence rates for specific online solvers.
Revised: Finally, we theoretically analyze our approach— characterizing how the degree of constraint on the adversary can effect worst risk estimation and excess risk (pessimism) bounds, as well as convergence rates for specific online solvers.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Prior works in robust ML (e.g., Li et al., 2018; Lipton et al., 2018; Goodfellow et al., 2014) address various forms of adversarial or structured shifts.
Revised: Prior works in robust ML (e.g., Li et al., 2018; Lipton et al., 2018; Goodfellow et al., 2014) address various forms of adversarial or structured shifts.

Original: We specifically review prior work on robustness to group shifts.
Revised: We specifically review prior work on robustness to group shifts.

Original: While those based on DRO optimize for worst-case shifts in an explicit uncertainty set, the robust set is implicit for some others, with most using some form of importance weighting.
Revised: While those based on DRO optimize for worst-case shifts in an explicit uncertainty set, the robust set is implicit for some others, with most using some form of importance weighting.

Original: Distributionally robust optimization (DRO).
Revised: Distributionally robust optimization (DRO).

Original: DRO methods generally optimize for worst-case performance on joint ( x , y ) distributions that lie in an f -divergence ball (uncertainty set) around the training distribution (Ben-Tal et al., 2013; Rahimian & Mehrotra, 2019; Bertsimas et al., 2018;
Revised: DRO methods generally optimize for worst-case performance on joint ( x , y ) distributions that lie in an f -divergence ball (uncertainty set) around the training distribution (Ben-Tal et al., 2013; Rahimian & Mehrotra, 2019; Bertsimas et al., 2018;

Original: Blanchet & Murthy, 2019; Miyato et al., 2018; Duchi et al., 2016; Duchi & Namkoong, 2021).
Revised: Blanchet & Murthy, 2019; Miyato et al., 2018; Duchi et al., 2016; Duchi & Namkoong, 2021).

Original: Hu et al.
Revised: Hu et al.

Original: (2018) highlights that the conservative nature of DRO may lead to degenerate solutions when the unrestricted adversary uniformly upweights all misclassified points.
Revised: (2018) highlights that the conservative nature of DRO may lead to degenerate solutions when the unrestricted adversary uniformly upweights all misclassified points.

Original: Sagawa et al.
Revised: Sagawa et al.

Original: (2019) proposes to address this by limiting the adversary to shifts that only differ in marginals over predefined groups.
Revised: (2019) proposes to address this by limiting the adversary to shifts that only differ in marginals over predefined groups.

Original: However, in addition to it being difficult to obtain this information, Kearns et al.
Revised: However, in addition to it being difficult to obtain this information, Kearns et al.

Original: (2018) raise “gerrymandering” concerns with notions of robustness that fix a small number of groups apriori.
Revised: (2018) raise “gerrymandering” concerns with notions of robustness that fix a small number of groups apriori.

Original: While they propose a solution that looks at exponentially many subgroups defined over protected attributes, our method does not assume access to such attributes and aims to be fair on them as long as they are realized by simple functions.
Revised: While they propose a solution that looks at exponentially many subgroups defined over protected attributes, our method does not assume access to such attributes and aims to be fair on them as long as they are realized by simple functions.

Original: Finally, Zhai et al.
Revised: Finally, Zhai et al.

Original: (2021) avoid conservative solutions by solving the DRO objective over randomized predictors learned through boosting.
Revised: (2021) avoid conservative solutions by solving the DRO objective over randomized predictors learned through boosting.

Original: We consider deterministic and over-parameterized learners and instead constrain the adversary’s class.
Revised: We consider deterministic and over-parameterized learners and instead constrain the adversary’s class.

Original: Constraining the DRO uncertainty set.
Revised: Constraining the DRO uncertainty set.

Original: In the marginal DRO setting, Duchi et al.
Revised: In the marginal DRO setting, Duchi et al.

Original: (2019) limit the adversary via easier-to-control reproducing kernel hilbert spaces (RKHS) or bounded H¨older continuous functions (Liu & Ziebart, 2014; Wen et al., 2014).
Revised: (2019) limit the adversary via easier-to-control reproducing kernel hilbert spaces (RKHS) or bounded H¨older continuous functions (Liu & Ziebart, 2014; Wen et al., 2014).

Original: While this reduces the statistical error in worst risk estimation, the size of the uncertainty set (scales with the data) remains too large to avoid cases where an adversary can re-weight mislabeled and hard examples from the majority set (Carlini et al., 2019).
Revised: While this reduces the statistical error in worst risk estimation, the size of the uncertainty set (scales with the data) remains too large to avoid cases where an adversary can re-weight mislabeled and hard examples from the majority set (Carlini et al., 2019).

Original: In contrast, we restrict the adversary even for large datasets where the estimation error would be low, as this would reduce excess risk when we only care about robustness to rare sub-populations defined by simple functions.
Revised: In contrast, we restrict the adversary even for large datasets where the estimation error would be low, as this would reduce excess risk when we only care about robustness to rare sub-populations defined by simple functions.

Original: Additionally, while their analysis and method prefers the adversary’s objective to have a strong dual, we show empirical results on real-world datasets and generalization bounds where the adversary’s objective is not necessarily convex.
Revised: Additionally, while their analysis and method prefers the adversary’s objective to have a strong dual, we show empirical results on real-world datasets and generalization bounds where the adversary’s objective is not necessarily convex.

Original: Robustness to group shifts without demographics.
Revised: Robustness to group shifts without demographics.

Original: Recent works (Sohoni et al., 2020; Creager et al., 2021; Bao & Barzilay, 2022) that aim to achieve group robustness without access to group labels employ various heuristics where the robust set is implicit while others require data from multiple domains (Arjovsky et al., 2019; Yao et al., 2022) or ability to query test samples (Lee et al., 2022).
Revised: Recent works (Sohoni et al., 2020; Creager et al., 2021; Bao & Barzilay, 2022) that aim to achieve group robustness without access to group labels employ various heuristics where the robust set is implicit while others require data from multiple domains (Arjovsky et al., 2019; Yao et al., 2022) or ability to query test samples (Lee et al., 2022).

Original: Liu et al.
Revised: Liu et al.

Original: (2021) use training losses for a heavily regularized model trained with empirical risk minimization (ERM) to directly identify minority data points with higher losses and re-train on the dataset that up-weights the identified set.
Revised: (2021) use training losses for a heavily regularized model trained with empirical risk minimization (ERM) to directly identify minority data points with higher losses and re-train on the dataset that up-weights the identified set.

Original: Nam et al.
Revised: Nam et al.

Original: (2020) take a similar approach.
Revised: (2020) take a similar approach.

Original: Other methods (Idrissi et al., 2022) propose simple baselines that subsample the majority class in the absence of group demographics and the majority group in its presence.
Revised: Other methods (Idrissi et al., 2022) propose simple baselines that subsample the majority class in the absence of group demographics and the majority group in its presence.

Original: Hashimoto et al.
Revised: Hashimoto et al.

Original: (2018) find
Revised: (2018) find

Original: DRO over a χ 2 -divergence ball can reduce the otherwise increasing disparity of per-group risks in a dynamical system.
Revised: DRO over a χ 2 -divergence ball can reduce the otherwise increasing disparity of per-group risks in a dynamical system.

Original: Since it does not use features to upweight points (like BR-DRO ) it is vulnerable to label noise.
Revised: Since it does not use features to upweight points (like BR-DRO ) it is vulnerable to label noise.

Original: Same can be said about some other works ( e.g., Liu et al.
Revised: Same can be said about some other works ( e.g., Liu et al.

Original: ; Nam et al.
Revised: ; Nam et al.

Original: Importance weighting in deep learning.
Revised: Importance weighting in deep learning.

Original: Finally, numerous works (Duchi et al., 2016; Levy et al., 2020; Lipton et al., 2018; Oren et al., 2019) enforce robustness by re-weighting losses on individual data points.
Revised: Finally, numerous works (Duchi et al., 2016; Levy et al., 2020; Lipton et al., 2018; Oren et al., 2019) enforce robustness by re-weighting losses on individual data points.

Original: Recent investigations (Soudry et al., 2018; Byrd & Lipton, 2019; Lu et al., 2022) reveal that such objectives have little impact on the learned solution in interpolation regimes.
Revised: Recent investigations (Soudry et al., 2018; Byrd & Lipton, 2019; Lu et al., 2022) reveal that such objectives have little impact on the learned solution in interpolation regimes.

Original: One way to avoid this pitfall is to train with heavily regularized models (Sagawa et al., 2019; 2020) and employ early stopping.
Revised: One way to avoid this pitfall is to train with heavily regularized models (Sagawa et al., 2019; 2020) and employ early stopping.

Original: Another way is to subsample certain points, as opposed to up-weighting (Idrissi et al., 2022).
Revised: Another way is to subsample certain points, as opposed to up-weighting (Idrissi et al., 2022).

Original: In this work, we use both techniques while training our objective and the baselines, ensuring that the regularized class is robust to shifts under misspecification (Wen et al., 2014).
Revised: In this work, we use both techniques while training our objective and the baselines, ensuring that the regularized class is robust to shifts under misspecification (Wen et al., 2014).

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We introduce the notation we use in the rest of the paper and describe the DRO problem.
Revised: We introduce the notation we use in the rest of the paper and describe the DRO problem.

Original: In the following section, we will formalize our assumptions on the nature of the shift before introducing our optimization objective and algorithm.
Revised: In the following section, we will formalize our assumptions on the nature of the shift before introducing our optimization objective and algorithm.

Original: Notation.
Revised: Notation.

Original: With covariates X ⊂ R d
Revised: With covariates X ⊂ R d

Original: and labels Y , the given source P and unknown true target
Revised: and labels Y , the given source P and unknown true target

Original: Q 0 are measures over the measurable space ( X × Y , Σ) and have densities p and q 0 respectively (w.r.t.
Revised: Q 0 are measures over the measurable space ( X × Y , Σ) and have densities p and q 0 respectively (w.r.t.

Original: base measure µ ).
Revised: base measure µ ).

Original: The learner’s choice is a hypothesis h : X (cid:55)→ Y in class H ⊂
Revised: The learner’s choice is a hypothesis h : X (cid:55)→ Y in class H ⊂

Original: L 2 ( P ) , and the adversary’s action in standard DRO is a target distribution Q in set Q P,κ :=
Revised: L 2 ( P ) , and the adversary’s action in standard DRO is a target distribution Q in set Q P,κ :=

Original: { Q : Q ≪
Revised: { Q : Q ≪

Original: P, D f ( Q || P ) ≤ κ } .
Revised: P, D f ( Q || P ) ≤ κ } .

Original: Here, D f is the f -divergence between Q and P for a convex function f 1 with f (1) = 0 .
Revised: Here, D f is the f -divergence between Q and P for a convex function f 1 with f (1) = 0 .

Original: An equivalent action space for the adversary is the set of re-weighting functions:
Revised: An equivalent action space for the adversary is the set of re-weighting functions:

Original: W P,κ = { w : X × Y
Revised: W P,κ = { w : X × Y

Original: (cid:55)→ R : w is measurable under P, E P [ w ] = 1 , E P f ( w ) ≤ κ } (1)
Revised: (cid:55)→ R : w is measurable under P, E P [ w ] = 1 , E P f ( w ) ≤ κ } (1)

Original: For a convex loss function l : Y × Y (cid:55)→ R + , we denote l ( h ) as the function over ( x , y ) that evaluates l ( h ( x ) , y ) , and use l 0 − 1 to denote the loss function 1 ( h ( x )
Revised: For a convex loss function l : Y × Y (cid:55)→ R + , we denote l ( h ) as the function over ( x , y ) that evaluates l ( h ( x ) , y ) , and use l 0 − 1 to denote the loss function 1 ( h ( x )

Original: ̸ = y ) .
Revised: ̸ = y ) .

Original: Given either distribution
Revised: Given either distribution

Original: Q ∈ Q P,κ , or a re-weighting function w ∈ W P,κ , the risk of a learner h is:
Revised: Q ∈ Q P,κ , or a re-weighting function w ∈ W P,κ , the risk of a learner h is:

Original: [Equation]
Revised: [Equation]

Original: Note the overload of notation for R ( h, · ) .
Revised: Note the overload of notation for R ( h, · ) .

Original: If the adversary is stochastic it picks a mixed action δ ∈ ∆( W P,κ ) , which is the set of all distributions over W P,κ .
Revised: If the adversary is stochastic it picks a mixed action δ ∈ ∆( W P,κ ) , which is the set of all distributions over W P,κ .

Original: Whenever it is clear, we drop P, κ .
Revised: Whenever it is clear, we drop P, κ .

Original: Unconstrained DRO (Ben-Tal et al., 2013).
Revised: Unconstrained DRO (Ben-Tal et al., 2013).

Original: This is a min-max optimization problem understood as a two-player game, where the learner chooses a hypothesis, to minimize risk on the worst distribution that the adversary can choose from its set.
Revised: This is a min-max optimization problem understood as a two-player game, where the learner chooses a hypothesis, to minimize risk on the worst distribution that the adversary can choose from its set.

Original: Formally, this is given by Equation 3.
Revised: Formally, this is given by Equation 3.

Original: The first equivalence is clear from the definitions and for the second since R ( h, Q ) is linear in Q , the supremum over ∆( W P,κ ) is a Dirac delta over the best weighting in W P,κ .
Revised: The first equivalence is clear from the definitions and for the second since R ( h, Q ) is linear in Q , the supremum over ∆( W P,κ ) is a Dirac delta over the best weighting in W P,κ .

Original: In the next section, we will see how a bitrate-constrained adversary can only pick certain actions from ∆( W P,κ ) .
Revised: In the next section, we will see how a bitrate-constrained adversary can only pick certain actions from ∆( W P,κ ) .

Original: [Equation]
Revised: [Equation]

Original: Group Shift.
Revised: Group Shift.

Original: While the DRO framework in Section 3 is broad and addresses any unstructured shift, we focus on the specific case of group shift.
Revised: While the DRO framework in Section 3 is broad and addresses any unstructured shift, we focus on the specific case of group shift.

Original: First, for a given pair of measures P, Q we define what we mean by the group structure G P,Q (Definition 3.1).
Revised: First, for a given pair of measures P, Q we define what we mean by the group structure G P,Q (Definition 3.1).

Original: Intuitively, it is a set of sub-populations along which the distribution shifts, defined in a way that makes them uniquely identifiable.
Revised: Intuitively, it is a set of sub-populations along which the distribution shifts, defined in a way that makes them uniquely identifiable.

Original: For e.g., in the
Revised: For e.g., in the

Original: Waterbirds dataset (Figure 1), there are four groups given by combinations of (label, background).
Revised: Waterbirds dataset (Figure 1), there are four groups given by combinations of (label, background).

Original: Corollary 3.2 follows immediately from the definition of G P,Q .
Revised: Corollary 3.2 follows immediately from the definition of G P,Q .

Original: Using this definition, the standard group shift assumption (Sagawa et al., 2019) can be formally re-stated as Assumption 3.3.
Revised: Using this definition, the standard group shift assumption (Sagawa et al., 2019) can be formally re-stated as Assumption 3.3.

Original: Definition 3.1 (group structure G P,Q ) .
Revised: Definition 3.1 (group structure G P,Q ) .

Original: For Q ≪ P the group structure G P,Q = { G k } Kk =1 is the smallest finite set of disjoint groups { G k } Kk =1 s.t. Q ( ∪ Kk =1 G k )
Revised: For Q ≪ P the group structure G P,Q = { G k } Kk =1 is the smallest finite set of disjoint groups { G k } Kk =1 s.t. Q ( ∪ Kk =1 G k )

Original: =1 and ∀
Revised: =1 and ∀

Original: k
Revised: k

Original: (i) G k ∈ Σ , Q ( G k ) > 0 and (ii) p ( x , y | G k )
Revised: (i) G k ∈ Σ , Q ( G k ) > 0 and (ii) p ( x , y | G k )

Original: =
Revised: =

Original: q ( x , y | G k ) > 0 a.e . in µ .
Revised: q ( x , y | G k ) > 0 a.e . in µ .

Original: If such a structure exists then G P,Q is well defined.
Revised: If such a structure exists then G P,Q is well defined.

Original: Corollary 3.2 (uniqueness of G P,Q ) .
Revised: Corollary 3.2 (uniqueness of G P,Q ) .

Original: ∀ P, Q , the structure G ( P, Q ) is unique if it is well defined.
Revised: ∀ P, Q , the structure G ( P, Q ) is unique if it is well defined.

Original: Assumption 3.3 (standard group shift) .
Revised: Assumption 3.3 (standard group shift) .

Original: There exists a well-defined group structure G P,Q 0 s.t. target
Revised: There exists a well-defined group structure G P,Q 0 s.t. target

Original: Q 0 differs from P only in terms of marginal probabilities over all G ∈ G P,Q 0 .
Revised: Q 0 differs from P only in terms of marginal probabilities over all G ∈ G P,Q 0 .

Original:  B ITRATE -C ONSTRAINED DRO
Revised:  B ITRATE -C ONSTRAINED DRO

Original: We begin with a note on the expressivity of the adversary in Unconstrained DRO and formally introduce the assumption we make on the nature of shift.
Revised: We begin with a note on the expressivity of the adversary in Unconstrained DRO and formally introduce the assumption we make on the nature of shift.

Original: Then, we build intuition for why unconstrained adversaries fail but restricted ones do better under our assumption.
Revised: Then, we build intuition for why unconstrained adversaries fail but restricted ones do better under our assumption.

Original: Finally, we state our main objective and discuss a specific instance of it.
Revised: Finally, we state our main objective and discuss a specific instance of it.

Original: How expressive is unconstrained adversary?
Revised: How expressive is unconstrained adversary?

Original: Note that the set W P,κ includes all measurable functions (under P ) such that the re-weighted distribution is bounded in f -divergence (by κ ).
Revised: Note that the set W P,κ includes all measurable functions (under P ) such that the re-weighted distribution is bounded in f -divergence (by κ ).

Original: While prior works (Shafieezadeh Abadeh et al., 2015; Duchi et al., 2016) shrink κ to construct confidence intervals, this only controls the total mass that can be moved between measurable sets G 1 , G 2 ∈ Σ , but does not restrict the choice of G 1 and G 2 itself.
Revised: While prior works (Shafieezadeh Abadeh et al., 2015; Duchi et al., 2016) shrink κ to construct confidence intervals, this only controls the total mass that can be moved between measurable sets G 1 , G 2 ∈ Σ , but does not restrict the choice of G 1 and G 2 itself.

Original: As noted by Hu et al.
Revised: As noted by Hu et al.

Original: , such an adversary is highly expressive, and optimizing for the worst case only leads to the solution of empirical risk minimization (ERM) under l 0 − 1 loss.
Revised: , such an adversary is highly expressive, and optimizing for the worst case only leads to the solution of empirical risk minimization (ERM) under l 0 − 1 loss.

Original: Thus, we can conclude that DRO recovers degenerate solutions because the worst target in W P,κ lies far from the subspace of naturally occurring targets.
Revised: Thus, we can conclude that DRO recovers degenerate solutions because the worst target in W P,κ lies far from the subspace of naturally occurring targets.

Original: Since it is hard to precisely characterize natural targets we make a nuanced assumption: the target Q 0 only upsamples those rare subpopulations that are misclassified by simple features.
Revised: Since it is hard to precisely characterize natural targets we make a nuanced assumption: the target Q 0 only upsamples those rare subpopulations that are misclassified by simple features.

Original: We state this formally in Assumption 4.2 after we define the bitrate-constrained function class W ( γ ) in Definition 4.1.
Revised: We state this formally in Assumption 4.2 after we define the bitrate-constrained function class W ( γ ) in Definition 4.1.

Original: Definition 4.1.
Revised: Definition 4.1.

Original: A function class W ( γ ) is bitrate-constrained if there exists a data independent prior π , s.t. W ( γ ) =
Revised: A function class W ( γ ) is bitrate-constrained if there exists a data independent prior π , s.t. W ( γ ) =

Original: { E [ δ ] : δ ∈ ∆( W ) , KL ( δ || π ) ≤ γ } .
Revised: { E [ δ ] : δ ∈ ∆( W ) , KL ( δ || π ) ≤ γ } .

Original: Assumption 4.2 (simple group shift) .
Revised: Assumption 4.2 (simple group shift) .

Original: Target Q 0 satisfies Assumption 3.3 (group shift) w.r.t.
Revised: Target Q 0 satisfies Assumption 3.3 (group shift) w.r.t.

Original: source
Revised: source

Original: P .
Revised: P .

Original: Additionally, for some prior π and a small γ ∗ , the re-weighting function q 0 /p lies in a bitrateconstrained class W ( γ ∗ ) .
Revised: Additionally, for some prior π and a small γ ∗ , the re-weighting function q 0 /p lies in a bitrateconstrained class W ( γ ∗ ) .

Original: In other words, for every group G ∈ G ( P, Q 0 ) , ∃ w G ∈ W ( γ ∗ ) s.t.
Revised: In other words, for every group G ∈ G ( P, Q 0 ) , ∃ w G ∈ W ( γ ∗ ) s.t.

Original: (( x , y ) ∈ G )
Revised: (( x , y ) ∈ G )

Original: =
Revised: =

Original: w G a.e.. We refer to such a G as a simple group that is realized in W ( γ ∗ ) .
Revised: w G a.e.. We refer to such a G as a simple group that is realized in W ( γ ∗ ) .

Original: Under the principle of minimum description length (Gr ¨ unwald, 2007)
Revised: Under the principle of minimum description length (Gr ¨ unwald, 2007)

Original: any deviation from the prior ( i.e., KL ( δ || π ) ) increases the description length of the encoding δ ∈ ∆( W ) , thus we refer to W ( γ ) as being bitrate-constrained in the sense that it contains functions (means of distributions) that can be described with a limited number of bits given the prior π .
Revised: any deviation from the prior ( i.e., KL ( δ || π ) ) increases the description length of the encoding δ ∈ ∆( W ) , thus we refer to W ( γ ) as being bitrate-constrained in the sense that it contains functions (means of distributions) that can be described with a limited number of bits given the prior π .

Original: Next we present arguments for why identifiability of simple (satisfy Assumption 4.2) minority groups can be critical for robustness.
Revised: Next we present arguments for why identifiability of simple (satisfy Assumption 4.2) minority groups can be critical for robustness.

Original: Neural networks can perform poorly on simple minorities.
Revised: Neural networks can perform poorly on simple minorities.

Original: For a fixed target Q 0 , let’s say there exists two groups: G min and G maj ∈ G ( P, Q 0 ) such that P ( G min )
Revised: For a fixed target Q 0 , let’s say there exists two groups: G min and G maj ∈ G ( P, Q 0 ) such that P ( G min )

Original: ≪ P ( G maj ) .
Revised: ≪ P ( G maj ) .

Original: By Assumption 4.2, both G min and G maj are simple (realized in W ( γ ∗ ) ), and are thus separated by some simple feature.
Revised: By Assumption 4.2, both G min and G maj are simple (realized in W ( γ ∗ ) ), and are thus separated by some simple feature.

Original: The learner’s class H is usually a class of overparameterized neural networks.
Revised: The learner’s class H is usually a class of overparameterized neural networks.

Original: When trained with stochastic gradient descent (SGD), these are biased towards learning simple features that classify a majority of the data (Shah et al., 2020; Soudry et al., 2018).
Revised: When trained with stochastic gradient descent (SGD), these are biased towards learning simple features that classify a majority of the data (Shah et al., 2020; Soudry et al., 2018).

Original: Thus, if the simple feature separating
Revised: Thus, if the simple feature separating

Original: G min and G maj itself correlates with the label y on G maj , then neural networks would fit on this feature.
Revised: G min and G maj itself correlates with the label y on G maj , then neural networks would fit on this feature.

Original: This is precisely the case in the Waterbirds example, where the groups are defined by whether the simple feature background correlates with the label (Figure 1).
Revised: This is precisely the case in the Waterbirds example, where the groups are defined by whether the simple feature background correlates with the label (Figure 1).

Original: Thus our assumption on the nature of shift complements the nature of neural networks perform poorly on simple minorities.
Revised: Thus our assumption on the nature of shift complements the nature of neural networks perform poorly on simple minorities.

Original: The bitrate constraint helps identify simple unfair minorities in G ( P, Q 0 ) .
Revised: The bitrate constraint helps identify simple unfair minorities in G ( P, Q 0 ) .

Original: Any method that aims to be robust on Q 0 must up-weight data points from G min but without knowing its identity.
Revised: Any method that aims to be robust on Q 0 must up-weight data points from G min but without knowing its identity.

Original: Since the unconstrained adversary upsamples any group of data points with high loss and low probability, it cannot distinguish between a rare group that is realized by simple functions in W ( γ ∗ ) and a rare group of examples that share no feature in common or may even be mislabeled.
Revised: Since the unconstrained adversary upsamples any group of data points with high loss and low probability, it cannot distinguish between a rare group that is realized by simple functions in W ( γ ∗ ) and a rare group of examples that share no feature in common or may even be mislabeled.

Original: On the other hand, the group of mislabeled examples cannot be separated from the rest by functions in W ( γ ∗ ) .
Revised: On the other hand, the group of mislabeled examples cannot be separated from the rest by functions in W ( γ ∗ ) .

Original: Thus, a bitrate constraint adversary can only identify simple groups and upsamples those that incur high losses – possibly due to the simplicity bias of neural networks.
Revised: Thus, a bitrate constraint adversary can only identify simple groups and upsamples those that incur high losses – possibly due to the simplicity bias of neural networks.

Original: BR-DRO objective.
Revised: BR-DRO objective.

Original: According to Assumption 4.2, there cannot exist a target Q 0 such that minority
Revised: According to Assumption 4.2, there cannot exist a target Q 0 such that minority

Original: G min ∈ G ( P, Q 0 ) is not realized in bitrate constrained class W ( γ ∗ ) .
Revised: G min ∈ G ( P, Q 0 ) is not realized in bitrate constrained class W ( γ ∗ ) .

Original: Thus, by constraining our adversary to a class W ( γ ) (for some γ that is user defined), we can possibly evade issues emerging from optimizing for performance on mislabeled or hard examples, even if they were rare.
Revised: Thus, by constraining our adversary to a class W ( γ ) (for some γ that is user defined), we can possibly evade issues emerging from optimizing for performance on mislabeled or hard examples, even if they were rare.

Original: This gives us the objective in Equation 4 where the equalities hold from the linearity of ⟨· , ·⟩ and Definition 4.1.
Revised: This gives us the objective in Equation 4 where the equalities hold from the linearity of ⟨· , ·⟩ and Definition 4.1.

Original: [Equation]
Revised: [Equation]

Original: BR-DRO in practice.
Revised: BR-DRO in practice.

Original: We parameterize the learner θ h ∈ Θ h and adversary θ w ∈ Θ w as neural networks 2 .
Revised: We parameterize the learner θ h ∈ Θ h and adversary θ w ∈ Θ w as neural networks 2 .

Revised: In practice, we implement the adversary either as a one hidden layer variational information bottleneck (VIB)

Revised: Alemi et al.

Revised: (2016), where the Kullback-Leibler (KL) constraint on the latent variable z (output of VIB’s hidden layer) directly constrains the bitrate; or as an l 2 norm constrained linear layer.

Revised: The objective for the VIB ( l 2 ) version is obtained by setting β vib ̸ = 0 ( β l 2 ̸ = 0 ) in

Original: Therefore, the objective in Equation 5 is no longer convex-concave and can have multiple local equilibria or stationary points (Mangoubi & Vishnoi, 2021).
Revised: Equation 5 below. See Appendix A.2 for details. Note that the objective in Equation 5 is no longer convex-concave and can have multiple local equilibria or stationary points (Mangoubi & Vishnoi, 2021).

Original: The adversary’s objective also does not have a strong dual that can be solved through conic programs—a standard practice in DRO literature (Namkoong & Duchi, 2016).
Revised: The adversary’s objective also does not have a strong dual that can be solved through conic programs—a standard practice in DRO literature (Namkoong & Duchi, 2016).

Original: Thus, we provide an algorithm where both learner and adversary optimize BR-DRO iteratively through stochastic gradient ascent/descent (Algorithm 1).
Revised: Thus, we provide an algorithm where both learner and adversary optimize BR-DRO iteratively through stochastic gradient ascent/descent (Algorithm 1).

Original: The adversary’s action space W ( γ ) is constrained either with an information bottleneck penalty by setting β vib ̸ = 0 or l 2 norm penalty by setting β l 2 ̸ = 0 in Equation 5 below.

Original: While we can choose to constrain the adversary with both forms of constraints simultaneously we find that in practice picking only one of them for a given problem instance helps with tuning the degree of constraint.

Original: For more details on the architecture and other details see Appendix A.
Revised: For more details on the architecture and other details see Appendix A.

Original: [Equation]
Revised: [Equation]

Original: Training.
Revised: Training.

Original: For each example, the adversary takes as input: (i) the last layer output of the current learner’s feature network; and (ii) the input label.
Revised: For each example, the adversary takes as input: (i) the last layer output of the current learner’s feature network; and (ii) the input label.

Original: The adversary then outputs a weight (in [0 , 1] ).
Revised: The adversary then outputs a weight (in [0 , 1] ).

Original: The idea of applying the adversary directly on the learner’s features (instead of the original input) is based on recent literature (Rosenfeld et al., 2022; Kirichenko et al., 2022) that suggests re-training the prediction head is sufficient for robustness to shifts.
Revised: The idea of applying the adversary directly on the learner’s features (instead of the original input) is based on recent literature (Rosenfeld et al., 2022; Kirichenko et al., 2022) that suggests re-training the prediction head is sufficient for robustness to shifts.

Original: The adversary tries to maximize weights on examples with value ≥ η (hyperparameter) and minimize on others.
Revised: The adversary tries to maximize weights on examples with value ≥ η (hyperparameter) and minimize on others.

Original: For the learner, in addition to the example it takes as input the adversary assigned weight for that example from the previous round and uses it to reweigh its loss in a minibatch.
Revised: For the learner, in addition to the example it takes as input the adversary assigned weight for that example from the previous round and uses it to reweigh its loss in a minibatch.

Original: Both players are updated in a round (Algorithm 1).
Revised: Both players are updated in a round (Algorithm 1).

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: The main objective of our analysis of BR-DRO is to show how adding a bitrate constraint on the adversary can: (i) give us tighter statistical estimates of the worst risk; and (ii) control the pessimism (excess risk) of the learned solution.
Revised: The main objective of our analysis of BR-DRO is to show how adding a bitrate constraint on the adversary can: (i) give us tighter statistical estimates of the worst risk; and (ii) control the pessimism (excess risk) of the learned solution.

Original: First, we provide worst risk generalization guarantees using the
Revised: First, we provide worst risk generalization guarantees using the

Original: PAC-Bayes framework (Catoni, 2007), along with a result for kernel adversary.
Revised: PAC-Bayes framework (Catoni, 2007), along with a result for kernel adversary.

Original: Then, we provide convergence rates and pessimism guarantees for the solution found by our online solver for a specific instance of W ( γ )
Revised: Then, we provide convergence rates and pessimism guarantees for the solution found by our online solver for a specific instance of W ( γ )

Original: For both these, we analyze the constrained form of the conditional value at risk (CVaR) DRO objective (Levy et al., 2020) below.
Revised: For both these, we analyze the constrained form of the conditional value at risk (CVaR) DRO objective (Levy et al., 2020) below.

Original: Bitrate-Constrained CVaR DRO.
Revised: Bitrate-Constrained CVaR DRO.

Original: When the uncertainty set Q is defined by the set of all distributions Q that have bounded likelihood i.e., ∥ q/p ∥ ∞ ≤ 1 /α 0 , we recover the original CVaR DRO objective (Duchi & Namkoong, 2021).
Revised: When the uncertainty set Q is defined by the set of all distributions Q that have bounded likelihood i.e., ∥ q/p ∥ ∞ ≤ 1 /α 0 , we recover the original CVaR DRO objective (Duchi & Namkoong, 2021).

Original: The bitrate-constrained version of CVaR DRO is given in
Revised: The bitrate-constrained version of CVaR DRO is given in

Original: Equation 6 (see Appendix C for derivation).
Revised: Equation 6 (see Appendix C for derivation).

Original: Note that, slightly different from Section 3, we define
Revised: Note that, slightly different from Section 3, we define

Original: W as the set of all measurable functions w : X× Y (cid:55)→ [0 , 1] , since the other convex restrictions in
Revised: W as the set of all measurable functions w : X× Y (cid:55)→ [0 , 1] , since the other convex restrictions in

Original: Equation 1 are handled by dual variable η .
Revised: Equation 1 are handled by dual variable η .

Original: As in Section 4, W ( γ ) is derived from W using Definition 4.1.
Revised: As in Section 4, W ( γ ) is derived from W using Definition 4.1.

Original: In Equation 6, if we replace the bitrate-constrained class W ( γ ) with the unrestricted W then we recover the variational form of unconstrained CVaR DRO in Duchi et al.[Equation]
Revised: In Equation 6, if we replace the bitrate-constrained class W ( γ ) with the unrestricted W then we recover the variational form of unconstrained CVaR DRO in Duchi et al.

Original: η
Revised: L

Original: [Equation]
Revised: [Equation]

Original:  w ) where, R ( h, η, w [Equation] [Equation] l ( h
Revised: R ( h, η, w ) where, R ( h, η, w ) =

Original: [Equation]

Original: − η, w

Original: [Equation] P [Equation] η (6)
Revised: (1 /α 0 ) ⟨ l ( h ) − η, w ⟩ P + η (6)

Original: Worst risk estimation bounds for BR-DRO .
Revised: Worst risk estimation bounds for BR-DRO .

Original: Since we are only given a finite sampled dataset D ∼
Revised: Since we are only given a finite sampled dataset D ∼

Original: P n , we solve the objective in Equation 6 using the empirical distribution ˆ P n .
Revised: P n , we solve the objective in Equation 6 using the empirical distribution ˆ P n .

Original: We denote the plug-in estimates as ˆ h γD , ˆ η γD .
Revised: We denote the plug-in estimates as ˆ h γD , ˆ η γD .

Original: This incurs an estimation error for the true worst risk.
Revised: This incurs an estimation error for the true worst risk.

Original: But when we restrict our adversary to ∆( W , γ ) , for a fixed learner h we reduce the worst-case risk estimation error which scales with the bitrate KL ( · || π ) of the solution (deviation from prior π ).
Revised: But when we restrict our adversary to ∆( W , γ ) , for a fixed learner h we reduce the worst-case risk estimation error which scales with the bitrate KL ( · || π ) of the solution (deviation from prior π ).

Original: Expanding this argument to every learner in H , with high probability we also reduce the estimation error for the worst risk of ˆ h γD .
Revised: Expanding this argument to every learner in H , with high probability we also reduce the estimation error for the worst risk of ˆ h γD .

Original: Theorem 5.1 states this generalization guarantee more precisely.
Revised: Theorem 5.1 states this generalization guarantee more precisely.

Original: Theorem 5.1 (worst-case risk generalization) .
Revised: Theorem 5.1 (worst-case risk generalization) .

Original: With probability ≥ 1 − δ over D ∼ P n , the worst bitrate-constrained α 0 -CVaR risk for ˆ h γD can be upper bounded by the following oracle inequality:
Revised: With probability ≥ 1 − δ over D ∼ P n , the worst bitrate-constrained α 0 -CVaR risk for ˆ h γD can be upper bounded by the following oracle inequality:

Original: [Equation]
Revised: [Equation]

Original: when l ( · , · ) is [0 , M ] -bounded, L -Lipschitz and H is parameterized by convex set Θ ⊂ R d .
Revised: when l ( · , · ) is [0 , M ] -bounded, L -Lipschitz and H is parameterized by convex set Θ ⊂ R d .

Original: Informally, Theorem 5.1 tells us that bitrate-constraint γ gracefully controls the estimation error
Revised: Informally, Theorem 5.1 tells us that bitrate-constraint γ gracefully controls the estimation error

Original: O ( (cid:112) ( γ + C ( H )) /n )
Revised: O ( (cid:112) ( γ + C ( H )) /n )

Original: (where C ( H ) is a complexity measure) if we know that Assumption 4.2 is satisfied.
Revised: (where C ( H ) is a complexity measure) if we know that Assumption 4.2 is satisfied.

Original: While this only tells us that our estimator is consistent with O p (1 / √ n ) , the estimate may itself be converging to a degenerate predictor, i.e., L ∗ cvar ( γ ) may be very high.
Revised: While this only tells us that our estimator is consistent with O p (1 / √ n ) , the estimate may itself be converging to a degenerate predictor, i.e., L ∗ cvar ( γ ) may be very high.

Original: For example, if the adversary can cleanly separate mislabeled points even after the bitrate constraint, then presumably these noisy points with high losses would be the ones mainly contributing to the worst risk, and upweighting these points would result in a learner that has memorized noise.
Revised: For example, if the adversary can cleanly separate mislabeled points even after the bitrate constraint, then presumably these noisy points with high losses would be the ones mainly contributing to the worst risk, and upweighting these points would result in a learner that has memorized noise.

Original: Thus, it becomes equally important for us to analyze the excess risk (or the pessimism) for the learned solution.
Revised: Thus, it becomes equally important for us to analyze the excess risk (or the pessimism) for the learned solution.

Original: Since this is hard to study for any arbitrary bitrate-constrained class W ( γ ) , we shall do so for the specific class of reproducing kernel Hilbert space (RKHS) functions.
Revised: Since this is hard to study for any arbitrary bitrate-constrained class W ( γ ) , we shall do so for the specific class of reproducing kernel Hilbert space (RKHS) functions.

Original: Special case of bounded RKHS.
Revised: Special case of bounded RKHS.

Original: Let us assume there exists a prior Π such that W ( γ ) in Definition 4.1 is given by an RKHS induced by Mercer kernel k : X × X (cid:55)→ R , s.t.
Revised: Let us assume there exists a prior Π such that W ( γ ) in Definition 4.1 is given by an RKHS induced by Mercer kernel k : X × X (cid:55)→ R , s.t.

Original: the eigenvalues of the kernel operator decay polynomially, i.e., µ j < ∼ j − 2 /γ
Revised: the eigenvalues of the kernel operator decay polynomially, i.e., µ j < ∼ j − 2 /γ

Original: ( γ < 2) .
Revised: ( γ < 2) .

Original: Then, if we solve for ˆ h γD , ˆ η γD by doing kernel ridge regression over norm bounded ( ∥ f
Revised: Then, if we solve for ˆ h γD , ˆ η γD by doing kernel ridge regression over norm bounded ( ∥ f

Original: ∥ W ( γ ) ≤ B ≤ 1 ) smooth functions f then we can control: (i) the pessimism of the learned solution; and (ii) the generalization error (Theorem 5.2).
Revised: ∥ W ( γ ) ≤ B ≤ 1 ) smooth functions f then we can control: (i) the pessimism of the learned solution; and (ii) the generalization error (Theorem 5.2).

Original: Formally, we refer to pessimism for estimates ˆ h γD , ˆ η γD as excess risk defined as:
Revised: Formally, we refer to pessimism for estimates ˆ h γD , ˆ η γD as excess risk defined as:

Original: [Equation]
Revised: [Equation]

Original: Theorem 5.2 (bounded RKHS) .
Revised: Theorem 5.2 (bounded RKHS) .

Original: For l, H in Theorem 5.1, and for W ( γ ) described above ∃ γ 

Revised: For l, H in Theorem 5.1, and for W ( γ ) described above ∃ γ

Original: s.t.
Revised: s.t.

Original: for all sufficiently bitrate-constrained W ( γ ) i.e., γ ≤ γ 0 , w.h.p.
Revised: for all sufficiently bitrate-constrained W ( γ ) i.e., γ ≤ γ 0 , w.h.p.

Original: 1 − δ worst risk generalization error is O (cid:0) (1 /n ) (cid:0) log(1 /δ )
Revised: 1 − δ worst risk generalization error is O (cid:0) (1 /n ) (cid:0) log(1 /δ )

Original: + ( d + 1) log( nB − γ L γ/ 2 ) (cid:1)(cid:1)
Revised: + ( d + 1) log( nB − γ L γ/ 2 ) (cid:1)(cid:1)

Original: and the excess risk is O ( B ) for ˆ h γD , ˆ η γD above.
Revised: and the excess risk is O ( B ) for ˆ h γD , ˆ η γD above.

Original: Thus, in the setting described above we have shown how bitrate-constraints given indirectly by γ, R can control both the pessimism and statistical estimation errors.
Revised: Thus, in the setting described above we have shown how bitrate-constraints given indirectly by γ, R can control both the pessimism and statistical estimation errors.

Original: Here, we directly analyzed the estimates ˆ h γ D , ˆ η γ D but did not describe the specific algorithm used to solve the objective in Equationwith ˆ P n .
Revised: Here, we directly analyzed the estimates ˆ h γ D , ˆ η γ D but did not describe the specific algorithm used to solve the objective in Equationwith ˆ P n .

Original: Now, we look at an iterative online algorithm to solve the same objective and see how bitrate-constraints can also influence convergence rates in this setting.
Revised: Now, we look at an iterative online algorithm to solve the same objective and see how bitrate-constraints can also influence convergence rates in this setting.

Original: Convergence and excess risk analysis for an online solver.
Revised: Convergence and excess risk analysis for an online solver.

Original: In the following, we provide an algorithm to solve the objective in Equation 6 and analyze how bitrate-constraint impacts the solver and the solution.
Revised: In the following, we provide an algorithm to solve the objective in Equation 6 and analyze how bitrate-constraint impacts the solver and the solution.

Original: For convex losses, the min-max objective in Equation 6 has a unique solution and this matches the unique Nash equilibrium for the generic online algorithm (game) we describe (Lemma 5.3).
Revised: For convex losses, the min-max objective in Equation 6 has a unique solution and this matches the unique Nash equilibrium for the generic online algorithm (game) we describe (Lemma 5.3).

Original: The algorithm is as follows: Consider a two-player zero-sum game where the learner uses a no-regret strategy to first play h ∈ H , η ∈ R to minimize E w ∼ δ R
Revised: The algorithm is as follows: Consider a two-player zero-sum game where the learner uses a no-regret strategy to first play h ∈ H , η ∈ R to minimize E w ∼ δ R

Original: ( h, η, w ) .
Revised: ( h, η, w ) .

Original: Then, the adversary plays follow the regularized leader (FTRL) strategy to pick distribution δ ∈ ∆( W ( γ )) to maximize the same.
Revised: Then, the adversary plays follow the regularized leader (FTRL) strategy to pick distribution δ ∈ ∆( W ( γ )) to maximize the same.

Original: Our goal is to analyze the bitrate-constraint γ ’s effect on the above algorithm’s convergence rate and the pessimistic nature of the solution found.
Revised: Our goal is to analyze the bitrate-constraint γ ’s effect on the above algorithm’s convergence rate and the pessimistic nature of the solution found.

Original: For this, we need to first characterize the bitrate-constraint class W ( γ ) .
Revised: For this, we need to first characterize the bitrate-constraint class W ( γ ) .

Original: If we assume there exists a prior Π such that W ( γ ) is
Revised: If we assume there exists a prior Π such that W ( γ ) is

Original: Vapnik-Chervenokis (VC) class of dimension O ( γ ) , then in Theorem 5.4, we see that the iterates of our algorithm converge to the equilibrium (solution) in O ( (cid:112) γ log n/T ) steps.
Revised: Vapnik-Chervenokis (VC) class of dimension O ( γ ) , then in Theorem 5.4, we see that the iterates of our algorithm converge to the equilibrium (solution) in O ( (cid:112) γ log n/T ) steps.

Original: Clearly, the degree of bitrate constraint can significantly impact the convergence rate for a generic solver that solves the constrained DRO objective.
Revised: Clearly, the degree of bitrate constraint can significantly impact the convergence rate for a generic solver that solves the constrained DRO objective.

Original: Theorem 5.4 also bounds the excess risk (Equation 7) on ˆ P n .
Revised: Theorem 5.4 also bounds the excess risk (Equation 7) on ˆ P n .

Original: Lemma 5.3 (Nash equilibrium) .
Revised: Lemma 5.3 (Nash equilibrium) .

Original: For convex l ( h ) , l ( h ) ∈ [0 , M ] , the objective in Equation 6 has a unique solution which is also the Nash equilibrium of the game above when played over compact sets H × [0 , M ] , ∆( W , γ ) .
Revised: For convex l ( h ) , l ( h ) ∈ [0 , M ] , the objective in Equation 6 has a unique solution which is also the Nash equilibrium of the game above when played over compact sets H × [0 , M ] , ∆( W , γ ) .

Original: We denote this equilibrium as h ∗ D ( γ ) , η ∗ D ( γ ) , δ ∗ D ( γ ) .
Revised: We denote this equilibrium as h ∗ D ( γ ) , η ∗ D ( γ ) , δ ∗ D ( γ ) .

Original: Theorem 5.4.
Revised: Theorem 5.4.

Original: At time step t , if the learner plays ( h t , η t ) with no-regret and the adversary plays δ t with FTRL strategy that uses a negative entropy regularizer on δ then average iterates (¯ h T , ¯ η T , ¯ δ T ) =
Revised: At time step t , if the learner plays ( h t , η t ) with no-regret and the adversary plays δ t with FTRL strategy that uses a negative entropy regularizer on δ then average iterates (¯ h T , ¯ η T , ¯ δ T ) =

Original: (1 /T )
Revised: (1 /T )

Original: (cid:80) Tt =1 ( h t , η t , δ t ) converge to the equilibrium ( h ∗ D ( γ ) , η ∗ D ( γ ) , δ ∗ D ( γ )) at rate
Revised: (cid:80) Tt =1 ( h t , η t , δ t ) converge to the equilibrium ( h ∗ D ( γ ) , η ∗ D ( γ ) , δ ∗ D ( γ )) at rate

Original: O ( (cid:112) γ log n/T ) .
Revised: O ( (cid:112) γ log n/T ) .

Original: Further the excess risk defined above is O (( M/α 0 ) (cid:0) 1 − 1 n γ (cid:1) ) .
Revised: Further the excess risk defined above is O (( M/α 0 ) (cid:0) 1 − 1 n γ (cid:1) ) .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our experiments aim to evaluate the performance of BR-DRO and compare it with ERM and group shift robustness methods that do not require group annotations for training examples.
Revised: Our experiments aim to evaluate the performance of BR-DRO and compare it with ERM and group shift robustness methods that do not require group annotations for training examples.

Original: We conduct empirical analyses along the following axes: (i) worst group performance on datasets that exhibit known spurious correlations; (ii) robustness to random label noise in the training data; (iii) average performance on hybrid covariate shift datasets with unspecified groups; and (iv) accuracy in identifying minority groups.
Revised: We conduct empirical analyses along the following axes: (i) worst group performance on datasets that exhibit known spurious correlations; (ii) robustness to random label noise in the training data; (iii) average performance on hybrid covariate shift datasets with unspecified groups; and (iv) accuracy in identifying minority groups.

Original: See Appendix B for additional experiments and details.
Revised: See Appendix B for additional experiments and details.

Original: Baselines.
Revised: Baselines.

Original: Since our objective is to be robust to group shifts without group annotations on training examples, we explore baselines that either optimize for the worst minority group (CVaR DRO (Levy et al., 2020)) or use training losses to identify specific minority points (LfF (Nam et al., 2020),
Revised: Since our objective is to be robust to group shifts without group annotations on training examples, we explore baselines that either optimize for the worst minority group (CVaR DRO (Levy et al., 2020)) or use training losses to identify specific minority points (LfF (Nam et al., 2020),

Original: JTT (Liu et al., 2021)).
Revised: JTT (Liu et al., 2021)).

Original: Group DRO (Sagawa et al., 2019) is treated as an oracle.
Revised: Group DRO (Sagawa et al., 2019) is treated as an oracle.

Original: We also compare with the simple re-weighting baseline (RWY) proposed by Idrissi et al.
Revised: We also compare with the simple re-weighting baseline (RWY) proposed by Idrissi et al.

Original: Implementation details.
Revised: Implementation details.

Original: We train using Resnet-50 (He et al., 2016) for all methods and datasets except CivilComments, where we use BERT (Wolf et al., 2019).
Revised: We train using Resnet-50 (He et al., 2016) for all methods and datasets except CivilComments, where we use BERT (Wolf et al., 2019).

Original: For our VIB adversary, we use a 1 -hidden layer neural network encoder and decoder (one for each label).
Revised: For our VIB adversary, we use a 1 -hidden layer neural network encoder and decoder (one for each label).

Original: As mentioned in Section 4, the adversary takes as input the learner model’s features and the true label to generate weights.
Revised: As mentioned in Section 4, the adversary takes as input the learner model’s features and the true label to generate weights.

Original: All implementation and design choices for baselines were adopted directly from Liu et al.
Revised: All implementation and design choices for baselines were adopted directly from Liu et al.

Original: ; Idrissi et al.
Revised: ; Idrissi et al.

Original: We provide model selection methodology and other details in Appendix B.
Revised: We provide model selection methodology and other details in Appendix B.

Original: Datasets.
Revised: Datasets.

Original: For experiments in the known groups and label noise settings we use
Revised: For experiments in the known groups and label noise settings we use

Original: : (i) Waterbirds (Wah et al., 2011) (background is spurious), CelebA (Liu et al., 2015) (binary gender is spuriously correlated with label “blond”); and CivilComments (WILDS) (Borkan et al., 2019) where the task is to predict “toxic” texts and there are 16 predefined groups Koh et al.
Revised: : (i) Waterbirds (Wah et al., 2011) (background is spurious), CelebA (Liu et al., 2015) (binary gender is spuriously correlated with label “blond”); and CivilComments (WILDS) (Borkan et al., 2019) where the task is to predict “toxic” texts and there are 16 predefined groups Koh et al.

Original: We use FMoW and
Revised: We use FMoW and

Original: Camelyon17 (Koh et al., 2021) to test methods on datasets that do not have explicit group shifts.
Revised: Camelyon17 (Koh et al., 2021) to test methods on datasets that do not have explicit group shifts.

Original: In FMoW the task is to predict land use from satellite images where the training/test set comprises of data before/after 2013.
Revised: In FMoW the task is to predict land use from satellite images where the training/test set comprises of data before/after 2013.

Original: Test involves both subpopulation shifts over regions ( e.g., Africa, Asia) and domain generalization over time (year).
Revised: Test involves both subpopulation shifts over regions ( e.g., Africa, Asia) and domain generalization over time (year).

Original: Camelyon17 presents a domain generalization problem where the task is to detect tumor in tissue slides from different sets of hospitals in train and test sets.
Revised: Camelyon17 presents a domain generalization problem where the task is to detect tumor in tissue slides from different sets of hospitals in train and test sets.

Original: 6. I S BR-DRO ROBUST TO GROUP SHIFTS WITHOUT TRAINING DATA GROUP ANNOTATIONS ?
Revised: 6. I S BR-DRO ROBUST TO GROUP SHIFTS WITHOUT TRAINING DATA GROUP ANNOTATIONS ?

Original: Table 1 compares the average and worst group accuracy for BR-DRO with ERM and four group shift robustness baselines: JTT, LtF, SUBY, and CVaR DRO.
Revised: Table 1 compares the average and worst group accuracy for BR-DRO with ERM and four group shift robustness baselines: JTT, LtF, SUBY, and CVaR DRO.

Original: First, we see that unconstrained CVaR
Revised: First, we see that unconstrained CVaR

Original: DRO underperforms other heuristic algorithms.
Revised: DRO underperforms other heuristic algorithms.

Original: This matches the observation made by Liu et al.
Revised: This matches the observation made by Liu et al.

Original: Next
Revised: Next

Original: [Table]
Revised: [Table]

Original: we see that adding bitrate constraints on the adversary via a KL term or l 2 penalty significantly improves the performance of BR-DRO (VIB) or BR-DRO ( l 2 )
Revised: we see that adding bitrate constraints on the adversary via a KL term or l 2 penalty significantly improves the performance of BR-DRO (VIB) or BR-DRO ( l 2 )

Original: [Table]
Revised: [Table]

Original: which now matches the
Revised: which now matches the

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: best performing baseline (JTT).
Revised: best performing baseline (JTT).

Original: Thus, we see the less conservative nature of BR-DRO allows it to recover a large portion of the performance gap between Group DRO and CVaR DRO.
Revised: Thus, we see the less conservative nature of BR-DRO allows it to recover a large portion of the performance gap between Group DRO and CVaR DRO.

Original: Indirectly, this partially validates our Assumption 4.2, which states that the minority group is identified by a low bitrate adversary class.
Revised: Indirectly, this partially validates our Assumption 4.2, which states that the minority group is identified by a low bitrate adversary class.

Original: In Section 6.4 we discuss exactly what fraction of the minority group is identified
Revised: In Section 6.4 we discuss exactly what fraction of the minority group is identified

Original: [Figure]
Revised: [Figure]

Original: and the role played by the strength of bitrate-constraint
Revised: and the role played by the strength of bitrate-constraint

Original: [Figure]
Revised: [Figure]

Original: 6. B ITRATE DRO IS MORE ROBUST TO RANDOM LABEL NOISE
Revised: 6. B ITRATE DRO IS MORE ROBUST TO RANDOM LABEL NOISE

Original: Several methods for group robustness ( e.g., CVaR DRO, JTT) are based on the idea of up weighting points with high training losses.
Revised: Several methods for group robustness ( e.g., CVaR DRO, JTT) are based on the idea of up weighting points with high training losses.

Original: The goal is to obtain a learner with matching performance on every (small) fraction of points in the dataset.
Revised: The goal is to obtain a learner with matching performance on every (small) fraction of points in the dataset.

Original: However, when training data has mislabeled examples, such an approach will likely yield degenerate solutions.
Revised: However, when training data has mislabeled examples, such an approach will likely yield degenerate solutions.

Original: This is because the adversary directly upweights any example where the learner has high loss, including datapoints with incorrect labels.
Revised: This is because the adversary directly upweights any example where the learner has high loss, including datapoints with incorrect labels.

Original: Hence, even if the learner’s prediction matches the (unknown) true label, this formulation would force the learner to memorize incorrect labelings at the expense of learning the true underlying function.
Revised: Hence, even if the learner’s prediction matches the (unknown) true label, this formulation would force the learner to memorize incorrect labelings at the expense of learning the true underlying function.

Original: On the other hand, if the adversary is sufficiently bitrate constrained, it cannot upweight the arbitrary set of randomly mislabeled points, as this would require it to memorize those points.
Revised: On the other hand, if the adversary is sufficiently bitrate constrained, it cannot upweight the arbitrary set of randomly mislabeled points, as this would require it to memorize those points.

Original: Our Assumption 4. also dictates that the distribution shift would not upsample such high bitrate noisy examples.
Revised: Our Assumption 4. also dictates that the distribution shift would not upsample such high bitrate noisy examples.

Original: Thus, our constraint on the adversary ensures BR-DRO is robust to label noise in the training data and our assumption on the target distribution retains its robustness to test time distribution shifts.
Revised: Thus, our constraint on the adversary ensures BR-DRO is robust to label noise in the training data and our assumption on the target distribution retains its robustness to test time distribution shifts.

Original: In Figure 2b we highlight this failure mode of unconstrained up-weighting methods in contrast to
Revised: In Figure 2b we highlight this failure mode of unconstrained up-weighting methods in contrast to

Original: BR-DRO .
Revised: BR-DRO .

Original: We first induce random label noise (Carlini et al., 2019) of varying degrees into the Waterbirds and CelebA training sets.
Revised: We first induce random label noise (Carlini et al., 2019) of varying degrees into the Waterbirds and CelebA training sets.

Original: Then we run each method and compare worst group performance.
Revised: Then we run each method and compare worst group performance.

Original: In the absence of noise we see that the performance of JTT is comparable with BR-DRO , if not slightly better (Table 1).
Revised: In the absence of noise we see that the performance of JTT is comparable with BR-DRO , if not slightly better (Table 1).

Original: Thus, both BR-DRO and JTT perform reasonably well in identifying and upsampling the simple minority group in the absence of noise.
Revised: Thus, both BR-DRO and JTT perform reasonably well in identifying and upsampling the simple minority group in the absence of noise.

Original: In its presence, BR-DRO significantly outperforms JTT and other approaches on both Waterbirds and CelebA, as it only upsamples the minority examples misclassified by simple features, ignoring the noisy examples for the reasons above.
Revised: In its presence, BR-DRO significantly outperforms JTT and other approaches on both Waterbirds and CelebA, as it only upsamples the minority examples misclassified by simple features, ignoring the noisy examples for the reasons above.

Original: To further verify our claims, we set up a noisily labeled synthetic dataset (see Appendix B for details).
Revised: To further verify our claims, we set up a noisily labeled synthetic dataset (see Appendix B for details).

Original: In Figure 2a we plot training samples as well as the solutions learned by BR-DRO and and JTT on synthetic data.
Revised: In Figure 2a we plot training samples as well as the solutions learned by BR-DRO and and JTT on synthetic data.

Original: In Figure 1 (right) we also plot exactly which points are upweighted by BR-DRO and JTT.
Revised: In Figure 1 (right) we also plot exactly which points are upweighted by BR-DRO and JTT.

Original: Using both figures, we note that JTT mainly upweights the noisy points (in red) and memorizes them using x noise .
Revised: Using both figures, we note that JTT mainly upweights the noisy points (in red) and memorizes them using x noise .

Original: Without any weights on minority, it memorizes them as well and learns component along spurious feature.
Revised: Without any weights on minority, it memorizes them as well and learns component along spurious feature.

Original: On the contrary, when we restrict the adversary with
Revised: On the contrary, when we restrict the adversary with

Original: BR-DRO to be sparse ( l 1 penalty), it only upweights minority samples, since no sparse predictor can separate noisy points in the data.
Revised: BR-DRO to be sparse ( l 1 penalty), it only upweights minority samples, since no sparse predictor can separate noisy points in the data.

Original: Thus, the learner can no longer memorize the upweighted minority and we recover the robust predictor along core feature.
Revised: Thus, the learner can no longer memorize the upweighted minority and we recover the robust predictor along core feature.

Original: 6. H OW DOES BR-DRO PERFORM ON MORE GENERAL COVARIATE SHIFTS ?
Revised: 6. H OW DOES BR-DRO PERFORM ON MORE GENERAL COVARIATE SHIFTS ?

Original: In Figure 2 we report the average test accuracies for BR-DRO and baselines on the hybrid dataset
Revised: In Figure 2 we report the average test accuracies for BR-DRO and baselines on the hybrid dataset

Original: FMoW and domain generalization dataset Camelyon17.
Revised: FMoW and domain generalization dataset Camelyon17.

Original: In ( · ) we report the standard error of the mean accuracy across five runs.
Revised: In ( · ) we report the standard error of the mean accuracy across five runs.

Original: Given its hybrid nature, on FMoW we also report worst region accuracy.
Revised: Given its hybrid nature, on FMoW we also report worst region accuracy.

Original: First, we note that on these datasets group shift robustness baselines do not do better than ERM.
Revised: First, we note that on these datasets group shift robustness baselines do not do better than ERM.

Original: Some are either too pessimistic ( e.g., CVaR DRO), or require heavy assumptions
Revised: Some are either too pessimistic ( e.g., CVaR DRO), or require heavy assumptions

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: ( e.g., Group DRO) to be robust to domain generalization.
Revised: ( e.g., Group DRO) to be robust to domain generalization.

Original: This is also noted by Gulrajani & Lopez-Paz (2020).
Revised: This is also noted by Gulrajani & Lopez-Paz (2020).

Original: Next, we see that
Revised: Next, we see that

Original: BR-DRO ( l 2 version) does better than other group shift baselines on both both worst region and average datasets and matches ERM performance on Camelyon17.
Revised: BR-DRO ( l 2 version) does better than other group shift baselines on both both worst region and average datasets and matches ERM performance on Camelyon17.

Original: One explanation could be that even though these datasets test models on new domains, there maybe some latent groups defining these domains that are simple and form a part of latent subpopulation shift.
Revised: One explanation could be that even though these datasets test models on new domains, there maybe some latent groups defining these domains that are simple and form a part of latent subpopulation shift.

Original: Investigating this claim further is a promising line of future work.
Revised: Investigating this claim further is a promising line of future work.

Original: 6. W HAT FRACTION OF MINORITY IS RECOVERED BY B ITRATE -C ONSTRAINED DRO?
Revised: 6. W HAT FRACTION OF MINORITY IS RECOVERED BY B ITRATE -C ONSTRAINED DRO?

Original: We claim that our less pessimistic objective can more accurately recover (upsample) the true minority group if indeed the minority group is simple (see Assumption 4.2 for our definition of simple).
Revised: We claim that our less pessimistic objective can more accurately recover (upsample) the true minority group if indeed the minority group is simple (see Assumption 4.2 for our definition of simple).

Original: In this section, we aim to verify this claim.
Revised: In this section, we aim to verify this claim.

Original: If we treat examples in the top 10% (chosen for post hoc analysis) fraction of examples as our predicted minorities, we can check precision and recall of this decision on the Waterbirds and CelebA datasets.
Revised: If we treat examples in the top 10% (chosen for post hoc analysis) fraction of examples as our predicted minorities, we can check precision and recall of this decision on the Waterbirds and CelebA datasets.

Original: Figure 3 plots these metrics at each training epoch for BR-DRO (with varying β vib ), JTT and CVaR DRO.
Revised: Figure 3 plots these metrics at each training epoch for BR-DRO (with varying β vib ), JTT and CVaR DRO.

Original: Precision of the random baseline tells us the true fraction of minority examples in the data.
Revised: Precision of the random baseline tells us the true fraction of minority examples in the data.

Original: First we note that BR-DRO consistently performs much better on this metric than unconstrained CVaR DRO.
Revised: First we note that BR-DRO consistently performs much better on this metric than unconstrained CVaR DRO.

Original: In fact, as we reduce strength of β
Revised: In fact, as we reduce strength of β

Original: vib we recover precision/recall close to the latter.
Revised: vib we recover precision/recall close to the latter.

Original: This controlled experiment shows that the bitrate constraint is helpful (and very much needed) in practice to identify rare simple groups.
Revised: This controlled experiment shows that the bitrate constraint is helpful (and very much needed) in practice to identify rare simple groups.

Original: In Figurewe observe that asymptotically, the precision of BR-DRO is better than JTT on both datasets, while the recall is similar.
Revised: In Figurewe observe that asymptotically, the precision of BR-DRO is better than JTT on both datasets, while the recall is similar.

Original: Since importance weighting has little impact in later stages with exponential tail losses (Soudry et al., 2018; Byrd & Lipton, 2019), other losses ( e.g., polytail Wang et al.
Revised: Since importance weighting has little impact in later stages with exponential tail losses (Soudry et al., 2018; Byrd & Lipton, 2019), other losses ( e.g., polytail Wang et al.

Original: (2021)) may further improve the performance of BR-DRO as it gets better at identifying the minority classes when trained longer.
Revised: (2021)) may further improve the performance of BR-DRO as it gets better at identifying the minority classes when trained longer.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we proposed a method for making machine learning models more robust.
Revised: In this paper, we proposed a method for making machine learning models more robust.

Original: While prior methods optimize robustness on a per-example or per-group basis, our work focuses on features.
Revised: While prior methods optimize robustness on a per-example or per-group basis, our work focuses on features.

Original: In doing so, we avoid requiring group annotations on training samples, but also avoid the excessively conservative solutions that might arise from CVaR DRO with fully unconstrained adversaries.
Revised: In doing so, we avoid requiring group annotations on training samples, but also avoid the excessively conservative solutions that might arise from CVaR DRO with fully unconstrained adversaries.

Original: Our results show that our method avoids learning spurious features, is robust to noise in the training labels, and does better on other forms of covariate shifts compared to prior approaches.
Revised: Our results show that our method avoids learning spurious features, is robust to noise in the training labels, and does better on other forms of covariate shifts compared to prior approaches.

Original: Our theoretical analysis also highlights other provable benefits in some settings like reduced estimation error, lower excess risk and faster convergence rates for certain solvers.
Revised: Our theoretical analysis also highlights other provable benefits in some settings like reduced estimation error, lower excess risk and faster convergence rates for certain solvers.

Original: Limitations.
Revised: Limitations.

Original: While our method lifts the main limitation of Group DRO (access to training group annotations), it does so at the cost of increased complexity.
Revised: While our method lifts the main limitation of Group DRO (access to training group annotations), it does so at the cost of increased complexity.

Original: Further, to tune hyperparameters, like prior work we assume access to a some group annotations on validation set but also get decent performance (on some datasets) with only a balanced validation set (see Appendix B).
Revised: Further, to tune hyperparameters, like prior work we assume access to a some group annotations on validation set but also get decent performance (on some datasets) with only a balanced validation set (see Appendix B).

Original: Adapting group shift methods to more generic settings remains an important and open problem.
Revised: Adapting group shift methods to more generic settings remains an important and open problem.

Original: A PPENDIX O UTLINE
Revised: A PPENDIX O UTLINE

Original: A  BR-DRO ALGORITHM
Revised: A I MPLEMENTING BR-DRO IN PRACTICE A. BR-DRO ALGORITHM

Original: If the bitrate constraint is applied via the KL term in Equation 5, we implement the adversary as a variational information bottleneck (Alemi et al., 2016) (VIB), where the KL divergence with respect to a standard Gaussian prior controls the bitrate of the adversary’s feature set z ∼ p ( z | x ; θ w ) .
Revised: If the bitrate constraint is applied via the KL term in Equation 5, we implement the adversary as a variational information bottleneck (Alemi et al., 2016) (VIB), where the KL divergence with respect to a standard Gaussian prior controls the bitrate of the adversary’s feature set z ∼ p ( z | x ; θ w ) .

Original: Increasing β vib can be seen as enforcing lower bitrate features i.e., reducing γ in W ( γ ) (smaller value of KL ( δ || π ) in the primal formulation in Definition 4.1).
Revised: Increasing β vib can be seen as enforcing lower bitrate features i.e., reducing γ in W ( γ ) (smaller value of KL ( δ || π ) in the primal formulation in Definition 4.1).

Original: If the constraint is applied via the l 2 term we implement the adversary as a linear layer.
Revised: If the constraint is applied via the l 2 term we implement the adversary as a linear layer.

Original: In some cases ( e.g., Section 6.2) we use a sparsity constraint ( l 1 norm) on the linear adversary.
Revised: In some cases ( e.g., Section 6.2) we use a sparsity constraint ( l 1 norm) on the linear adversary.

Original: Algorithm
Revised: Algorithm

Original: Input: Adversary VIB penalty β vib; Step sizes ηl , ηw ; Dataset D = 
Revised: Input: Adversary VIB penalty β vib Step sizes η , η ; Dataset D = x

Original: ( x i , yi ) ni=
Revised: , y ) ni

Original: Initialize θ (1) h and θ (1) w for t = 1 , . .
Revised: Initialize θ (1) h and θ (1) w for t = 1 , . .

Original: , T do
Revised: , T do

Original: From D , sample x , y ∼ D /* Sample datapoint */ θ ( t +1)
Revised: From D , sample x , y ∼ D /* Sample datapoint */ θ ( t +1)

Original: h ← Π Θ h (cid:16) θ ( t ) h − η h ∇ θ h (cid:104) l
Revised: h ← Π Θ h (cid:16) θ ( t ) h − η h ∇ θ h (cid:104) l

Original: ( θ ( t ) h ( x ) , y ) · θ w ( x , y ) (cid:105)(cid:17) /
Revised: ( θ ( t ) h ( x ) , y ) · θ w ( x , y ) (cid:105)(cid:17) /

Original: * Update θ h */ θ ( t +1)
Revised: * Update θ h */ θ ( t +1)

Original: w ← Π Θ w (cid:16) θ ( t ) w + η w ∇ θ w L
Revised: w ← Π Θ w (cid:16) θ ( t ) w + η w ∇ θ w L

Original: adv
Revised: adv

Original: θ ( t ) w ; θ ( t ) h , β vib , β l 2 , η ) (cid:17) /
Revised: θ ( t ) w ; θ ( t ) h , β vib , β l 2 , η ) (cid:17) /

Original: * Update θ w */ end
Revised: * Update θ w */ end

Original: Output: ¯ θ h = 1 T (cid:80) Tt =1 θ ( t ) h , ¯ θ w = 1 T (cid:80)
Revised: Output: ¯ θ h = 1 T (cid:80) Tt =1 θ ( t ) h , ¯ θ w [Equation] T [Equation]

Revised: A. BR-DRO OBJECTIVE IN E QUATION

Original: Tt =1 θ  ( t ) w
Revised: When describing the actual BR-DRO objective in Equation 5, for brevity we used θ h to denote both the parameters of the learner and the learner itself (similarly for θ w ). Here, we describe in detail the parameterized version of the objective in Equation 4, and clarify the shallow nature of the adversary. Let us denote the learner as h ( · ; θ h ) : X (cid:55)→ Y , and the class of learners H :=

Revised: { h ( · ; θ h ) : θ h ∈ Θ h } .

Revised: A learner is composed of two parts: i) a feature extractor f : X (cid:55)→ R p that maps the input into a p -dimensional feature vector and; ii) a classifier c : R p (cid:55)→ Y that maps the features into the predicted label.

Revised: Similarly, in the bitrate-constrained class W ( γ ) each adversary is parameterized as w ( · ; θ w ) : R p ×Y (cid:55)→ [0 , 1] 3 where with some overloading of notation we use w ( x , y ) to denote w ( f ( x ) , y ) i.e., the adversary only operates on the features output by the feature extractor f ( x ) and the given label y. Since the features are extracted by the deep neural network, the adversary is implemented as a shallow network.

Revised: We shall now describe each version of BR-DRO separately.

Revised: BR-DRO (VIB) :

Revised: For each label y, w ( f ( x ) , y ) is a one hidden layer neural network with ReLU activations (1-layer VIB).

Revised: The first layer takes as input feature vector f ( x ) and outputs a 2 d dimensional vector u .

Revised: The d dimensional latent encoding z ( x ) (dependence on x is made explicit) is sampled from multivariate Normal z ( x ) ∼ N ( u [ :d ] , I d · u [ d: ]) where u [ :d ] is the mean and I d · u [ d: ] is a diagonal covariance matrix.

Revised: Following Alemi et al.

Revised: (2016), a KL constraint is applied on latent variable z ( x ) , with strength given by scalar β vib .

Revised: If we assume the generative model for x as one defined by latent factors of variation ( e.g., orientation, background), then presumably the group identity is a function of these factors.

Revised: If we believe that the KL constraint on z ( x ) helps recover some of these factors, then learning a linear function over it would amount to learning a simple group function.

Revised: Finally, an output layer with a sigmoid activation maps z ( x ) into a weight between [0 , 1] .

Revised: BR-DRO ( l 2 ) :

Revised: For each label y, w ( f ( x ) , y ) is a linear layer.

Revised: It takes as input feature vector f ( x ) and maps it to a scalar which when passed through a sigmoid yields a weight between [0 , 1] .

Revised: The lconstraint over the linear layer parameters is controlled by scalar β l 2 .

Revised: We can incorporate the above parameterizations for VIB and l 2 versions of BR-DRO into the

Revised: BR-DRO objective in Equation 4 through Equation 8 and Equation 9 respectively to yield the final objective in Equation 5.

Revised: Note that as we mention in Section 4, we can switch between the two versions of BR-DRO by setting β vib = 0 (for l 2 ) or β l 2 = 0 (for VIB).

Revised: While we can choose to constrain the adversary with both forms of constraints simultaneously we find that in practice picking only one of them for a given problem instance helps with tuning the degree of constraint.

Revised: [Equation]

Revised: A. C ONNECTING BITRATE - CONSTRAINED W ( γ ) TO SIMPLE GROUPS ( A PRACTICAL EXAMPLE ).

Revised: First, let us recall the definition of a simple group in Assumption 4.2.

Revised: We defined a group G to be simple, if the indicator function 1 (( x , y ) ∈ G ) identifying said group is containing in the bitrateconstrained class of functions W ( γ ) .

Revised: Next, we shall see what this means in terms of a specific prior π and the constraint KL ( · || π ) ≤ γ that defines the class W ( γ ) in Definition 4.1.

Revised: Let us assume that the adversary ( θ w ) is parameterized as a linear classifier in R d where each parameter θ w corresponds to a re-weighting function in W P,κ (Equation 1), and the prior π is designed to have a higher likelihood over low norm solutions.

Revised: Specifically, π takes the form:

Revised: [Equation]

Revised: Now consider the class of densities:

Revised: [Equation]

Revised: Here, it is easy to verify that:

Revised: [Equation]

Revised: Note, that while Definition 4.1 concerns with any δ ∈ ∆( W ) , we restrict ourselves to the subset ∆(Θ w ) ⊂ ∆( W ) for the sake of mathematical convenience, and find that even this set is rich enough to easily violate the bitrate-constraint as we shall see next.

Revised: Applying Definition 4.1 on the set ∆(Θ w ) we get:

Revised: [Equation]

Revised: Further if we compute KL ( δ θ w || π ) , we find that KL ( δ θ w || π ) =

Revised: ∥ θ w ∥ 22 + C (for some constant

Revised: C).

Revised: Thus, the bitrate constraint KL ( δ θ w || π ) ≤ γ directly transfers into an l 2 norm constraint on the mean parameter θ w , and W ( γ ) is simply the set of parameters θ w ∈ Θ w that have their lnorms bounded above by some constant γ − C .

Revised: The objective for the l 2 version of our adversary in

Revised: Equation 5 reflects this form.

Original: B A DDITIONAL EMPIRICAL RESULTS AND OTHER EXPERIMENT DETAILS B. H YPER - PARAMETER TUNING METHODOLOGY
Revised: B A DDITIONAL EMPIRICAL RESULTS AND OTHER EXPERIMENT DETAILS B. H YPER - PARAMETER TUNING METHODOLOGY

Original: There are two ways in which we tune hyperparameters on datasets with known groups (CelebA,
Revised: There are two ways in which we tune hyperparameters on datasets with known groups (CelebA,

Original: Waterbirds, CivilComments): (i) on average validation performance; (ii) worst group accuracy.
Revised: Waterbirds, CivilComments): (i) on average validation performance; (ii) worst group accuracy.

Original: The former does not use group annotations while the latter does.
Revised: The former does not use group annotations while the latter does.

Original: Similar to prior works Liu et al.
Revised: Similar to prior works Liu et al.

Original: Idrissi et al.
Revised: Idrissi et al.

Original: we note that using group annotations (on a small validation set) does improve performance.
Revised: we note that using group annotations (on a small validation set) does improve performance.

Original: In Table 3 we report our study which varies the the fraction p of group labels that are available at test time.
Revised: In Table 3 we report our study which varies the the fraction p of group labels that are available at test time.

Original: For each setting of p , we do model selection by taking weighted (by p ) mean over two entities (i) average validation on all samples, (ii) worst group validation on a fraction p of minority samples.
Revised: For each setting of p , we do model selection by taking weighted (by p ) mean over two entities (i) average validation on all samples, (ii) worst group validation on a fraction p of minority samples.

Original: In the case where p = 0 , we only use average validation.
Revised: In the case where p = 0 , we only use average validation.

Original: We report our results on CelebA and Waterbirds dataset.
Revised: We report our results on CelebA and Waterbirds dataset.

Original: For the two WILDS datasets we tune hyper-parameters on OOD
Revised: For the two WILDS datasets we tune hyper-parameters on OOD

Original: Validation set.
Revised: Validation set.

Revised: [Table]

Revised: of group annotations in the training data

Revised: affect performance.

Original:  B. S YNTHETIC DATASET DETAILS
Revised: For each dataset and method, we tune its hyper-parameters on the average validation and worst group (only on the small fraction p that is available). We see that while all methods consistently improve as we increase group annotations and tune for worst group accuracy on the annotated samples, BR-DRO does do better that prior works when tuned on just average validation ( p = 0 . ). At the same time, we note that this still does not match the performance of BR-DRO when tuned on worst group validation (seen in Table 1). B. S YNTHETIC DATASET DETAILS

Original: We follow the explicit-memorization setup in Sagawa et al.
Revised: We follow the explicit-memorization setup in Sagawa et al.

Original: (2020) which we summarize here briefly.
Revised: (2020) which we summarize here briefly.

Original: Let input x =
Revised: Let input x =

Original: [ x core , x spu , x noise ] where x core | y ∼ N ( y, σ 2 core ) , x spu | a ∼ N ( a, σ 2 spu ) and x noise ∼
Revised: [ x core , x spu , x noise ] where x core | y ∼ N ( y, σ 2 core ) , x spu | a ∼ N ( a, σ 2 spu ) and x noise ∼

Original: N ( 0 , ( σ 2 noise I d ) / d ) .
Revised: N ( 0 , ( σ 2 noise I d ) / d ) .

Original: Here a ∈ {− 1 , 1 } refers to a spurious attribute, and label is y ∈ {− 1 , 1 } , We set a = y with probability P ( maj )
Revised: Here a ∈ {− 1 , 1 } refers to a spurious attribute, and label is y ∈ {− 1 , 1 } , We set a = y with probability P ( maj )

Original: = 1 − P ( min ) .
Revised: = 1 − P ( min ) .

Original: The level of correlation between a and y is controlled by P ( maj ) .
Revised: The level of correlation between a and y is controlled by P ( maj ) .

Original: Additionally, we flip true label with probability P ( noise ) .[Table]
Revised: Additionally, we flip true label with probability P ( noise ) .

Original: B. D EGREE OF CONSTRAINT
Revised: B. D EGREE OF CONSTRAINT

Original: In Figure 4 we see how worst group performance varies on Waterbirds and CelebA as a function of increasing constraint.
Revised: In Figure 4 we see how worst group performance varies on Waterbirds and CelebA as a function of increasing constraint.

Original: We also plot average performance on the Camelyon dataset.
Revised: We also plot average performance on the Camelyon dataset.

Original: We mainly note that for either of the constraint implementations, only when we significantly increase the capacity do we actually see the performance of BR-DRO improve.
Revised: We mainly note that for either of the constraint implementations, only when we significantly increase the capacity do we actually see the performance of BR-DRO improve.

Original: The effect is more prominent on groups shift datasets with simple groups (Waterbirds, CelebA).
Revised: The effect is more prominent on groups shift datasets with simple groups (Waterbirds, CelebA).

Original: Under less restrictive capacity constraints we note that its performance is similar to CVaR DRO (see Figure 3).
Revised: Under less restrictive capacity constraints we note that its performance is similar to CVaR DRO (see Figure 3).

Original: This is expected since CVaR
Revised: This is expected since CVaR

Original: DRO is the completely unconstrained version of our objective.
Revised: DRO is the completely unconstrained version of our objective.

Original: [Figure]

Original: B. H YPER - PARAMETER DETAILS .
Revised: B. H YPER - PARAMETER DETAILS .

Original: For all hyper-parameters of prior methods we use the ones state in their respective prior works.
Revised: For all hyper-parameters of prior methods we use the ones state in their respective prior works.

Original: The implementation Group DRO, JTT, CVaR DRO is borrowed from the implementation made public by authors of Liu et al.
Revised: The implementation Group DRO, JTT, CVaR DRO is borrowed from the implementation made public by authors of Liu et al.

Original: For datasets Waterbirds, CelebA and CivilComments we choose the hyper-parameters (whenever applicable) learning rate, batch size, weight decay on learner, optimizer, early stopping criterion, learning rate schedules used by Liu et al.
Revised: For datasets Waterbirds, CelebA and CivilComments we choose the hyper-parameters (whenever applicable) learning rate, batch size, weight decay on learner, optimizer, early stopping criterion, learning rate schedules used by Liu et al.

Original: (2021) for their implementation of CVaR DRO method.
Revised: (2021) for their implementation of CVaR DRO method.

Original: For datasets FMoW and Camelyon17 we choose values for these  hyper-parameters to be the ones used by Koh et al.
Revised: For datasets FMoW and Camelyon17 we choose values for these [Figure] hyper-parameters to be the ones used by Koh et al.

Original: (2021) for the ERM baseline.
Revised: (2021) for the ERM baseline.

Original: Details on BR-DRO specific hyper-parameters that we tuned are in Table 4.
Revised: Details on BR-DRO specific hyper-parameters that we tuned are in Table 4.

Original: Also, note that we release our implementation with this submission. 
Revised: Also, note that we release our implementation with this submission. Hyper-parameter Waterbirds CelebA

Revised: CivilComments

Revised: FMoW

Original:  [Table]
Revised: Camelyon [Table]

Original: C O MITTED P ROOFS
Revised: C O MITTED P ROOFS

Original: First we shall state some a couple of technical lemmas that we shall refer to at multiple points.
Revised: First we shall state some a couple of technical lemmas that we shall refer to at multiple points.

Original: Then, we prove our theoretical claims in our analysis Section 5, in the order in which they appear.
Revised: Then, we prove our theoretical claims in our analysis Section 5, in the order in which they appear.

Original: Before we get into those we provide proof for our Corollary 3.2 and the derivation of Bitrate-Constrained
Revised: Before we get into those we provide proof for our Corollary 3.2 and the derivation of Bitrate-Constrained

Original: CVaR DRO in Equation 6.
Revised: CVaR DRO in Equation 6.

Original: Lemma C.1 (Hoeffding bound Wainwright (2019)) .
Revised: Lemma C.1 (Hoeffding bound Wainwright (2019)) .

Original: Let X 1 , . . .
Revised: Let X 1 , . . .

Original: , X n be a set of µ i centered independent sub-Gaussians, each with parameter σ i .
Revised: , X n be a set of µ i centered independent sub-Gaussians, each with parameter σ i .

Original: Then for all t ≥ 0 , we have
Revised: Then for all t ≥ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma C.2 (Lipschitz functions of Gaussians Wainwright (2019)) .
Revised: Lemma C.2 (Lipschitz functions of Gaussians Wainwright (2019)) .

Original: Let X 1 , . . .
Revised: Let X 1 , . . .

Original: , X n be a vector of iid Gaussian variables and f : R n (cid:55)→ R be L -Lipschitz with respect to the Euclidean norm.
Revised: , X n be a vector of iid Gaussian variables and f : R n (cid:55)→ R be L -Lipschitz with respect to the Euclidean norm.

Original: Then the random variable f ( X ) − E [ f ( X )] is sub-Gaussian with parameter at most L , thus: 
Revised: Then the random variable f ( X ) − E [ f ( X )] is sub-Gaussian with parameter at most L , thus: [Equation]

Original: [Equation]
Revised: [Equation]

Original: C. P ROOF OF C OROLLARY 3. 
Revised: C. P ROOF OF C OROLLARY 3.

Original: Let us recall the definition of a well defined group structure.
Revised: Let us recall the definition of a well defined group structure.

Original: For a pair of measures Q ≪ P we say
Revised: For a pair of measures Q ≪ P we say

Original: G ( P, Q ) is well defined if given there exists a set of disjoint measurable sets G P,Q = { G k } Kk =1 such that G k ∈ Σ , Q ( G k ) > 0 , Q ( G ( P, Q ))
Revised: G ( P, Q ) is well defined if given there exists a set of disjoint measurable sets G P,Q = { G k } Kk =1 such that G k ∈ Σ , Q ( G k ) > 0 , Q ( G ( P, Q ))

Original: = 

Revised: =

Original: and we have:
Revised: and we have:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now by definition K is finite.
Revised: Now by definition K is finite.

Original: Thus if there exists two well defined group structures G 1 ( P, Q ) and
Revised: Thus if there exists two well defined group structures G 1 ( P, Q ) and

Original: G 2 ( P, Q ) for the same pair P, Q then it must be the case that K = G 1 ( P, Q ) =
Revised: G 2 ( P, Q ) for the same pair P, Q then it must be the case that K = G 1 ( P, Q ) =

Original: G 2 ( P, Q ) .
Revised: G 2 ( P, Q ) .

Original: Then, there must exist G ∈ G 1 ( P, Q ) such that Q ( G ) > 0 and G ′ , G ′′ ∈ G 2 ( P, Q ) where
Revised: Then, there must exist G ∈ G 1 ( P, Q ) such that Q ( G ) > 0 and G ′ , G ′′ ∈ G 2 ( P, Q ) where

Original: Q ( G ′ ) , Q ( G ′′ ) > 0 and Q ( G ∩ G ′ ) , Q ( G ∩ G ′′ ) > 0 .
Revised: Q ( G ′ ) , Q ( G ′′ ) > 0 and Q ( G ∩ G ′ ) , Q ( G ∩ G ′′ ) > 0 .

Original: Note that since G, G ′ , G ′′ ∈ Σ that is closed under countable unions, we have that G ∩ G ′ and
Revised: Note that since G, G ′ , G ′′ ∈ Σ that is closed under countable unions, we have that G ∩ G ′ and

Original: G ∩ G ′′ are two sets where q ( x , y ) > 0 ∀ ( x , y ) ∈ G ∩ G ′ , G ∩ G ′′ .
Revised: G ∩ G ′′ are two sets where q ( x , y ) > 0 ∀ ( x , y ) ∈ G ∩ G ′ , G ∩ G ′′ .

Original: Let ( x 1 , y 1 ) ∈
Revised: Let ( x 1 , y 1 ) ∈

Original: ( G ∩ G ′ )
Revised: ( G ∩ G ′ )

Original: and ( x 2 , y 2 ) ∈ ( G ∩ G ′′ ) .
Revised: and ( x 2 , y 2 ) ∈ ( G ∩ G ′′ ) .

Original: From definition we know that q ( x 2 , y 2 ) , q ( x 1 , y 1 ) > 

Revised: From definition we know that q ( x 2 , y 2 ) , q ( x 1 , y 1 ) >

Original: and .
Revised: and .

Original: Since both ( x 1 , y 1 ) and ( x 2 , y 2 ) are in G we have that: 
Revised: Since both ( x 1 , y 1 ) and ( x 2 , y 2 ) are in G we have that: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Thus, we can conclude that Q ( G ′ )
Revised: Thus, we can conclude that Q ( G ′ )

Original: P ( G ′ )
Revised: P ( G ′ )

Original: =
Revised: =

Original: Q ( G ′′ ) P ( G ′′ ) .
Revised: Q ( G ′′ ) P ( G ′′ ) .

Original: This implies that G ′ ∪ G ′′ also satisfies the following that Q ( G ′ ∪ G ′′ ) > 0 and q ( x , y | G ′ ∪ G ′′ )
Revised: This implies that G ′ ∪ G ′′ also satisfies the following that Q ( G ′ ∪ G ′′ ) > 0 and q ( x , y | G ′ ∪ G ′′ )

Original: =
Revised: =

Original: p ( x , y | G ′ ∪ G ′′ ) .
Revised: p ( x , y | G ′ ∪ G ′′ ) .

Original: Thus, we can construct a new G 3 ( P, Q ) =
Revised: Thus, we can construct a new G 3 ( P, Q ) =

Original: { G ∈ G 2 ( P, Q ) : G / ∈ { G ′ , G ′′ }} ∪ { G ′ ∪ G ′′ } .
Revised: { G ∈ G 2 ( P, Q ) : G / ∈ { G ′ , G ′′ }} ∪ { G ′ ∪ G ′′ } .

Original: Clearly,
Revised: Clearly,

Original: G 3 ( P, Q ) satisfies all group structure properties and is smaller than G 2 ( P, Q ) .
Revised: G 3 ( P, Q ) satisfies all group structure properties and is smaller than G 2 ( P, Q ) .

Original: Thus, we arrive at a contradiction which proves the claim that G ( P, Q ) is indeed unique whenever well defined.
Revised: Thus, we arrive at a contradiction which proves the claim that G ( P, Q ) is indeed unique whenever well defined.

Original: C. D ERIVATION OF B ITRATE -C ONSTRAINED CV A R DRO IN E QUATION
Revised: C. D ERIVATION OF B ITRATE -C ONSTRAINED CV A R DRO IN E QUATION

Original: Recall that we define W as the set of all measurable functions w : X × Y (cid:55)→ [0 , 1] , since the other convex restrictions in Equation 1 are handled by dual variable η .
Revised: Recall that we define W as the set of all measurable functions w : X × Y (cid:55)→ [0 , 1] , since the other convex restrictions in Equation 1 are handled by dual variable η .

Original: As in Section 4, W ( γ ) is derived from the new W using Definition 4.1.
Revised: As in Section 4, W ( γ ) is derived from the new W using Definition 4.1.

Original: With that let us first state the CVaR objective (Levy et al., 2020). 
Revised: With that let us first state the CVaR objective (Levy et al., 2020). [Equation]

Original: [Equation]
Revised: [Equation]

Original: The objective in q is linear with convex constraints, and has a strong dual (see Duchi et al.
Revised: The objective in q is linear with convex constraints, and has a strong dual (see Duchi et al.

Original: Boyd et al.
Revised: Boyd et al.

Original: (2004) for the derivation) which is given by:
Revised: (2004) for the derivation) which is given by:

Original: [Equation]
Revised: [Equation]

Original:  The last equality is true since the set 1 ( l ( h ) − η ≥ 0) is measurable under P (based on our setup in Section 3).
Revised: [Equation] The last equality is true since the set 1 ( l ( h ) − η ≥ 0) is measurable under P (based on our setup in Section 3).

Original: Note that for any h , the objective 1 α 0 ⟨ ( l ( h ) − η ) , w ⟩ P + η is linear in w , and
Revised: Note that for any h , the objective 1 α 0 ⟨ ( l ( h ) − η ) , w ⟩ P + η is linear in w , and

Original: η .
Revised: η .

Original: If we further assume the loss l ( h ) to be the l 0 − 1 loss, it is bounded, and thus the optimization over η can be restricted to a compact set.
Revised: If we further assume the loss l ( h ) to be the l 0 − 1 loss, it is bounded, and thus the optimization over η can be restricted to a compact set.

Original: Next, W is also a compact set of functions since we restrict our solvers to measurable functions that take values bounded in [0 , 1] .
Revised: Next, W is also a compact set of functions since we restrict our solvers to measurable functions that take values bounded in [0 , 1] .

Original: [Equation]
Revised: [Equation]

Original: The above objective is precisely the Bitrate-Constrained CVaR DRO objective we have in Equation 6.
Revised: The above objective is precisely the Bitrate-Constrained CVaR DRO objective we have in Equation 6.

Original: Later in the Appendix we shall need an equivalent form of the objective which we shall derive below.
Revised: Later in the Appendix we shall need an equivalent form of the objective which we shall derive below.

Original: We can now invoke the Weierstrass’ theorem in Boyd et al.
Revised: We can now invoke the Weierstrass’ theorem in Boyd et al.

Original: (2004) to give us the following:
Revised: (2004) to give us the following:

Original: [Equation]
Revised: [Equation]

Original: Now, the final objective inf h ∈H L cvar ( h, P ) is given by:
Revised: Now, the final objective inf h ∈H L cvar ( h, P ) is given by:

Original: [Equation]
Revised: [Equation]

Original: In the above equation we can now replace the unconstrained class W with our bitrate-constrained class W ( γ ) to get the following:
Revised: In the above equation we can now replace the unconstrained class W with our bitrate-constrained class W ( γ ) to get the following:

Original: [Equation]
Revised: [Equation]

Original: C. P ROOF OF T HEOREM 5. 
Revised: C. P ROOF OF T HEOREM 5.

Original: For convenience we shall first restate the Theorem here.
Revised: For convenience we shall first restate the Theorem here.

Original: Theorem C.3 ([restated) .
Revised: Theorem C.3 ([restated) .

Original: worst-case risk generalization] With probability ≥ 1 − δ over sample
Revised: worst-case risk generalization] With probability ≥ 1 − δ over sample

Original: D ∼ P n , the worst risk for ˆ h γD can be upper bounded by the following oracle inequality:
Revised: D ∼ P n , the worst risk for ˆ h γD can be upper bounded by the following oracle inequality:

Original: [Equation]
Revised: [Equation]

Original: when l ( · , · ) is [0 , M ] -bounded, L -Lipschitz and H is parameterized by convex set Θ ⊂ R d .
Revised: when l ( · , · ) is [0 , M ] -bounded, L -Lipschitz and H is parameterized by convex set Θ ⊂ R d .

Original: The overview of the proof can be split into two parts:
Revised: The overview of the proof can be split into two parts:

Original: • For each learner, first obtain the oracle PAC-Bayes (McAllester, 1998) worst risk generalization guarantee over the adversary’s action space ∆( W , γ ) .
Revised: • For each learner, first obtain the oracle PAC-Bayes (McAllester, 1998) worst risk generalization guarantee over the adversary’s action space ∆( W , γ ) .

Original: • Then, apply uniform convergence bounds using a union bound over a covering of the class H to get the final result.
Revised: • Then, apply uniform convergence bounds using a union bound over a covering of the class H to get the final result.

Original: Intuition: The only tricky part lies in the fact that oracle PAC-Bayes inequality would not give us arbitrary control over the generalization error for each learner, which we would typically get in
Revised: Intuition: The only tricky part lies in the fact that oracle PAC-Bayes inequality would not give us arbitrary control over the generalization error for each learner, which we would typically get in

Original: Hoeffding type bounds.
Revised: Hoeffding type bounds.

Original: Hence, we need to ensure that the the worst risk generalization rate decays faster than how the size of the covering would increase for a ball of radius defined by the worst generalization error.
Revised: Hence, we need to ensure that the the worst risk generalization rate decays faster than how the size of the covering would increase for a ball of radius defined by the worst generalization error.

Original: Now, we shall invoke the following PAC-Bayes generalization guarantee stated (Lemma C.4) since
Revised: Now, we shall invoke the following PAC-Bayes generalization guarantee stated (Lemma C.4) since

Original: R ( h, η, w ) ∈ [0 , M/α 0 ] .
Revised: R ( h, η, w ) ∈ [0 , M/α 0 ] .

Original: Lemma C.4 (PAC-Bayes (Catoni, 2007; McAllester, 1998)) .
Revised: Lemma C.4 (PAC-Bayes (Catoni, 2007; McAllester, 1998)) .

Original: With probability ≥ 1 − δ over choice of dataset D of size n the following inequality is satisfied
Revised: With probability ≥ 1 − δ over choice of dataset D of size n the following inequality is satisfied

Original: [Equation]
Revised: [Equation]

Original: A direct application of this gives us that with probability at least 1 − ω : .
Revised: A direct application of this gives us that with probability at least 1 − ω : .

Original: [Equation]
Revised: [Equation]

Original: 2 n −

Original: Let ˆ R D
Revised: Let ˆ R D

Original: ( h, η, w )
Revised: ( h, η, w )

Original: = 1 α 

Revised: = 1 α

Original: ⟨ l ( h ) − η, w ⟩ ˆ P n + η Since the above inequality holds for any data dependent δ :.
Revised: ⟨ l ( h ) − η, w ⟩ ˆ P n + η Since the above inequality holds for any data dependent δ :.

Original: [Equation]
Revised: [Equation]

Original: Further, we make use of the fact KL ( δ || π ) ≤ γ .
Revised: Further, we make use of the fact KL ( δ || π ) ≤ γ .

Original: [Equation]
Revised: [Equation]

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: To actually apply this uniformly over h, η , we would first need two sided concentration which we derive below as follows:
Revised: To actually apply this uniformly over h, η , we would first need two sided concentration which we derive below as follows:

Original: Let a i
Revised: Let a i

Original: =
Revised: =

Original: ˆ R D
Revised: ˆ R D

Original: ( h, η, δ )
Revised: ( h, η, δ )

Original: − R
Revised: − R

Original: ( h, η, δ ) , Since R ( h, η, δ ) ≤ M/α 0 , we can apply Hoeffding bound with t
Revised: ( h, η, δ ) , Since R ( h, η, δ ) ≤ M/α 0 , we can apply Hoeffding bound with t

Original: = λ/n in Lemma C.1 on a i to get:
Revised: = λ/n in Lemma C.1 on a i to get:

Original: [Equation]
Revised: [Equation]

Original: Applying Fubini’s Theorem, followed by the Donsker Varadhan variational formulation we get:
Revised: Applying Fubini’s Theorem, followed by the Donsker Varadhan variational formulation we get:

Original: [Equation]
Revised: [Equation]

Original: The Chernoff bound finally gives us with probability ≥ 1 − ω :
Revised: The Chernoff bound finally gives us with probability ≥ 1 − ω :

Original: [Equation]
Revised: [Equation]

Original: Using the reverse form of the empirical PAC Bayes inequality, we can do a derivation similar to the one following the PAC-Bayes bound in Lemma C.4 to get for any fixed η ∈ [0 , M ] , h ∈ H we get: (cid:12)(cid:12)(cid:12)
Revised: Using the reverse form of the empirical PAC Bayes inequality, we can do a derivation similar to the one following the PAC-Bayes bound in Lemma C.4 to get for any fixed η ∈ [0 , M ] , h ∈ H we get: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: Because we see that in the above bound the dependence on δ , is given by a log term we are essentially getting an ”exponential-like” concentration.
Revised: Because we see that in the above bound the dependence on δ , is given by a log term we are essentially getting an ”exponential-like” concentration.

Original: So we can think about applying uniform convergence bounds over the class H × [0 , M ] to bounds the above with high probability ∀ ( h, η ) pairs.
Revised: So we can think about applying uniform convergence bounds over the class H × [0 , M ] to bounds the above with high probability ∀ ( h, η ) pairs.

Original: We will now try to get uniform convergence bounds with two approaches that make different assumptions on the class of functions l ( h ) .
Revised: We will now try to get uniform convergence bounds with two approaches that make different assumptions on the class of functions l ( h ) .

Original: The first is very generic and we will show why such a generic assumption is not sufficient to get an upper bound on the generalization that is O (1 / √ n ) in the worst case.
Revised: The first is very generic and we will show why such a generic assumption is not sufficient to get an upper bound on the generalization that is O (1 / √ n ) in the worst case.

Original: Then, in the second approach we show how assuming a parameterization will fetch us a rate of that form if we additionally assume that the loss function is L -Lipschitz.
Revised: Then, in the second approach we show how assuming a parameterization will fetch us a rate of that form if we additionally assume that the loss function is L -Lipschitz.

Original: Approach 1:
Revised: Approach 1:

Original: Assume l ( h ) lies in a class of ( α, 1) -H¨older continuous functions Now we shall use the following covering number bound for ( α, 1) -H¨older continuous functions to get a uniform convergence bound over H × [0 , M ] .
Revised: Assume l ( h ) lies in a class of ( α, 1) -H ¨ older continuous functions Now we shall use the following covering number bound for ( α, 1) -H ¨ older continuous functions to get a uniform convergence bound over H × [0 , M ] .

Original: Lemma C.5 (Covering number ( α, 1) -H¨older continuous) .
Revised: Lemma C.5 (Covering number ( α, 1) -H ¨ older continuous) .

Original: Let X be a bounded convex subset of
Revised: Let X be a bounded convex subset of

Original: R d with non-empty interior.
Revised: R d with non-empty interior.

Original: Then, there exists a constant K depending only on α and d such that
Revised: Then, there exists a constant K depending only on α and d such that

Original: [Equation]
Revised: [Equation]

Original: for every ϵ > 0 , where λ ( X 1 ) is the Lebesgue measure of the set { x : ∥ x −X∥ ≤ 1 } .
Revised: for every ϵ > 0 , where λ ( X 1 ) is the Lebesgue measure of the set { x : ∥ x −X∥ ≤ 1 } .

Original: Here, C α 1 ( X ) refers to the class of ( α, 1) -H¨older continuous functions.
Revised: Here, C α 1 ( X ) refers to the class of ( α, 1) -H¨older continuous functions.

Original: We assume that l ( h ) is ( α, 1) -H¨older continuous.
Revised: We assume that l ( h ) is ( α, 1) -H¨older continuous.

Original: And therefore by definition, of
Revised: And therefore by definition, of

Original: R ( h, η, · ) , the function is ( α, 1) -H¨older continuous in ( l ( h ) , η ) .
Revised: R ( h, η, · ) , the function is ( α, 1) -H ¨ older continuous in ( l ( h ) , η ) .

Original: Similat argument applies for sup δ ∈ ∆( W ,γ ) E w ∼ δ R
Revised: Similat argument applies for sup δ ∈ ∆( W ,γ ) E w ∼ δ R

Original: ( h, η, w ) since taking a pointwise supremum for a linear function over a convex set ∆( W , γ ) would retain H¨older continuity for some value of α .
Revised: ( h, η, w ) since taking a pointwise supremum for a linear function over a convex set ∆( W , γ ) would retain H ¨ older continuity for some value of α .

Original: Applying the above we get:log
Revised: Applying the above we get:

Original: [Equation]
Revised: [Equation]

Original: Now, we can show that with probability at least 1 − δ , ∀ h ∈ H we get:
Revised: Now, we can show that with probability at least 1 − δ , ∀ h ∈ H we get:

Original: [Equation]
Revised: [Equation]

Original: Note that in the above bound we cannot see if this upper bound shrinks as n → ∞ , without assuming something very strong about α .
Revised: Note that in the above bound we cannot see if this upper bound shrinks as n → ∞ , without assuming something very strong about α .

Original: Thus, we need covering number bounds that do not grow exponentially with the input dimension.
Revised: Thus, we need covering number bounds that do not grow exponentially with the input dimension.

Original: And for this we turn to parameterized classes, which is the next approach we take.
Revised: And for this we turn to parameterized classes, which is the next approach we take.

Original: It is more for the convenience of analysis that we introduce the following parameterization.
Revised: It is more for the convenience of analysis that we introduce the following parameterization.

Original: Approach 2:
Revised: Approach 2:

Original: Let l ( · , · ) be a [0 , M ] bounded L -Lipschitz function in ∥ · ∥ 2 over Θ where H be parameterized by a convex subset Θ ⊂ R d .
Revised: Let l ( · , · ) be a [0 , M ] bounded L -Lipschitz function in ∥ · ∥ 2 over Θ where H be parameterized by a convex subset Θ ⊂ R d .

Original: Thus we need to get a covering of the loss function sup δ E w ∼ δ R
Revised: Thus we need to get a covering of the loss function sup δ E w ∼ δ R

Original: ( θ, η, w ) in ∥ · ∥ ∞ norm, for a radius ϵ .
Revised: ( θ, η, w ) in ∥ · ∥ ∞ norm, for a radius ϵ .

Original: A standard practice is to bound this with a covering N (Θ , ϵL , ∥ · ∥ 2 ) , where ∥ · ∥ 2 is Euclidean norm defined on Θ ⊂ R d .
Revised: A standard practice is to bound this with a covering N (Θ , ϵL , ∥ · ∥ 2 ) , where ∥ · ∥ 2 is Euclidean norm defined on Θ ⊂ R d .

Original: Lemma C.6 (Covering number for N (Θ × [0 , M ] , ϵL , ∥ · ∥ 2 ) Wainwright (2019)) .
Revised: Lemma C.6 (Covering number for N (Θ × [0 , M ] , ϵL , ∥ · ∥ 2 ) Wainwright (2019)) .

Original: Let Θ be a bounded convex subset of R d with .
Revised: Let Θ be a bounded convex subset of R d with .

Original: [Equation]
Revised: [Equation]

Original: [Equation] We now re-iterate the steps we took previously: 
Revised: We now re-iterate the steps we took previously: [Equation]

Revised: (cid:118)(cid:117)(cid:117)(cid:117)(cid:116)

Original: [Equation]
Revised: [Equation]

Original: Note that the above holds with probability atleast 1 − δ and for ∀ h, η .
Revised: Note that the above holds with probability atleast 1 − δ and for ∀ h, η .

Original: Thus, we can apply it twice:
Revised: Thus, we can apply it twice:

Original: [Equation]
Revised: [Equation]

Original: where h ∗ , η ∗ are the optimal for L ∗ cvar .
Revised: where h ∗ , η ∗ are the optimal for L ∗ cvar .

Original: Combining the two above proves the statement in Theorem 5.1.
Revised: Combining the two above proves the statement in Theorem 5.1.

Original: C. P ROOF OF T HEOREM 5. 
Revised: C. P ROOF OF T HEOREM 5.

Original: Setup.
Revised: Setup.

Original: Let us assume there exists a prior Π such that W ( γ ) in Definition 4.1 is given by an RKHS induced by Mercer kernel k : X × X (cid:55)→ R , s.t.
Revised: Let us assume there exists a prior Π such that W ( γ ) in Definition 4.1 is given by an RKHS induced by Mercer kernel k : X × X (cid:55)→ R , s.t.

Original: the eigenvalues of the kernel operator decay polynomially:
Revised: the eigenvalues of the kernel operator decay polynomially:

Original: [Equation]
Revised: [Equation]

Original: for ( γ < 2) .
Revised: for ( γ < 2) .

Original: We solve for ˆ h γD , ˆ η γD by doing kernel ridge regression over norm bounded ( ∥ f
Revised: We solve for ˆ h γD , ˆ η γD by doing kernel ridge regression over norm bounded ( ∥ f

Original: ∥ W ( γ ) ≤ M ) smooth functions f .
Revised: ∥ W ( γ ) ≤ M ) smooth functions f .

Original: Thus, W ( γ ) is compact.
Revised: Thus, W ( γ ) is compact.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We show that we can control: (i) the pessimism of the learned solution; and (ii) the generalization error (Theorem 5.2).
Revised: We show that we can control: (i) the pessimism of the learned solution; and (ii) the generalization error (Theorem 5.2).

Original: Formally, we refer to pessimism for estimates ˆ h γD , ˆ η γD : excess risk or pessimism:
Revised: Formally, we refer to pessimism for estimates ˆ h γD , ˆ η γD : excess risk or pessimism:

Original: [Equation]
Revised: [Equation]

Original: Theorem C.7 ((restated for convenience) bounded RKHS) .
Revised: Theorem C.7 ((restated for convenience) bounded RKHS) .

Original: For l, H in Theorem 5.1, and for W ( γ ) described above ∃ γ 

Revised: For l, H in Theorem 5.1, and for W ( γ ) described above ∃ γ

Original: s.t.
Revised: s.t.

Original: for all sufficiently bitrate-constrained W ( γ ) i.e., γ ≤ γ 0 , w.h.p.
Revised: for all sufficiently bitrate-constrained W ( γ ) i.e., γ ≤ γ 0 , w.h.p.

Original: 1 − δ worst risk generalization error is O (cid:0) (1 /n ) (cid:0) log(1 /δ )
Revised: 1 − δ worst risk generalization error is O (cid:0) (1 /n ) (cid:0) log(1 /δ )

Original: + ( d + 1) log( nR − γ L γ/ 2 ) (cid:1)(cid:1)
Revised: + ( d + 1) log( nR − γ L γ/ 2 ) (cid:1)(cid:1)

Original: and the excess risk is
Revised: and the excess risk is

Original: O ( M ) for ˆ h γD , ˆ η γD above.
Revised: O ( M ) for ˆ h γD , ˆ η γD above.

Original: Generalization error proof:
Revised: Generalization error proof

Original: Note that the objective in Equation 30 is a non-parametric classification problem.
Revised: Note that the objective in Equation 32 is a non-parametric classification problem.

Original: We can convert this to the following non-parametric regression problem, after replacing the expectation with plug-in ˆ
Revised: We can convert this to the following non-parametric regression problem, after replacing the expectation with plug-in ˆ

Original: P n .
Revised: P n .

Original: [Equation]
Revised: [Equation]

Original: where λ n → 0 as n → ∞ .
Revised: where λ n → 0 as n → ∞ .

Original: Essentially, for non-parametric kernel ridge regression regression the regularization can be controlled to scale with the critical radius, that would give us better estimates and tighter localization bounds as we will see.
Revised: Essentially, for non-parametric kernel ridge regression regression the regularization can be controlled to scale with the critical radius, that would give us better estimates and tighter localization bounds as we will see.

Original: Note that in the above problem we add variable ϵ i which represents random noise ∼ N (0 , σ 2 ) .
Revised: Note that in the above problem we add variable ϵ i which represents random noise ∼ N (0 , σ 2 ) .

Original: Let σ 2 = 1 for convenience.
Revised: Let σ 2 = 1 for convenience.

Original: Since the noise is zero mean and random, any estimator maximizing the above objective on ˆ P n would be consistent with the estimator that has a noise free version.
Revised: Since the noise is zero mean and random, any estimator maximizing the above objective on ˆ P n would be consistent with the estimator that has a noise free version.

Original: We can also thing of this as a form regularization (similar to λ ), if we consider the kernel ridge regression problem as the means to obtain the Bayesian predictive posterior under a Bayesian prior that is a
Revised: We can also thing of this as a form regularization (similar to λ ), if we consider the kernel ridge regression problem as the means to obtain the Bayesian predictive posterior under a Bayesian prior that is a

Original: Gaussian Process GP ( 0 , σ 2 k ( x , x )) , under the same kernel as defined above.
Revised: Gaussian Process GP ( 0 , σ 2 k ( x , x )) , under the same kernel as defined above.

Original: First we will show estimation error bounds for the following KRR estimate: 
Revised: First we will show estimation error bounds for the following KRR estimate: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The estimation error would be measured in terms of ˆ P n norm i.e., ∥ ˆ w γD − w ∗ ∥
Revised: The estimation error would be measured in terms of ˆ P n norm i.e., ∥ ˆ w γD − w ∗ ∥

Original: ˆ P n where
Revised: ˆ P n where

Original: [Equation]
Revised: [Equation]

Original: is the best solution to the optimization objective in population.
Revised: is the best solution to the optimization objective in population.

Original: Next steps:
Revised: Next steps:

Original: • First, we get the estimation error in ∥ ˆ w γD − w ∗ γ ∥
Revised: • First, we get the estimation error in ∥ ˆ w γD − w ∗ γ ∥

Original: ˆ P n of ˆ P n .
Revised: ˆ P n of ˆ P n .

Original: • Then using uniform laws (Wainwright, 2019) we can extend it to L 2 ( P ) norm i.e., ∥ˆ w γD − w ∗ ∥ p .
Revised: • Then using uniform laws (Wainwright, 2019) we can extend it to L 2 ( P ) norm i e., ∥

Original:  • Then we shall prove that if we convert the ˆ w γD and w ∗ into prediction rules: ˆ w γD ≥ 0 and w ∗ γ , then we can get the estimation error of prdedictor ˆ w γD ≥ 0 with respect to the optimal decision rule w ∗ γ ≥ 0 in class W ( γ ) .
Revised: ˆ w γD − w ∗ ∥ p . • Then we shall prove that if we convert the ˆ w γD and w ∗ into prediction rules: ˆ w γD ≥ 0 and w ∗ γ , then we can get the estimation error of prdedictor ˆ w γD ≥ 0 with respect to the optimal decision rule w ∗ γ ≥ 0 in class W ( γ ) .

Original: • The final step would give us an oracle inequality of the form in Theorem 5.1.
Revised: • The final step would give us an oracle inequality of the form in Theorem 5.1.

Original: Based on the outline above, let us start with getting ∥ ˆ w γD − w ∗ ∥
Revised: Based on the outline above, let us start with getting ∥ ˆ w γD − w ∗ ∥

Original: ˆ P n .
Revised: ˆ P n .

Original: For this we shall use concentration inequalities from localization bounds (see Lemma C.8).
Revised: For this we shall use concentration inequalities from localization bounds (see Lemma C.8).

Original: Before we use that, we define the quantity δ n , which is the critical radius (see Ch.
Revised: Before we use that, we define the quantity δ n , which is the critical radius (see Ch.

Original: 13.4 in Wainwright (2019)).
Revised: 13.4 in Wainwright (2019)).

Original: For convenience, we also state it here.
Revised: For convenience, we also state it here.

Original: Formally, δ n is the smallest value of δ that satisfies the following inequality (critical condition):
Revised: Formally, δ n is the smallest value of δ that satisfies the following inequality (critical condition):

Original: [Equation]
Revised: [Equation]

Original: where,
Revised: where,

Original: R n ( δ ) :
Revised: R n ( δ ) :

Original: =
Revised: =

Revised: E

Original: [Equation]
Revised: [Equation]

Original: and ϵ is some sub-Gaussian zero mean random variable.
Revised: and ϵ is some sub-Gaussian zero mean random variable.

Original: Lemma C.8 ( (Wainwright, 2019)) .
Revised: Lemma C.8 ( (Wainwright, 2019)) .

Original: For some convex RKHS class F Let ˆ f be defined as:
Revised: For some convex RKHS class F Let ˆ f be defined as:

Original: [Equation]
Revised: [Equation]

Original: then, with probability ≥ 1 − c 2 exp (cid:16) − c 3 nR 

Revised: then, with probability ≥ 1 − c 2 exp (cid:16) − c 3 nR

Original: δ 2 n σ 2 (cid:17) and when λ n ≥ δ 2 n we get:
Revised: δ 2 n σ 2 (cid:17) and when λ n ≥ δ 2 n we get:

Original: [Equation]
Revised: [Equation]

Original: Note that it is standard exercise in statistics to derive the following closed form for the problem in
Revised: Note that it is standard exercise in statistics to derive the following closed form for the problem in

Original: Equation 32:
Revised: Equation 34:

Original: [Equation]
Revised: [Equation]

Original: where l ( h ) D is the loss vector and ϵ D is the noise vector for dataset D and ˆ K n is the empirical kernel matrix given by ˆ K i,j = 1 n k (( x i , y i ) , ( x j , y j )) , and Z is a matrix of ( x, y ) pairs in dataset.
Revised: where l ( h ) D is the loss vector and ϵ D is the noise vector for dataset D and ˆ K n is the empirical kernel matrix given by ˆ K i,j = 1 n k (( x i , y i ) , ( x j , y j )) , and Z is a matrix of ( x, y ) pairs in dataset.

Original: Corollary C.9.
Revised: Corollary C.9.

Original: Wainwright (2019) Let ˆ µ j
Revised: Wainwright (2019) Let ˆ µ j

Original: be
Revised: be

Original: the eigen values ˆ µ 1 ≥ [Equation] µ 
Revised: the eigen values ˆ µ 1 ≥ ˆ µ 2 . . .

Original: [Equation]

Original: ≥ ˆ µ n for the empirical
Revised: ≥ ˆ µ n for the empirical

Original: Kernel matrix ˆ K , then we have for any δ satisfying
Revised: Kernel matrix ˆ K , then we have for any δ satisfying

Original: [Equation]
Revised: [Equation]

Original: , it is necessary that δ satisfies the critical condition in Equation 33.
Revised: , it is necessary that δ satisfies the critical condition in Equation 35.

Original: To show the above critical condition we shall now use the polynomial decaying property that for the specific kernel induced by W ( γ ) , as stated in our assumption in the beginning of this section.
Revised: To show the above critical condition we shall now use the polynomial decaying property that for the specific kernel induced by W ( γ ) , as stated in our assumption in the beginning of this section.

Original: For this we take standard approach taken for polynomial decay kernels Zhang et al.
Revised: For this we take standard approach taken for polynomial decay kernels Zhang et al.

Original: Let ∃ C for some large C > 0 such that ˆ µ j ≤ Cj − 2 /γ .
Revised: Let ∃ C for some large C > 0 such that ˆ µ j ≤ Cj − 2 /γ .

Original: Then for some k , such that δ 2 ≥ ck − 2 /γ
Revised: Then for some k , such that δ 2 ≥ ck − 2 /γ

Original: [Equation]
Revised: [Equation]

Original: Now, setting the above into the critical condition equation from Corollary above:
Revised: Now [Equation] setting the above into the critical condition equation from Corollary above:

Original: [Equation]
Revised: [Equation]

Original: This tells us that:
Revised: This tells us that:

Original: [Equation]
Revised: [Equation]

Original: is the critical radius.
Revised: is the critical radius.

Original: We shall later plug this into the bound we have into a uniform bound over the concentration inequality in Lemma C.8.
Revised: We shall later plug this into the bound we have into a uniform bound over the concentration inequality in Lemma C.8.

Original: The reason we need a uniform bound over Lemma C.8 is that in its current form, it only bounds ∥
Revised: The reason we need a uniform bound over Lemma C.8 is that in its current form, it only bounds ∥

Original: ˆ w γD − w γ ∗ ∥ 2ˆ P n for a specific choice of η, h .
Revised: ˆ w γD − w γ ∗ ∥ 2ˆ P n for a specific choice of η, h .

Original: In order to arrive at the worst risk generalization error of the form we have in Theorem 5.1 we need to satisfy that with high probability 1 − δ ∀
Revised: In order to arrive at the worst risk generalization error of the form we have in Theorem 5.1 we need to satisfy that with high probability 1 − δ ∀

Original: η, h , a critical concentration bound of the form in Lemma C.8 but over sup η,h ∥
Revised: η, h , a critical concentration bound of the form in Lemma C.8 but over sup η,h ∥

Original: ˆ w γD − w γ ∗ ∥ 2ˆ P n .
Revised: ˆ w γD − w γ ∗ ∥ 2ˆ P n .

Original: Let ϵ =
Revised: Let ϵ =

Original: c 2 exp (cid:16) c 3 nR 2 δ n 2 σ 2 (cid:17) .
Revised: c 2 exp (cid:16) c 3 nR 2 δ n 2 σ 2 (cid:17) .

Original: Since δ 2 n needs to be large enough (see condition in Equation 34), we use Lemma C.8 in the following bound, incorporating δ n condition we derived.
Revised: Since δ 2 n needs to be large enough (see condition in Equation 36), we use Lemma C.8 in the following bound, incorporating δ n condition we derived.

Original: With high probability 1 − ϵ :
Revised: With high probability 1 − ϵ :

Original: [Equation]
Revised: [Equation]

Original: To apply uniform convergence argument on the above we would need to apply a union bound on a covering of Θ × [0 , M ] , so that we get the probability bound to hold for all η, h .
Revised: To apply uniform convergence argument on the above we would need to apply a union bound on a covering of Θ × [0 , M ] , so that we get the probability bound to hold for all η, h .

Original: For this we use the same technique as in the proof of Theorem 5.1.
Revised: For this we use the same technique as in the proof of Theorem 5.1.

Original: First, we shall use Lemma C. to get a covering number bound for bounded convex subset Θ of R d that parameterizes the learner (Theorem 5.2) .
Revised: First, we shall use Lemma C. to get a covering number bound for bounded convex subset Θ of R d that parameterizes the learner (Theorem 5.2) .

Original: [Equation]
Revised: [Equation]

Original: [Equation] And we know that a covering of Θ × [0 , M ] in radius β/L , will fetch a covering for l ( h ) − η in β , since we assume l ( · ) to be Lipschitz in θ .
Revised: And we know that a covering of Θ × [0 , M ] in radius β/L , will fetch a covering for l ( h ) − η in β , since we assume l ( · ) to be Lipschitz in θ .

Original: Thus, all we need to prove bound Equation 35 holds uniformly is to get a covering in radius R 2 max (cid:16)(cid:0) 1 nR 2 (cid:1) 

Revised: Thus, all we need to prove bound Equation 37 holds uniformly is to get a covering in radius R 2 max (cid:16)(cid:0) 1 nR 2 (cid:1)

Original: γ +2 , (cid:0) log(1 /ϵ )
Revised: γ +2 , (cid:0) log(1 /ϵ )

Original: 1 nR 2 (cid:1)(cid:17) .
Revised: 1 nR 2 (cid:1)(cid:17) .

Original: Thus, a acovering in R 2 (cid:16)(cid:0) 1 nR 2 (cid:1) 2 γ +2 (cid:17) .
Revised: Thus, a acovering in R 2 (cid:16)(cid:0) 1 nR 2 (cid:1) 2 γ +2 (cid:17) .

Original: Thus, the number of elements in cover are:
Revised: Thus, the number of elements in cover are:

Original: [Equation]
Revised: [Equation]

Original: For union bound we need:
Revised: For union bound we need:

Original: [Equation]
Revised: [Equation]

Original: The uniform convergence bound that we get is R 2 max (cid:18)(cid:0) [Equation] J/ϵ [Equation]
Revised: The uniform convergence bound that we get is R 2 max (cid:18)(cid:0) 1 nR 2 (cid:1) γ +2 γ , (cid:0) log( J/ϵ ) 1 nR 2 (cid:1)(cid:19) .

Original: In the above sequence of steps we have shown that, due to the size of J , the second term would be maximum, or at least there exists a γ 0 , such that the second term would be higher for all γ ≥ γ 0 , for any sample size.
Revised: In the above sequence of steps we have shown that, due to the size of J , the second term would be maximum, or at least there exists a γ 0 , such that the second term would be higher for all γ ≥ γ 0 , for any sample size.

Original: Thus, we get the following probabilistic uniform convergence.
Revised: Thus, we get the following probabilistic uniform convergence.

Original: With probability ≥ 1 − ϵ , ∀ η, h :
Revised: With probability ≥ 1 − ϵ , ∀ η, h :

Original: [Equation]
Revised: [Equation]

Original: Applying the above twice, once one ˆ w γD and another on w γ ∗ we prove the generalization bound in
Revised: Applying the above twice, once one ˆ w γD and another on w γ ∗ we prove the generalization bound in

Original: Theorem 5.2.
Revised: Theorem 5.2.

Original: Excess risk bound:
Revised: Excess risk bound:

Original: In the same setting we shall now prove the excess risk bound.
Revised: In the same setting we shall now prove the excess risk bound.

Original: Recall the definition of excess risk:
Revised: Recall the definition of excess risk:

Original: [Equation]
Revised: [Equation]

Original: Let h ∗ ( w ) , η ∗ ( w ) =
Revised: Let h ∗ ( w ) , η ∗ ( w ) =

Original: inf h,η R ( h, η, w ) , then:
Revised: inf h,η R ( h, η, w ) , then:

Original: [Equation]
Revised: [Equation]

Original: Note, that according to our assumption ∥ w ∥ W ( γ ) ≤ B i.e., the smooth functions are bounded in
Revised: Note, that according to our assumption ∥ w ∥ W ( γ ) ≤ B i.e., the smooth functions are bounded in

Original: RKHS norm.
Revised: RKHS norm.

Original: The following lemma relates bounds in RKHS norm to bound in L 2 ( P ) bound for kernels with bounded operator norms:
Revised: The following lemma relates bounds in RKHS norm to bound in L 2 ( P ) bound for kernels with bounded operator norms:

Original: Lemma C.10.
Revised: Lemma C.10.

Original: For an RKHS H k with norm ∥ · ∥ H k :
Revised: For an RKHS H k with norm ∥ · ∥ H k :

Original: [Equation]
Revised: [Equation]

Original: Proof:
Revised: Proof:

Original: [Equation]
Revised: [Equation]

Original: In the above λ
Revised: In the above λ

Original: j are the Eigen values of the kernel and the Eigen functions ϕ
Revised: j are the Eigen values of the kernel and the Eigen functions ϕ

Original: j are orthonormal and span L 2 ( P ) /
Revised: j are orthonormal and span L 2 ( P ) /

Original: Thus, ∥ f
Revised: Thus, ∥ f

Original: ∥ L 2 ( P ) ≤ ∥ T 1 / 2 K ∥ op · ∥ f ∥ H k .
Revised: ∥ L 2 ( P ) ≤ ∥ T 1 / 2 K ∥ op · ∥ f ∥ H k .

Original: Since we assume polynomially decaying Eigen values for our kernel, it is easy to see that ∥ T 1 / 2 K ∥ op
Revised: Since we assume polynomially decaying Eigen values for our kernel, it is easy to see that ∥ T 1 / 2 K ∥ op

Original: = O (1) .
Revised: = O (1) .

Original: Applying Lemma C.10 to Equation 44, directly gives us the excess risk bound and completes the proof.[Equation] C
Revised: Applying Lemma C.10 to Equation 46, directly gives us the excess risk bound and completes the proof.

Original: [Equation]
Revised: [Equation]

Original:  P ROOF  
Revised: C. P ROOF OF T HEOREM 5.

Original: [Equation] T HEOREM 5.

Original: Setup.
Revised: Setup.

Original: The algorithm is as follows: Consider a two-player zero-sum game where the learner uses a no-regret strategy to first play h ∈ H , η ∈ R to minimize E w ∼ δ R
Revised: The algorithm is as follows: Consider a two-player zero-sum game where the learner uses a no-regret strategy to first play h ∈ H , η ∈ R to minimize E w ∼ δ R

Original: ( h, η, w ) .
Revised: ( h, η, w ) .

Original: Then, the adversary plays follow the regularized leader (FTRL) strategy to pick distribution δ ∈ ∆( W , γ ) to maximize the same.
Revised: Then, the adversary plays follow the regularized leader (FTRL) strategy to pick distribution δ ∈ ∆( W , γ ) to maximize the same.

Original: The regularizer used is a negative entropy regularizer.
Revised: The regularizer used is a negative entropy regularizer.

Original: Our goal is to analyze the bitrateconstraint γ ’s effect on the above algorithm’s convergence rate and the pessimistic nature of the solution found.
Revised: Our goal is to analyze the bitrateconstraint γ ’s effect on the above algorithm’s convergence rate and the pessimistic nature of the solution found.

Original: For this, we need to first characterize the bitrate-constraint class W ( γ ) .
Revised: For this, we need to first characterize the bitrate-constraint class W ( γ ) .

Original: So we assume there exists a prior Π such that W ( γ ) is Vapnik-Chervenokis (VC) class of dimension O ( γ ) .
Revised: So we assume there exists a prior Π such that W ( γ ) is Vapnik-Chervenokis (VC) class of dimension O ( γ ) .

Original: Note that R ( h, η, w ) is convex in h and linear in η, l .
Revised: Note that R ( h, η, w ) is convex in h and linear in η, l .

Original: Thus, as we discuss in the derivation for
Revised: Thus, as we discuss in the derivation for

Original: Equation 6 this objective optimized over convex sets has a unique saddle point (Nash equilibrium) by Weierstrass’s theorem.
Revised: Equation 6 this objective optimized over convex sets has a unique saddle point (Nash equilibrium) by Weierstrass’s theorem.

Original: Thus, to avoid repetition we only discuss the proofs for the other two claims on convergence and excess risk.
Revised: Thus, to avoid repetition we only discuss the proofs for the other two claims on convergence and excess risk.

Original: Convergence:
Revised: Convergence:

Original: Given that W ( γ ) is a VC class of dimension Cγ for some large C , we can use Sauer-Shelah Bartlett et al.
Revised: Given that W ( γ ) is a VC class of dimension Cγ for some large C , we can use Sauer-Shelah Bartlett et al.

Original: Lemma (stated) below to bound the total number of groups that can be identified by
Revised: Lemma (stated) below to bound the total number of groups that can be identified by

Original: W ( γ ) in n points.
Revised: W ( γ ) in n points.

Original: Lemma C.11 (Sauer’s Lemma) .
Revised: Lemma C.11 (Sauer’s Lemma) .

Original: The Vapnik-Chervonenkis dimension of a class F , denoted as VCdim( F ), and it is the cardinality of the largest set S shattered by F .
Revised: The Vapnik-Chervonenkis dimension of a class F , denoted as VCdim( F ), and it is the cardinality of the largest set S shattered by F .

Original: Let d = V C − dim ( F ) , then for all m , C [ m ]
Revised: Let d = V C − dim ( F ) , then for all m , C [ m ]

Original: = O ( m d )
Revised: = O ( m d )

Original: Thus, the total number of groups that can be proposed on n points by W ( γ ) is O ( n γ ) .
Revised: Thus, the total number of groups that can be proposed on n points by W ( γ ) is O ( n γ ) .

Original: A similar observation was made in Kearns et al.
Revised: A similar observation was made in Kearns et al.

Original: Different from them, our goal is to analyze the algorithm iterates for our solver described above and bound its pessimism.
Revised: Different from them, our goal is to analyze the algorithm iterates for our solver described above and bound its pessimism.

Original: First, for convergence rate we show that the above algorithm has a low regret—a standard exercise in online convex optimization.
Revised: First, for convergence rate we show that the above algorithm has a low regret—a standard exercise in online convex optimization.

Original: Note that any distribution picked by the adversary can be seen as multinomial over a finite set of possible groups that is let’s say K , and from discussion above we know that K
Revised: Note that any distribution picked by the adversary can be seen as multinomial over a finite set of possible groups that is let’s say K , and from discussion above we know that K

Original: = O ( n γ ) .
Revised: = O ( n γ ) .

Original: Further, the negative entropy regularizer is given as:
Revised: Further, the negative entropy regularizer is given as:

Original: [Equation]
Revised: [Equation]

Original: where the sum is over total possible groups identified by W ( γ ) .
Revised: where the sum is over total possible groups identified by W ( γ ) .

Original: Let the probability assigned to group i be denoted as δ i .
Revised: Let the probability assigned to group i be denoted as δ i .

Original: The FTRL strategy for adversary is given as:
Revised: The FTRL strategy for adversary is given as:

Original: [Equation]
Revised: [Equation]

Original: Then the regret for not having picked a single action δ is given as:
Revised: Then the regret for not having picked a single action δ is given as:

Original: [Equation]
Revised: [Equation]

Original: We bound the two terms in the above bound separately.
Revised: We bound the two terms in the above bound separately.

Original: With (cid:80) kk =1 δ k
Revised: With (cid:80) kk =1 δ k

Original: = 1 , we get the strong dual for the FTRL update above as:
Revised: = 1 , we get the strong dual for the FTRL update above as:

Original: [Equation]
Revised: [Equation]

Original: Solving we get:
Revised: Solving we get:

Original: [Equation]
Revised: [Equation]

Original: where E ˆ P n ( l ( h t ) − η t | G k ) is the expected empirical loss in group G k and δ t ( k ) is the adversary’s distribution at time step t for the k th group.
Revised: where E ˆ P n ( l ( h t ) − η t | G k ) is the expected empirical loss in group G k and δ t ( k ) is the adversary’s distribution at time step t for the k th group.

Original: Claim on stability:
Revised: Claim on stability:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The above statement is true because,
Revised: The above statement is true because,

Original: [Equation]
Revised: [Equation]

Original: Thus, if l ( h t ) ∈ [0 , M/α 0 ] , i.e., losses are bounded then:δ t +1 ( i ) ≥ δ t ( i ) ·
Revised: Thus, if l ( h t ) ∈ [0 , M/α 0 ] , i.e., losses are bounded then:

Original: e − 1 /c ≥ δ t ( i ) · (1 − 1 /c ) .

Original: [Equation]
Revised: [Equation]

Original: and our stability claim is easy to see.
Revised: and our stability claim is easy to see.

Original: Thus, we have bounded the first term in our regret bound above.
Revised: Thus, we have bounded the first term in our regret bound above.

Original: Further, we can to see that B ( x ) − B ( x 1 ) ≤ c log K .
Revised: Further, we can to see that B ( x ) − B ( x 1 ) ≤ c log K .

Original: Thus, we have bounded both terms in the regret bound above in terms of c .
Revised: Thus, we have bounded both terms in the regret bound above in terms of c .

Original: [Equation]
Revised: [Equation]

Original: Setting c = (cid:113) T log K , we get:
Revised: Setting c = (cid:113) T log K , we get:

Original: REGRET
Revised: REGRET

Original: [Equation]
Revised: [Equation]

Original: Now, our VC claim gave K = O ( n γ ) .
Revised: Now, our VC claim gave K = O ( n γ ) .

Original: Hence,
Revised: Hence,

Original: REGRET T T =
Revised: [Equation]

Original: O (cid:114)

Original: [Equation]
Revised: [Equation]

Original: T (55)

Original: Next, we use Theorem 9 from Abernethy et al.
Revised: Next, we use Theorem 9 from Abernethy et al.

Original: (2018) that maps low regret O ( ϵ ) algorithms in zero-sum convex-concave games to ϵ -optimal equilibriums.
Revised: (2018) that maps low regret O ( ϵ ) algorithms in zero-sum convex-concave games to ϵ -optimal equilibriums.

Original: Let regret be ϵ , then applying their theorem gives us:
Revised: Let regret be ϵ , then applying their theorem gives us:

Original: [Equation]
Revised: [Equation]

Original: 56) where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Excess risk:
Revised: Excess risk:

Original: For excess risk we need to bound:
Revised: For excess risk we need to bound:

Original: [Equation]
Revised: [Equation]

Original: In the above argument we used the fact that at equilibrium, δ ∗ ( γ ) would be uniform over all possible distinct group assignments.
Revised: In the above argument we used the fact that at equilibrium, δ ∗ ( γ ) would be uniform over all possible distinct group assignments.

Original: This completes our proof of Theorem 5.4.
Revised: This completes our proof of Theorem 5.4.

Revised: C. W ORST - CASE GENERALIZATION RISK FOR G ROUP DRO

Revised: Recall that the BR-DRO objective in Equation 4 involves an expectation over P , which in practice is replaced by empirical distribution ˆ P n .

Revised: This induces errors in estimating the worst-case risk for plug-in estimates ˆ h γD , ˆ η γD

Revised: In Theorem 5.1 we saw how the bitrate-constraint gracefully controls the worst-case generalization guarantee for estimates ˆ h γD , ˆ η γD through an oracle inequality.

Revised: Here, we ask: “How does the worst-case generalization for Group DRO compare with the bound in Theorem 5.1?”.

Revised: First, let us recall the objective for Group DRO (Equation 62) which assumes the knowledge of ground-truth groups G 1 , G 2 , . . .

Revised: , G K .

Revised: [Equation]

Revised: Now, let us denote the plug-in estimate as ˆ h KD which solves the above objective using the empirical distribution ˆ P n .

Revised: We are now ready to state Theorem C.12 which gives us the worst-risk generalization error for Group DRO.

Revised: Theorem C.12 (worst-case risk generalization (Group DRO)) .

Revised: With probability ≥ 1 − δ over D ∼

Revised: P n , the worst group risk for ˆ h KD can be upper bounded by the following oracle inequality:

Revised: [Equation]

Revised: when l ( · , · ) is [0 , M ] -bounded, L -Lipschitz and H is parameterized by convex set Θ ⊂ R d .

Revised: Proof Sketch.

Revised: We can use the covering number result in Lemma C.5, and plug it into the Hoeffding bound in

Revised: Lemma C.1 to bound the generalization gap for a specific group G k .

Revised: This yields the following result with probability at least 1 − δ :

Revised: [Equation]

Revised: Next, we use union bound over the K groups to get the final result in Theorem C.12, with probability at least 1 − δ .

Revised: The main difference between the generalization analysis BR-DRO and Group DRO is the order in which we bound the errors for the learner and the adversary (groups).

Revised: For Group DRO, we begin with the learner since we can use traditional uniform convergence bounds to bound per group generalization risk and then use union bound over the K groups to bound the worst-case risk.

Revised: We can do this precisely since we can use the pre-determined groups G 1 , G 2 , . . .

Revised: , G K assumed by

Revised: Group DRO.

Revised: On the other hand, since we do not assume this knowledge for BR-DRO , we first use

Revised: PAC-Bayes bound (Lemma C.4) to control the generalization error per group through the bitrate constraint γ .

Revised: The PAC-Bayes bound allows us to reason about the generalization error for a specific learner h ∈ H .

Revised: Finally, to get the bound in Thorem 5.1 we relied on a covering number argument for H parameterized as a convex subset of R d (Section C.3).