Original: R EGULARIZED L EARNING FOR D OMAIN A DAPTATION UNDER L ABEL S HIFTS
Revised: R EGULARIZED L EARNING FOR D OMAIN A DAPTATION UNDER L ABEL S HIFTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose Regularized Learning under Label shifts ( RLLS ), a principled and practical  approach to correct for shifts in the label distribution between the source andtarget domains, and is a special case of domain adaptation.
Revised: We propose Regularized Learning under Label shifts ( RLLS ), a principled and practical domain adaptation approach to correct for shifts in the label distribution between a source and a target domain.

Original: We  estimatethe importance weights using both labeled source and unlabeled target data and then train a classiﬁer on the weighted source samples.
Revised: We ﬁrst estimate importance weights using both labeled source and unlabeled target data before training a classiﬁer on the weighted source samples.

Original: Depending on the number of source and target samples, we  regularize the inﬂuence of these estimatedweights and train a corrected classiﬁer on the source set with the estimated weights.
Revised: Depending on the number of source and target samples, we thereby regularize the inﬂuence of these estimated weights.

Original: We derive a generalization bound for theloss of the classiﬁer on the target set. This bound is independent of the (ambient) data dimensions and instead depends on the complexity of the function class.
Revised: We derive a generalization bound for the classiﬁer on the target domain which is independent of the (ambient) data dimensions and instead depends on the complexity of the function class.

Original: To the best of our knowledge, this is the ﬁrst generalization bound for label-shift problems where the labels in the target domain are unknown.
Revised: To the best of our knowledge, this is the ﬁrst generalization bound for label-shift problems where the labels in the target domain are unknown.

Original: Based on this bound, we design a regularized estimator for importance weights that allows us to adapt to the uncertainty in the estimated weights for the small-sample regime.
Revised: Based on this bound, we propose a regularized estimator for the small-sample regime which accounts for the uncertainty in the estimated weights.

Original: Experiments  show thatregularized RLLSsigniﬁcantly improves  accuracy in the low(target) sample and large-shift regimes compared to previous methods.
Revised: Experiments on the CIFAR-10 and MNIST datasets show that RLLS improves classiﬁcation accuracy in the low sample and large-shift regimes compared to previous methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In supervised learning, the task to make predictions on a given unlabeled target setis usually solved by training a predictive model on a labeled source set.
Revised: In supervised learning, we usually make predictions on an unlabeled target set by training a predictive model on a labeled source set.

Original: This is a sensible approach particularly when the source and target sets are expected  have i.i.d.
Revised: This is a sensible approach especially when the source and target sets are expected to have i.i.d.

Original: samples from the same distribution.
Revised: samples from the same distribution.

Original: However, this assumption does not hold in many real-world applications.
Revised: However, this assumption does not hold in many real-world applications.

Original: For example, often times predictions for medical diagnostics are based on data from a particular population of patients and machines due to the limited variety of data that is available in the training phase.
Revised: For example, often times predictions for medical diagnostics are based on data from a particular population of patients and machines due to the limited variety of data that is available in the training phase.

Original: To further expand on the example above, consider the following scenario.
Revised: Let us consider the following scenario.

Original: A classiﬁer is trained to predict whether a patient has contracted a severe disease in country A based on the physical measurements and diagnosis (labeled source data).
Revised: A classiﬁer is trained to predict whether a patient has contracted a severe disease in country A based on the physical measurements and diagnosis (labeled source data).

Original: The disease is potentially even more prevalent in country B, wherehowever no doctors are available to make a diagnosis which can only be reliably made using expert knowledge.
Revised: The disease is potentially even more prevalent in country B, where no experts are available to make a reliable diagnosis.

Original: The following questions are relevant in this scenario:
Revised: The following questions could be relevant in this scenario:

Original: • If we send a group of volunteers to country B with the same equipment to record measurements of patients, how can the data in country
Revised: • If we send a group of volunteers to country B with the preliminary equipment to record measurements of patients, how can the data in country

Original: A
Revised: A

Original: and the new unlabeled data from country B be used to update the classiﬁer to give good diagnostic predictions for patients in country B?
Revised: and the new unlabeled data from country B be used to update the classiﬁer to give good diagnostic predictions for patients in country B?

Original: • If  the patients in country Bcome in slowly over time and the disease is deadly, can the classiﬁer be updated after only few patients have been measured, so as to maximize the accuracy of the diagnosis for the following patients?
Revised: • If our group members visit the patients in country B in online and daily manner rather than batch and the disease is deadly, can the classiﬁer be updated after only few patients have been measured, so as to maximize the accuracy of the diagnosis for the following patients?

Original: Similar problems arise in many other domains besides health care, but we will use this medical example throughout the paper for ease of presentation.
Revised: Similar problems arise in many other domains besides health care, but we will use this medical example throughout the paper for ease of presentation.

Original: If we view the data  in the source and target set, each consisting of an input x ∈ X and a label y in a ﬁnite domain Y , as i.i.d.
Revised: If we view the data points ( X i , Y i ) in the source and target set as i.i.d.

Original: samplesdrawn from  joint distributions P and Q  , we are dealing with the distributional shift problem when P (cid:54) =
Revised: samples from a joint distributions P and Q taking values in X × Y , we are dealing with the distributional shift problem when P (cid:54) =

Original: Q .
Revised: Q .

Original: There are various options to model the difference between P and Q , depending on the use case at hand.
Revised: There are various ways to model the difference between P and Q , depending on the use case at hand.

Original: Schölkopf et al.

Original: (2012) studies two main perspectives: (i)  causal learning where theexplanatory input x causes the label y , and ( ii )  anti-causal setting, where the label y causes the input x .
Revised: Traditionally, the two following perspectives are considered: (i) the causal setting where the input x explains the label y , and ( ii ) the anti-causal setting, where the label y causes the input x .

Original: For the causal setting, one usually makes the covariate shift assumption that the marginal distribution of the covariates x shifts from P to Q , whereas the conditional probability mass function of y given x stays constant, which we denote as p ( y | x ) =
Revised: For the causal setting, one usually assumes that the marginal distribution of the covariates x shifts between P and Q , whereas the conditional probability mass function of y given x stays constant, which we denote as p ( y | x ) =

Original: q ( y | x ) for all y ∈ Y and x ∈ X .
Revised: q ( y | x ) for all y ∈ Y and x ∈ X .

Original: Although this isa validmodel in some situations, in many data science applicationsthere is an inherent ground truth y that we want to learn given observations x , which can be "explained" by y .
Revised: Although this is valid in some situations, in many data science applications an inherent ground truth Y causes an observation X .

Original: For example, the physical symptomsreported by a patient might be approximately similar among all sick patients.
Revised: For example, reported physical symptoms might be approximately similar across all patients that have contracted a particular disease.

Original: This simpliﬁed relation can be modeled e.g. by a directed graphical model Y → X .
Revised: This simpliﬁed relation can be modeled e.g. by a directed graphical model Y → X .

Original: It is now possible that the percentage of ill patients in the two countries A and B are very different, i.e. P ( Y = i ) (cid:54) =
Revised: It is now possible that the percentage of ill patients in the two countries A and B are very different, i.e. P ( Y = y ) (cid:54) =

Original: Q ( Y = i ) , or p ( y ) (cid:54) =
Revised: Q ( Y = y ) ( p ( y ) (cid:54) =

Original: q ( y ) for short.
Revised: q ( y ) for short) for all y ∈ Y .

Original: Similarly, within one country, the number of ill patients may suddenly change because of an irregular event such as a natural catastrophe.
Revised: Similarly, within one country, the number of ill patients may suddenly change because of an irregular event such as a natural catastrophe.

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: In all of the above cases, there is a shift in the distribution of the label y(sick or healthy) , while given a ﬁxed label, the conditional distribution of x given y is constant.
Revised: In all of the above cases, there is a shift in the distribution of the label y , while given a ﬁxed label, the conditional distribution of x given y is constant, i.e., p ( x | y )

Original:  By the Bayes rule, label shift causes the covariate shift condition to be violated, i.e. p ( y | x ) (cid:54) =
Revised: = q ( x | y ) . By the Bayes rule, label shift causes the covariate shift condition to be violated, i.e. p ( y | x ) (cid:54) =

Original: q ( y | x ) .
Revised: q ( y | x ) .

Original: Despite being an intuitively natural scenario in some cases, even this simpliﬁed model has not yet been sufﬁciently studied in the literature.
Revised: Despite being an intuitively natural scenario in some cases, even this simpliﬁed model has not yet been sufﬁciently studied in the literature.

Original: This paper aims to ﬁll the gap in terms of both theoretical understanding and practical methods for the label shift setting and thereby move a step closer towards more complicated and realistic distributional shift models.
Revised: This paper aims to ﬁll the gap in terms of both theoretical understanding and practical methods for the label shift setting and thereby move a step closer towards more complicated and realistic distributional shift models.

Revised: In particular, our goal was to ﬁnd an efﬁcient method which is scalable to large-scale data where conventional approaches such as ones based on kernel mean embeddings, may become computationally feasible.

Original: Lipton et al.
Revised: Lipton et al.

Original: (2018) introduced a provably good importance-weight estimator called BBSE , which has access to the labeled source data set and a large batch of unlabeled target data, and demonstrated promising performance on real data. 
Revised: (2018) introduced an importance-weight estimator, which has access to the labeled source data set and a large batch of unlabeled target data, and demonstrated promising performance on real data. Using a black-box classiﬁer which can be biased, uncalibrated and inaccurate, they ﬁrst estimate importance weights q ( y ) /p ( y ) for the source samples and train a classiﬁer on the weighted data. In the following we refer to the procedure as black box shift learning ( BBSL ) which the authors proved to be good for large enough sample size.

Original: There area two natural follow-up questions which remain unanswered by their work: What are the generalization guarantees for the ﬁnal predictor which uses the weighted samples?
Revised: There are two relevant questions which remain unanswered by their work: What are the generalization guarantees for the ﬁnal predictor which uses the weighted samples?

Original: How do we deal with the uncertainty of the weight estimation when only few samples are available and how can we update the classiﬁer?
Revised: How do we deal with the uncertainty of the weight estimation when only few samples are available?

Original: In this work, we propose an efﬁcient method to estimate the importance weights and relax the problem-dependent warm-up sample complexity requirement for BBSE (which is unknown and can be arbitrary large).
Revised: In this work, our ﬁrst contribution is to propose an efﬁcient weight estimator for which we can obtain good statistical guarantees without a requirement on the problem-dependent minimum sample complexity as necessary for BBSL , which in their case can be arbitrarily large for small number of samples. We then derive a dimension-independent generalization bound for the ﬁnal Regularized

Original: We further improve the  estimation error in the ﬁnite-sample setting by a factor of k log( k ) over the prior work, where k is the number of classes. We deploy this estimator and propose a label-shift correcting estimator and provide generalization guarantees.
Revised: Learning under Label Shift ( RLLS ) classiﬁer trained on the reweighted samples. Our method improves the weight estimation error and the excess risk of the classiﬁer on reweighted samples by a factor of k log( k ) , where k is the number of classes, i.e. the cardinality of Y .

Original: Similar to Lipton et al.
Revised: It also follows from our analysis that explicitly controlling the inﬂuence of our regularized weight estimates could help in practice.

Original: (2018), we ﬁrst use a black-box classiﬁer, which can be biased, uncalibrated and inaccurate, to estimate importance weights for the source samples. Depending on the number ofsource and target samples, we regularize the inﬂuence of these estimated weights and train a corrected classiﬁer on the source set with the estimated weights. We then derive a dimension-independent generalization bound for the ﬁnal Regularized Learning under Label Shift ( RLLS ) classiﬁer.
Revised: In particular, it suggests that when the number of target samples is below a certain threshold, the classiﬁer would obtain higher accuracy if the weight estimates are not trusted because of high uncertainty and the samples stay unweighted.

Original: We empirically study the performance of RLLS and show weight estimation as well as prediction accuracy for a variety of shifts, including shifting source data and shifting target data with various parameters, different sample sizes and regularization parameters.
Revised: We empirically study the performance of RLLS and show weight estimation as well as prediction accuracy for a variety of shifts on the CIFAR-10 and MNIST datasets, including shifting source data and shifting target data with various parameters, different sample sizes and regularization parameters.

Original: We achieve an order of magnitude smaller weight estimation error than baseline methods in large shift cases and enjoy 20% higher accuracy or F-1 score in corresponding predictive tasks.
Revised: We achieve an order of magnitude smaller weight estimation error than baseline methods in large shift cases and enjoy 20% higher accuracy and F-1 score in corresponding predictive tasks.

Original: We also see at least 10% accuracy improvement using our regularized estimator when faced with small target sample sizes.
Revised: We also see at least 10% accuracy improvement using our regularized estimator when faced with small target sample sizes.

Original:  R EGULARIZED LEARNING OF LABEL SHIFTS ( RLLS )
Revised:  R EGULARIZED LEARNING OF LABEL SHIFTS ( RLLS )

Original: Let usformally deﬁne the short hand for the marginal probability mass functions of Y  with respect to P , Q as p, q : [ k ] → [0 , 1] with p ( i )
Revised: Formally let us the short hand for the marginal probability mass functions of Y on ﬁnite Y with respect to P , Q as p, q : [ k ] → [0 , 1] with p ( i )

Original: =
Revised: =

Original: P ( Y = i ) , and q ( i ) =
Revised: P ( Y = i ) , and q ( i ) =

Original: Q ( Y = i ) for all i ∈ [ k ] , representable by vectors in R k + which sum to one.
Revised: Q ( Y = i ) for all i ∈ [ k ] , representable by vectors in R k + which sum to one.

Original: In the label shift setting, we deﬁne the importance weight vector w ∈ R k between these two domains as w ( i ) =
Revised: In the label shift setting, we deﬁne the importance weight vector w ∈ R k between these two domains as w ( i ) =

Original: q ( i ) p ( i ) .
Revised: q ( i ) p ( i ) .

Original: We quantify the shift using the exponent of the inﬁnite and second order Renyi divergence as follows
Revised: We quantify the shift using the exponent of the inﬁnite and second order Renyi divergence as follows

Original: [Equation]
Revised: [Equation]

Original: Given a hypothesis class H , a loss function (cid:96) : Y × Y → [0 , 1]  our aim is to ﬁnd the hypothesis h ∈ H which minimizes
Revised: Given a hypothesis class H and a loss function (cid:96) : Y × Y → [0 , 1] , our aim is to ﬁnd the hypothesis h ∈ H which minimizes

Original: [Equation]
Revised: [Equation]

Original: In the usual ﬁnite sample setting however, L unknown and we are given samples { ( x j , y j ) } nj =1 from P instead.
Revised: In the usual ﬁnite sample setting however, L unknown and we observe samples { ( x j , y j ) } nj =1 from P instead.

Original: If we are given the vector of importance weights w we could then minimize the empirical loss with importance weighted samples deﬁned as
Revised: If we are given the vector of importance weights w we could then minimize the empirical loss with importance weighted samples deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where n is the number of available observations drawn from P used to learn the classiﬁer h .
Revised: where n is the number of available observations drawn from P used to learn the classiﬁer h .

Original: As w is unknown in practice, we have to ﬁnd the minimizer of the empirical loss with estimated importance weights
Revised: As w is unknown in practice, we have to ﬁnd the minimizer of the empirical loss with estimated importance weights

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) w are estimates of w .
Revised: where (cid:98) w are estimates of w .

Original: Given a set D p of n p samples from the source distribution P , we ﬁrst divide the data set into two  where we use (1 − β ) n p samples in set D weight p to compute the estimate (cid:98) w and the remaining n = βn p in the set D class p to ﬁnd the classiﬁer which minimizes the loss (1)
Revised: Given a set D p of n p samples from the source distribution P , we ﬁrst divide it into two sets where we use (1 − β ) n p samples in set D weight p to compute the estimate (cid:98) w and the remaining n = βn p in the set D class p to ﬁnd the classiﬁer which minimizes the loss (1), i.e.

Original: , i.e. (cid:98) h [Equation] = arg 
Revised: (cid:98) h (cid:98) w = arg min

Original: [Equation] h [Equation] w ) .
Revised: h ∈H L n ( h ; (cid:98) w ) .

Original: In the following, we describe how to estimate the weights (cid:98) w and provide guarantees for the resulting estimator (cid:98) h (cid:98) w .
Revised: In the following, we describe how to estimate the weights (cid:98) w and provide guarantees for the resulting estimator (cid:98) h (cid:98) w .

Original: Plug-in weight estimation
Revised: Plug-in weight estimation

Original: The following simple correlation between the label distributions p, q was noted in Lipton et al.
Revised: The following simple correlation between the label distributions p, q was noted in Lipton et al.

Original: (2018): for a ﬁxed hypothesis, if for all y ∈ Y it holds that q ( y ) ≥ 

Revised: (2018): for a ﬁxed hypothesis h , if for all y ∈ Y it holds that q ( y ) ≥

Original: = ⇒ p
Revised: = ⇒ p

Original: ( y ) ≥ 0( ∗ ) , we have
Revised: ( y ) ≥ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: for all i, j ∈ Y , which can equivalently be written in matrix vector notation(by slight abuse of notation) as
Revised: for all i, j ∈ Y . This can equivalently be written in matrix vector notation as

Original: [Equation]
Revised: [Equation]

Original: where C h is the confusion matrix, that is [ C h ] i,j = P ( h ( X )
Revised: where C h is the confusion matrix with [ C h ] i,j = p ( h ( X )

Original: =
Revised: =

Original: i, Y = j ) and q h is the vector which represents the probability mass function of h ( X ) under distribution Q .
Revised: i, Y = j ) and q h is the vector which represents the probability mass function of h ( X ) under distribution Q .

Original: The assumption ( ∗ )  is a reasonable condition since without any prior knowledge, there is no way to properly reason about a class in the target domain that is not represented in the source domain.
Revised: The requirement q ( y ) ≥ 0 = ⇒ p ( y ) ≥ 0 is a reasonable condition since without any prior knowledge, there is no way to properly reason about a class in the target domain that is not represented in the source domain.

Original: In reality, both q h and C h can only be estimated by the corresponding ﬁnite sample averages (cid:98) q h , (cid:98) C h .
Revised: In reality, both q h and C h can only be estimated by the corresponding ﬁnite sample averages (cid:98) q h , (cid:98) C h .

Original: Lipton et al.
Revised: Lipton et al.

Original: (2018) simply compute the inverse of the estimated confusion matrix (cid:98) C h in order to estimate the importance weight, i.e. (cid:98)
Revised: (2018) simply compute the inverse of the estimated confusion matrix (cid:98) C h in order to estimate the importance weight, i.e. (cid:98)

Original: w = (cid:98) C − 1 h (cid:98) q h .
Revised: w = (cid:98) C − 1 h (cid:98) q h .

Original: While C − 1 h (cid:98) q h is a statistically efﬁcient estimator, (cid:98) w with estimated (cid:98) C − 1 h can be arbitrarily bad  especially for small sample sizes and small minimum singular values.
Revised: While C − 1 h (cid:98) q h is a statistically efﬁcient estimator, (cid:98) w with estimated (cid:98) C − 1 h can be arbitrarily bad since (cid:98) C − 1 h can be arbitrary close to a singular matrix especially for small sample sizes and small minimum singular value of the confusion matrix.

Original: Intuitively, when there are very few samples, the weight estimation will have high variance in which case it might be better to avoid importance weighting altogether.
Revised: Intuitively, when there are very few samples, the weight estimation will have high variance in which case it might be better to avoid importance weighting altogether.

Original: Furthermore, even when the sample complexity in Lipton et al.
Revised: Furthermore, even when the sample complexity in Lipton et al.

Original: (2018), unknown in practice, is met, the resulting error of this estimator is linear in k which is problematic for large k .
Revised: (2018), unknown in practice, is met, the resulting error of this estimator is linear in k which is problematic for large k .

Original: We therefore aim to address these shortcomings by proposing the following two-step procedure to compute importance weights:
Revised: We therefore aim to address these shortcomings by proposing the following two-step procedure to compute importance weights.

Revised: In the case of no shift we have w = 1 so that we deﬁne the amount of weight shift as θ = w − 1 .

Original: Given a “decent” black box estimator which we denote by h 0 , we 
Revised: Given a “decent” black box estimator which we denote by h 0 , we make the ﬁnal classiﬁer less sensitive to the estimation performance of C (i.e. regularize the weight estimate) by

Original: calculate the measurement error adjusted (cid:98) θ as described in Section .1 for h2.
Revised: calculating the measurement error adjusted (cid:98) θ (described in Section 2.1 for h 0 ) and 2.

Original: compute the regularized weight (cid:98) w =1 + λ (cid:98) θ where λ depends on the sample size (1 − β ) n p  By "decent" we refer to a classiﬁer h 0 which yields a full rank confusion matrix C h [-
-]
Revised: computing the regularized weight (cid:98) w = + λ (cid:98) θ where λ depends on the sample size (1 − β ) n p . By "decent" we refer to a classiﬁer h 0 which yields a full rank confusion matrix C h [-
-]0 .

Original: An example forsuch a  classiﬁer h 0 is one that always outputs a ﬁxed class.
Revised: A trivial example for a non-”decent” classiﬁer h 0 is one that always outputs a ﬁxed class.

Original: As it does not capture any characteristics of thethe data, there is no hope to gain any statistical  information.
Revised: As it does not capture any characteristics of the data, there is no hope to gain any statistical information without any prior information.

Original: 2. E STIMATOR CORRECTING FOR FINITE SAMPLE ERRORS
Revised: 2. E STIMATOR CORRECTING FOR FINITE SAMPLE ERRORS

Original: Both the confusion matrix C h 0 and the label distribution q h 0 on the target for the black box hypothesis h 0 are unknown and we are instead only gave access to ﬁnite sample estimates (cid:98) C h 0 , (cid:98)
Revised: Both the confusion matrix C h 0 and the label distribution q h 0 on the target for the black box hypothesis h 0 are unknown and we are instead only given access to ﬁnite sample estimates (cid:98) C h 0 , (cid:98)

Original: q h 0 .
Revised: q h 0 .

Original: In what follows all empirical and population confusion matrices as well as label distributions are deﬁned with respect to the hypothesis h
Revised: In what follows all empirical and population confusion matrices, as well as label distributions, are deﬁned with respect to the hypothesis h

Original: = h 0 .
Revised: = h 0 .

Original: For notational simplicity, we thus drop the subscript h 0 in what follows.
Revised: For notation simplicity, we thus drop the subscript h 0 in what follows.

Original: In the unweighted case we have w = 1 so that deﬁne the amount of weight shift as θ = w − 1 .

Original: The reparameterized linear model (2) with respect to θ then reads
Revised: The reparameterized linear model (2) with respect to θ then reads

Original: [Equation]
Revised: [Equation]

Original: with corresponding ﬁnite sample quantity (cid:98) b = (cid:98) q − (cid:98) C 1 .
Revised: with corresponding ﬁnite sample quantity (cid:98) b = (cid:98) q − (cid:98) C 1 .

Original: When (cid:98) C is near singular, the estimation of θ becomes unstable.
Revised: When (cid:98) C is near singular, the estimation of θ becomes unstable.

Original: Furthermore, large values in the true shift θ result in large variances.
Revised: Furthermore, large values in the true shift θ result in large variances.

Original: We address this problem by adding a regularizing (cid:96) 2 penalty term to the usual loss and thus push the amount of shift towards 0 , a method that has been proposed in (Pires & Szepesvári, 2012).
Revised: We address this problem by adding a regularizing (cid:96) 2 penalty term to the usual loss and thus push the amount of shift towards 0 , a method that has been proposed in (Pires & Szepesvári, 2012).

Original: In particular, we compute
Revised: In particular, we compute

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) θ ρ is deﬁned by
Revised: where (cid:98) θ γ is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: The proof can be found in Section A.1. Here, ∆ C , ∆ b are parameters which will eventually be high probability upper bounds for (cid:107) (cid:98) C − C (cid:107) 2 and (cid:107) (cid:98) b − b (cid:107) 2 respectively.
Revised: Here, ∆ C , ∆ b are parameters which will eventually be high probability upper bounds for (cid:107) (cid:98) C − C (cid:107)and (cid:107) (cid:98) b − b (cid:107) 2 respectively, where (cid:107) · (cid:107) 2 denotes the spectral norm.

Original: Let σ min be the minimum singular value of the true confusion matrix C .
Revised: Let σ min be the minimum singular value of the true confusion matrix C .

Original: For the estimate (cid:98) θ , we then have the following upper bound
Revised: For the estimate (cid:98) θ , we then have the following upper bound

Original: Lemma 1 For (cid:98) θ as deﬁned in equation (3) , we have with probability at least 1 − δ that (cid:107) (cid:98) θ − θ (cid:107) 2 ≤  O
Revised: Lemma 1 For (cid:98) θ as deﬁned in equation (3) , we have with probability at least 1 − δ that(cid:107) (cid:98) θ − θ (cid:107) 2 ≤ (cid:15) θ ( n p , n q , (cid:107) θ (cid:107) 2 , δ ) : = O

Original: [Equation]
Revised: [Equation]

Original: where δ < 0 .
Revised: The proof of this lemma can be found in Appendix A.1.

Original: A couple of remarks are in order at this point.
Revised: A couple of remarks are in order at this point.

Revised: First of all, notice that the weight estimation procedure (3) does not require a minimum sample complexity which is in the order of σ − 2 min to obtain the guarantees for BBSL .

Revised: This is due to the fact that errors in the covariates are accounted for.

Original: First of all, the upper bound in Lemma 1 includes σ min ( C ) as a parameter, apart from the norm of the true weight shift θ .
Revised: In order to directly see the improvements in the upper bound of Lemma 1 compared to Theorem 3 in Lipton et al. (2018), ﬁrst observe that in order to obtain their upper bound with a probability of at least 1 − δ , it is necessary that [Equation] ≤ δ .

Revised: As a consequence, the upper bound in Theorem 3 of Lipton et al.

Revised: (2018) is bigger than

Revised: [Equation]

Original: Recall that these quantities are all heavily dependent on the  black box estimator h 0 that we are given a priori.
Revised: Thus Lemma 1 improves upon the previous upper bound by a factor of k . Furthermore, as in Lipton et al. (2018), this result holds for any black box estimator h 0 which enters the bound via σ min ( C h 0 ) . We can directly see how a good choice of h 0 helps to decrease the upper bound in Lemma 1.

Original: In particular, if h 0 is an ideal estimator, and the source set is balanced, C is the unit matrix with σ min = 1 /k .
Revised: In particular, if h 0 is an ideal estimator, and the source set is balanced, C is the unit matrix with σ min = 1 /k .

Original: In contrast, when the model h 0 is uncertain, the singular value σ min is close to zero.
Revised: In contrast, when the model h 0 is uncertain, the singular value σ min is close to zero.

Original: For least square problemsunder the Gaussian assumption and with  measurement errors in the matrix , it is standard to use regularized total least squares approaches which requires a single value decomposition.
Revised: Moreover, for least square problems with Gaussian measurement errors in both input and target variables, it is standard to use regularized total least squares approaches which requires a singular [Equation] [Equation]

Revised: [Equation]

Revised: value decomposition.

Original: Our choice for the alternative estimator in Eq.
Revised: Finally, our choice for the alternative estimator in Eq.

Original: 3 with norm instead of norm squared regularization is motivated by the label shift setting, where true shifts θ witha large  norm may shrink the estimate (cid:98) θ too much, away from the true θ .
Revised: 3 with norm instead of norm squared regularization is motivated by the cases with large shifts θ , where using the squared norm may shrink the estimate (cid:98) θ too much and away from the true θ .

Original: 2. R EGULARIZED ESTIMATOR AND GENERALIZATION BOUND
Revised: 2. R EGULARIZED ESTIMATOR AND GENERALIZATION BOUND

Original: When the number of samples in the target set is very small or the label shift is mild, the estimated weights might be too uncertain to be applied.
Revised: When a few samples from the target set are available or the label shift is mild, the estimated weights might be too uncertain to be applied.

Original: We therefore propose a regularized estimator deﬁned as follows
Revised: We therefore propose a regularized estimator deﬁned as follows

Original: [Equation]
Revised: [Equation]

Original: Note that (cid:98) w implicitly depends on λ, β .
Revised: Note that (cid:98) w implicitly depends on λ , and β .

Original: By rewriting (cid:98) w
Revised: By rewriting (cid:98) w

Original: =
Revised: =

Original: (1 − λ ) 

Revised: (1 − λ )

Original: + λ ( 1 + (cid:98) θ ) , we see that intuitively λshould be closer to 1 the more reason there is to believe that 1 + (cid:98) θ is in fact the true weight.
Revised: + λ ( 1 + (cid:98) θ ) , we see that intuitively λ closer to 1 the more reason there is to believe that 1 + (cid:98) θ is in fact the true weight.

Original: Let us deﬁne the set G ( (cid:96), H ) 
Revised: Deﬁne the set G ( (cid:96), H ) =

Original: = { g h ( x, y ) =
Revised: { g h ( x, y ) =

Original: w ( y ) (cid:96) ( h ( x ) , y ) : h ∈ H}for a loss function (cid:96) with (cid:107) (cid:96) (cid:107) ∞ ≤ 1 and hypothesis class H and the commonly used Rademacher complexity measure
Revised: w ( y ) (cid:96) ( h ( x ) , y ) : h ∈ H} and its Rademacher complexity measure

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] [Equation] (see e.g. Bartlett & Mendelson (2002)).
Revised: with ξ i , ∀ i as the Rademacher random variables (see e.g. Bartlett & Mendelson (2002)).

Original: We can now state a generalization bound for the classiﬁer (cid:98) h (cid:98) w in a general hypothesis class H , which is trained on source data with the estimated weights deﬁned in equation (4).
Revised: We can now state a generalization bound for the classiﬁer (cid:98) h (cid:98) w in a general hypothesis class H , which is trained on source data with the estimated weights deﬁned in equation (4).

Original: Theorem 1 (Generalization bound for (cid:98) h (cid:98) w ) Given n p samples from the source data set and n q samples from the target set, a hypothesis class H and loss function (cid:96) , the following upper bound |L ( (cid:98) h (cid:98)
Revised: Theorem 1 (Generalization bound for (cid:98) h (cid:98) w ) Given n p samples from the source data set and n q samples from the target set, a hypothesis class H and loss function (cid:96) , for (cid:15) G ( n p , δ ) : = 2 R n ( G ) + d ∞ ( q || p ) (cid:113) log(2 /δ )

Original: [Equation]
Revised: βn p

Original:  holds with probability at least 1 − δwhere δ ≤ 0 .
Revised: the following upper bound holds with probability at least 1 − δ

Original: The proof can be found in Section A.2 as well a tighter bound of O ( d ∞ ( q || p ) n q +

Original: [Equation]
Revised: [Equation]

Original: d
Revised: The proof can be found in Appendix A.4.

Original: [Equation] q || p ) n q ) in terms of dependence on the quantities d ( q || p ) for ﬁnite hypothesis classes in Section A.3.
Revised: Additionally, we derive the analysis also for ﬁnite hypothesis classes in Appendix A.5 to provide more insight into the proof of general hypothesis classes.

Original: The size of R n ( G ) is determined by the structure of the function class H and the loss (cid:96) .
Revised: The size of R n ( G ) is determined by the structure of the function class H and the loss (cid:96) .

Original: For example for the 0 / 1 loss, the VC dimension of H can be deployed to upper bound the Rademacher complexity.
Revised: For example for the 0 / 1 loss, the VC dimension of H can be deployed to upper bound the Rademacher complexity.

Original: In order to use Theorem 1 in practice and exploit the possibility of choosing parameters λ, β depending on the sample sizes, we need to assume some prior knowledge about the shift θ and σ min of the possible shifts we might encounter.
Revised: The bound (5) in Theorem 1 holds for all choices of λ .

Original: In particular, we assume that it is known that the label shift is not too big, i.e.  (cid:107) θ (cid:107) 2 ≤ θ max (on the other hand it is unreasonable to have knowledge about the shift estimator, i.e. σ min ) and we consider the following upper boundinstead for practical implementation
Revised: In order to exploit the possibility of choosing λ and β to have an improved accuracy depending on the sample sizes, we ﬁrst let the user deﬁne a set of shifts θ against which we want to be robust against, i.e. all shifts with (cid:107) θ (cid:107) 2 ≤ θ max . For these shifts, we obtain the following upper bound

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: The bound in equation (6) suggests using Algorithm 1 as our ultimate label shift correction procedure. where for step 2 of the algorithm, we choose λ (cid:63) = 1 whenever n q ≥

Original: [Equation]
Revised: [Equation]

Revised: hereby neglecting the log factors and thus dependencies on k ) and 0 else.

Revised: When using this rule, we obtain

Original: [Figure]
Revised: [Equation]

Original: The bound in equation (6) suggests to usealgorithm 1, in particular when only very few target samples have been observed before an update of  the classiﬁer is necessary. The bound (6) gives us guidance for a good choice of λ as a function of the number of samples in practice.
Revised: Notice that in practice, we do not know σ min in advance so that in Algorithm 1 we need to use an estimate of σ min , which could e.g. be the minimum eigenvalue of the empirical confusion matrix (cid:98) C with an additional computational complexity of at most O ( k 3 ) .

Original: Whenever n q ≥
Revised: 1.0 θ max 10101010n q σ

Original: [Figure]
Revised: [Figure]

Original: [Equation] hereby neglecting the log factors), the best choice is λ = 1  and0 else. In Figure 2.2 we show curves of ﬁxed σ min in parameter space, under which λ should be chosen as 0 and above which it should be 1 .
Revised: Figure 1 shows how the oracle thresholds vary with n q and σ min when n p is kept ﬁx. When the parameters are above the curves for ﬁxed n p , λ should be chosen as 1 otherwise the samples should be unweighted, i.e. λ = 0 .

Original: When the confusion matrix has small singular values,we see that the estimated weights should only be trusted for rather high n q and high believed shifts θ max .
Revised: This ﬁgure illustrates that when the confusion matrix has small singular values, the estimated weights should only be trusted for rather high n q and high believed shifts θ max .

Original: However, since θ max is just an upper bound on the true amount of shift (cid:107) θ(cid:107) 2 , when 
Revised: Although the overall statistical rate of the excess risk of the classiﬁer does not change as a function of the sample sizes, θ max could be signiﬁcantly smaller than (cid:15) θ when σ min is very small and thus the accuracy in this regime could improve.

Original: [Equation] λ should in fact ideally be0 . Thus for n q that are a little bit below the threshold (depending on the certainty of the belief on θ max ), it could be sensible to use an intermediate value λ ∈ (0 , 1) which we explore in the experimental section 3.3.
Revised: Indeed we observe this to be the case empirically in Section 3.3.

Original:  EXPERIMENTS
Revised:  EXPERIMENTS

Original: In this section we illustrate our theoretical bounds by runningour RLLSestimation procedure on a variety of artiﬁcially generated shifts on the MNIST (LeCun & Cortes, 2010) and CIFAR (Krizhevsky & Hinton, 2009) datasets. 
Revised: In this section we illustrate the theoretical analysis by running RLLS on a variety of artiﬁcially generated shifts on the MNIST (LeCun & Cortes, 2010) and CIFAR10 (Krizhevsky & Hinton, 2009) datasets. We ﬁrst randomly separate the entire dataset into two sets (source and target pool) of the same size.

Revised: Then we sample, unless speciﬁed otherwise, the same number of data points from each pool to form the source and target set respectively.

Revised: We chose to have equal sample sizes to allow for fair comparisons across shifts.

Revised: There are various kinds of shifts which we consider in our experiments.

Revised: In general we assume one of the source or target datasets to have uniform distribution over the labels.

Original: In order to ﬁnd (cid:98) ω
Revised: Within the non-uniform set, we consider two types of sampling strategies in the main text: the Minority-Class Shift refers to the case where we set a ﬁxed number of classes m to have probability p < 0 . 1 , while the distribution over the rest of the classes is uniform. For the Dirichlet shift , we draw a probability vector p from the Dirichlet distribution with concentration parameter set to α for all classes, before including sample points which correspond to the multinomial label variable according to p .

Revised: Results for the tweak-one shift strategy as in Lipton et al.

Revised: (2018) can be found in Section A.6.1.

Revised: After artiﬁcially shifting the label distribution in one of the source and target sets, we then follow algorithm 1, where we choose the black box predictor h 0 to be a two-layer fully connected neural network trained on the source dataset.

Revised: Note that any black box predictor could be employed here, though the higher the accuracy, the more likely weight estimation will be precise.

Revised: In order to compute (cid:98) ω

Original: = 1 + (cid:98) θ in Eq.
Revised: = 1 + (cid:98) θ in Eq.

Original: (3), performing a grid search over ρ on the entire set of natural numbers is not feasible.
Revised: (3), performing a grid search over γ on the entire set of natural numbers is not feasible.

Original: Instead, we call a built-in solver to directly solve the low dimensional problem min θ (cid:107) (cid:98)
Revised: Instead, we call a built-in solver to directly solve the low dimensional problem min θ (cid:107) (cid:98)

Original: Cθ − (cid:98) b (cid:107) 22 + ρ 3∆ C (cid:107) θ (cid:107) 2 where the choice of ρ = 0 .
Revised: Cθ − (cid:98) b (cid:107) 2 + γ 3∆ C (cid:107) θ (cid:107) 2 where the choice of γ = 0 .

Original: 01 is observed to yield a low loss in the ﬁrst line of Eq.
Revised: 01 is observed to yield a low loss in the ﬁrst line of Eq.

Original: (3) on various levels of label shift.
Revised: (3) on various levels of label shift pre-computed beforehand.

Original: We follow three different kinds of sampling strategies to generate a label shift between the source and target set: Tweak-one Shift , Dirichlet shift , and Minority-class shift . For the Tweak-one Shift , we randomly choose one class, e.g. i and set p ( y = i ) =
Revised: We thus treat it as a hyperparameter that can be chosen using standard cross validation methods.

Original: ρ while all other classes are distributed evenly. For the Dirichlet shift , we draw a probability vector p from the Dirichlet distribution with concentration parameter set to α for all classes, before including sample points which correspond to the multinomial label variable according to p . In Minority-Class Shift ,  we set a certain number of classes to have probability p = 0 .
Revised: Finally, we train a classiﬁer on the source samples weighted by (cid:98) ω , where we use a two-layer fully connected neural network for MNIST and a ResNet-18 (He et al., 2016) for CIFAR10.

Original: 001 , while the distribution over the rest of the classes is uniform. One of these sampling strategies is used to sample either the source or the target set, while the other set is sampled witha uniform distribution. After artiﬁcially shifting the label distribution in one of the source and target sets, we then follow algorithm 1, where we choose the black box predictor h 0 to be a two-layer fully connected neural network trained on the (unweighted) source dataset.
Revised: We sample 20 datasets with the label distributions for each shift parameter.

Original: We then evaluate the  mean square estimation error (MSE)  E (cid:107) (cid:98) w − w (cid:107) 2 ,{+
 and the+} predictive accuracy on the target set as well as F-1 score, 2( precision ∗ recall ) /
Revised: to evaluate the empirical mean square estimation error (MSE) and variance of the estimated weights E (cid:107) (cid:98) w − w (cid:107) {+
 and the+} predictive accuracy on the target set.

Original: ( precision + recall ) , and compare them with theperformance of the black box shift estimation method (BBSE) in Lipton et al.
Revised: We use these measures to compare our procedure with the black box shift learning method (BBSL) in Lipton et al.

Original: Here precision is the the proportion of true predicted labels in all the predicted labels, while recall is the proportion of true predicted labels in all the true labels.

Original: Theoretically, KMM  (Zhang et al., 2013) is also a possible baseline to compare with. However, as mentioned in Lipton et al. (2018), it is not scalable to large sample sizes for weight estimation.
Revised: Notice that although KMM methods (Zhang et al., 2013) would be another standard baseline to compare with, it is not scalable to large sample size regimes for n p , n q above n = 8000 as mentioned by Lipton et al.

Original: 3. W EIGHT E STIMATION C OMPARISON
Revised: 3. W EIGHT E STIMATION AND PREDICTIVE PERFORMANCE FOR SOURCE SHIFT

Original: In this set of experiments on the CIFAR10 dataset, we illustrate our weight estimation  performance using a simple two-layer neural network as the black box estimator h 0 by comparing it with BBSE from Lipton et al.
Revised: In this set of experiments on the MNIST dataset, we illustrate our weight estimation and prediction performance for Minority-Class source shifts and compare it with BBSL .

Original: The precise procedure to artiﬁcially create a particular shift is as follows: we sample target data uniformly and sample data points from the source set according to a probability vector p ( y ) that corresponds to different shift parameters ρ in tweak-one Shift and α in Dirichlet Shift for 20 times.

Original: Hereby, we ﬁx the number of  source and target data points to 10000 for the Dirichlet shift and tweak-one shift for ρ ≤ 0 .
Revised: For this set of experiments, we set the number of data points in both source and target set to 1000 and sample from the two pools without replacement.

Revised: Figure 2 illustrates the weight estimation alongside ﬁnal classiﬁcation performance for Minority-Class source shift of MNIST.

Original: For bigger shifts, i.e. ρ ≥ 0 .5 , we keep the maximum possible number of samples.
Revised: We created large shifts of three or more minority classes with p = 0 .

Original: In Figure 2, we compare the proposed RLLS weight estimation to the BBSE from (Lipton et al., 2018).
Revised: We use a ﬁxed black-box classiﬁer that is trained on biased source data, with tweak-one ρ = 0 . 5 .

Original: When the shift is small ( ρ is small), the two methods perform similarly, while RLLS outperforms BBSE dramatically when the shift is large. 3. P REDICTIVE T ASKS WITH LARGE SAMPLE SIZE
Revised: Observe that the MSE in weight estimation is relatively large and RLLS outperforms BBSL as the number of minority classes increases. As the shift increases the performance for all methods deteriorates.

Original: In this section, we compare the predictive performances in terms of accuracy and F-1 score between a classiﬁer trained on unweighted source data and the classiﬁers trained on weighted loss obtained by using the RLLS  and BBSE procedure (which is called BBSC).
Revised: Furthermore, Figure 2 (b) illustrates how the advantage of RLLS over the unweighted classiﬁer increases as the shift increases. Across all shifts, the RLLS based classiﬁer yields higher accuracy than the one based on BBSL . Results for smaller and larger values of p can be found in Section A.6.2.

Original: [Figure]
Revised: [Figure]

Original: When n q is large, λ is set to one andthus the intermediate weight estimate (cid:98) θ is "fully" used, i.e. corresponds to (cid:98) w = 1 + (cid:98) θ . In this case, better weight estimation should directly lead to better performance of the estimator.
Revised: In this section, we compare the predictive performances between a classiﬁer trained on unweighted source data and the classiﬁers trained on weighted loss obtained by the RLLS and BBSL procedure on CIFAR10. The target set is shifted using the Dirichlet shift with parameters α =

Original: One subtlety arises when we compare shifts on the target vs. shifts on the source domain.
Revised: The number of data points in both source and target set is 10000 .

Original: When the  target is seriously biased for smaller α , the predictive task becomes rather easy, such that even the majority class vote would give high accuracy, though it would have zero accuracy on all but one class.
Revised: In the case of target shifts, larger shifts actually make the predictive task easier, such that even a constant majority class vote would give high accuracy. However it would have zero accuracy on all but one class.

Original: Therefore,metrics that account for class imbalance and other types of errors in confusion matrix, like the F-1 score, are in order to allow for a more comprehensive performance evaluation for the prediction. Figure 3 depicts the MSE of the weight estimation (a), the corresponding performance comparison on accuracy (b) and F-1 score (c) when the target data on CIFAR10 is shifted using the Dirichlet
Revised: Therefore, in order to allow for a more comprehensive performance performance between the methods, we also compute the macro-averaged F-1 score by averaging the per-class quantity 2( precision · recall ) /

Original: Shift. We use the Dirichlet parameters α = [0 . 01 , 0 . 1 , 1 , 10] to bias the target set and run 20 different experiments for each parameter. We use the ResNet-18 (He et al., 2016) deep neural net as our hypotheis class H .
Revised: ( precision + recall ) over all classes.

Original: We can see that in the large shift case when α = 0 . 01 ,  the F-1 score for BBSC and unweighted classiﬁer can be very bad while the accuracy is still high.
Revised: For a class i , precision is the percentage of correct predictions among all samples predicted to have label i , while recall is the proportion of correctly predicted labels over the number of samples with true label i .

Original: This is due to classiﬁers making wrong predictions on minority classes in the target data.
Revised: This measure gives higher weight to the accuracies of minority classes which have no effect on the total accuracy.

Original: [Figure]
Revised: [Figure]

Original: A harder problem which does not allow "cheating" is when the shift is in fact on the source set and the target set has uniform label class distribution. Figure 4 shows results using Minority-Class shifted source data and uniform target data from MNIST. For this set of experiments, we ﬁrst randomly separate the original dataset into source and target before randomly sampling 5000 data points from the target set. The number ofminority class ranges from 1 to 4 to bias the source set and rundifferent experiments for each parameter.
Revised: Figure 3 depicts the MSE of the weight estimation (a), the corresponding performance comparison on accuracy (b) and F-1 score (c).

Original: To better see the difference between methods, we create relatively large shift with one or more class of data being minority. We also use a ﬁxed black-box classiﬁer that is trained on biased source data. We use two-layer fully connected neural networks for both weight estimation and predictive tasks.
Revised: Recall that the accuracy performance for low shifts is not comparable with standard CIFAR10 benchmark results because of the overall lower sample size chosen for the comparability between shifts.

Original: We can see thatthe MSE in weight estimation is relatively large.
Revised: We can see that in the large target shift case for α = 0 .

Original: As the error goes up, the performance for all methods goes down, except for BBSC approaching RLLS due to similar weight estimation in the extreme case.
Revised: 01 , the Fscore for BBSL and the unweighted classiﬁer is rather low compared to RLLS while the accuracy is high.

Original: Moreover, RLLS always outperforms BBSC whenever the weight estimation error is lower. Notice that generalizing from an imbalanced training set is a common data science task where reweighting the samples in  the source set is also standard practice (and it isa priori assumed that the  target set is uniform). [Figure]
Revised: As mentioned before, the reason for this observation and why in Figure 3 (b) the accuracy is higher when the shift is larger, is that the predictive task actually becomes easier with higher shift. In conclusion, our proposed RLLS procedure yields better weight estimation and predictive performance for a whole range of source and target shifts, while the improvement is bigger for bigger shifts.

Original: 3. L ABEL S HIFT P REDICTIVE T ASKS IN THE LOW SAMPLE REGIME 
Revised: 3. R EGULARIZED WEIGHTS IN THE LOW SAMPLE REGIME FOR SOURCE SHIFT

Original: As discussed in Section 2.2, λ in equation (4) may be chosen according to the believed upper and lower bounds on θ max , σ min .

Original: In the following, we presentsimulation results of our procedure in the low sample regime in Figure 5 where we choose different values ofλ and vary the number of target samples n q  .
Revised: In the following, we present the average accuracy of RLLS in Figure 4 as a function of the number of target samples n q for different values of λ for small n q .

Original: Here we ﬁx the sample size in  source as 10000 and investigate a Dirichlet Shift with ﬁxed α = 0 .
Revised: Here we ﬁx the sample size in the source set to n p = 1000 and investigate a Minority-Class source shift with ﬁxed p = 0 .

Original: 5 on the source data.
Revised: 01 and ﬁve minority classes. A motivation to use intermediate λ is discussed in Section 2.2, as λ in equation (4) may be chosen according to θ max , σ min . In practice, since θ max is just an upper bound on the true amount of shift (cid:107) θ (cid:107) 2 , in some cases λ should in fact ideally be 0 when [Figure]

Original:  Figure 5 suggests that our method with λ ∈ (0 , 1) outperforms the fully weighted case, i.e. λ = 1 , for small sample sizes.
Revised: Thus for target sample sizes n q that are a little bit above the threshold (depending on the certainty of the belief how close to θ max the norm of the shift is believed to be), it could be sensible to use an intermediate value λ ∈ (0 , 1) . [Figure] Figure 4 suggests that unweighted samples (red) yield the best classiﬁer for very few samples n q , while for 10 ≤ n q ≤ 500 an intermediate λ ∈ (0 , 1) (purple) has the highest accuracy and for n q > 1000 , the weight estimation is certain enough for the fully weighted classiﬁer (yellow) to have the best performance (see also the corresponding data points in Figure 2). The unweighted BBSL classiﬁer is also shown for completeness.

Original: This shows that we are able to adjust to  uncertainty on importance weights and generalize well for a varying range of target sample sizes.
Revised: We can conclude that regularizing the inﬂuence of the estimated weights allows us to adjust to the uncertainty on importance weights and generalize well for a wide range of target sample sizes.

Original: We also show a comparison of using different  black-box predictors h 0 for weight estimation and observe that fully weighted methods with λ = 1 achieve the best performance faster with a better trained black-box classiﬁer (a), while it takes longer for it to improve with  corrupted one (c).Here we train the black-box classiﬁer with full training data (a), tweak-one with ρ = 0 . 2 (b) and tweak-one with ρ = 0 .
Revised: Furthermore, the different plots in Figure 4 correspond to black-box predictors h 0 for weight estimation which are trained on more or less corrupted data, i.e. have a better or worse conditioned confusion matrix. The fully weighted methods with λ = 1 achieve the best performance faster with a better trained black-box classiﬁer (a), while it takes longer for it to improve with a corrupted one (c).

Original: 6 (c) to obtain different predictors.

Original: This reﬂects the relation between eigenvalue of confusion matrix σ min and target sample size n q in Theorem 1.
Revised: Furthermore, this reﬂects the relation between eigenvalue of confusion matrix σ min and target sample size n q in Theorem 1.

Original: In other words, we need more samples from the target data to compensate a bad predictor in weight estimation.
Revised: In other words, we need more samples from the target data to compensate a bad predictor in weight estimation.

Original: So the generalization error decreases faster with an increasing number of samples for good predictors.
Revised: So the generalization error decreases faster with an increasing number of samples for good predictors.

Original: [Figure]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The task of being robust against distributional shifts has been widely studied in the literature.
Revised: The task of being robust against distributional shifts has been widely studied in the literature.

Original: One line of work focuses on distributional robustness (see e.g. Esfahani & Kuhn; Namkoong & Duchi (2016)) without explicitly modeling the shift by making assumptions on the distribution. 
Revised: One line of work focuses on distributional robustness (see e.g. Esfahani & Kuhn; Namkoong & Duchi (2016)) without explicitly modeling the shift by making assumptions on the distribution. Our theoretical results adapt to any shift θ .

Original: However, these methods achieve robustness only against tiny changes in the distribution and are often impractical for large-scale high dimensional data.
Revised: However, these methods achieve robustness only against tiny changes in the distribution and are often impractical for large-scale high dimensional data.

Original: Another line of work has addressed the causal setting where the conditional distribution of Y given
Revised: Another line of work has addressed the causal setting where the conditional distribution of Y given

Original: X is assumed to be constant.
Revised: X is assumed to be constant.

Original: This assumption is sensible when there is reason to believe that there is a true optimal mapping from X to Y which does not change if the distribution of X changes.
Revised: This assumption is sensible when there is reason to believe that there is a true optimal mapping from X to Y which does not change if the distribution of X changes.

Original: In our example, if a given set of symptoms correctly predicts a disease for the “right reasons”, the predictive function should be the same, no matter which country you are moving to.
Revised: In our example, if a given set of symptoms correctly predicts a disease for the “right reasons”, the predictive function should be the same, no matter which country you are moving to.

Original: Among the various methods to correct for covariate shift, the majority uses the concept of importance weights q ( x ) /p
Revised: Among the various methods to correct for covariate shift, the majority uses the concept of importance weights q ( x ) /p

Original: ( x ) (Zadrozny, 2004; Cortes et al., 2010; Cortes & Mohri, 2014; Shimodaira, 2000), which are unknown but can be estimated for example via kernel embeddings (Huang et al., 2007; Gretton et al., 2009; 2012;
Revised: ( x ) (Zadrozny, 2004; Cortes et al., 2010; Cortes & Mohri, 2014; Shimodaira, 2000), which are unknown but can be estimated for example via kernel embeddings (Huang et al., 2007; Gretton et al., 2009; 2012;

Original: Zhang et al., 2013; Zaremba et al., 2013) or by learning a binary discriminative classiﬁer between source and target (Lopez-Paz & Oquab, 2016; Liu et al., 2017). 
Revised: Zhang et al., 2013; Zaremba et al., 2013) or by learning a binary discriminative classiﬁer between source and target (Lopez-Paz & Oquab, 2016; Liu et al., 2017). Finally, Sanderson & Scott (2014); Ramaswamy et al.

Revised: (2016) formulate the label shift problem as a mixture of the class conditional covariate distributions with unknown mixture weights.

Revised: Under the pairwise mutual irreducibility (Scott et al., 2013) assumption on the class conditional covariate distributions, they deploy the Neyman-Pearson criterion (Blanchard et al., 2010) to estimate the class distribution q ( y ) .

Original: Common issues shared by these methods is that they may result in a massive computational burden for large scale problems.
Revised: Common issues shared by these methods is that they either result in a massive computational burden for large scale problems or cannot be deployed for neural networks.

Original: Moreover, inaccurate estimates make classiﬁers suffer from high variance.

Original: Despite the wide applicability of label shift, approaches with global guarantees in high dimensional data regimes remain under-explored.
Revised: Despite the wide applicability of label shift, approaches with global guarantees in high dimensional data regimes remain under-explored.

Original: The correction of label shift mainly requires to estimate the importance weights q ( y ) /p
Revised: The correction of label shift mainly requires to estimate the importance weights q ( y ) /p

Original: ( y ) over the labels which typically live in a very low-dimensional space.
Revised: ( y ) over the labels which typically live in a very low-dimensional space.

Original: Bayesian approaches are studied when a prior over the marginal label distribution is assumed (Storkey, 2009; Chan & Ng, 2005).
Revised: Bayesian approaches are studied when a prior over the marginal label distribution is assumed (Storkey, 2009; Chan & Ng, 2005).

Original: However, these methods often require explicitly computing the posterior distribution of y and suffer from the curse of dimensionality.
Revised: However, these methods often require explicitly computing the posterior distribution of y and suffer from the curse of dimensionality.

Original: Recently, for the label shift setting,
Revised: Recently, for the label shift setting,

Original: Lipton et al.
Revised: Lipton et al.

Original: (2018) proposed Black Box Shift Estimation ( BBSE ), where a black box classiﬁer is used to estimate the importance weights q ( y ) /p
Revised: (2018) proposed Black Box Shift Estimation ( BBSL ) which can be applied to large scale data.

Original: ( y ) which are then used to train a classiﬁer on the source data.

Original: This approach is related to Buck et al.
Revised: This approach is related to Buck et al.

Original: (1966); Forman (2008);
Revised: (1966); Forman (2008);

Original: Saerens et al.
Revised: Saerens et al.

Original: in the low dimensional setting but lacks guarantees for the excess risk.
Revised: in the low dimensional setting but lacks guarantees for the excess risk.

Original: Generalization theory,which is usually used to bound the excess loss, has historically been developed for the case when P = Q (see e.g. (Vapnik, 1999; Bartlett & Mendelson, 2002; Kakade et al., 2009; Wainwright, 2019)).
Revised: On the other hand, generalization theory, usually used to bound the excess loss, has historically been developed for the case when P = Q (see e.g. Vapnik (1999) ; Bartlett & Mendelson (2002); Kakade et al. ; Wainwright (2019)).

Original: Ben-David et al.
Revised: Ben-David et al.

Original: (2010) provides theoretical analysis and generalization guarantee for domain adaption when the H-divergecne between join distributions is considered, while Crammer et al.
Revised: (2010) provides theoretical analysis and generalization guarantees for distribution shifts when the H-divergence between joint distributions is considered, whereas Crammer et al.

Original: (2008) shows the learning from multiple sources.
Revised: (2008) proves generalization bounds for learning from multiple sources.

Original: For the covariate shift setting, Cortes et al.
Revised: For the covariate shift setting, Cortes et al.

Original: (2010) provides a generalizing bound when q ( x ) /p
Revised: (2010) provides a generalization bound when q ( x ) /p

Original: ( x ) is known which however does not apply in practice.
Revised: ( x ) is known which however does not apply in practice.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this work, we establish the ﬁrst generalization guarantee for the label shift setting where no prior knowledge of q ( y ) /p
Revised: In this work, we establish the ﬁrst generalization guarantee for the label shift setting and propose an importance weighting procedure for which no prior knowledge of q ( y ) /p

Original: ( y ) is required.
Revised: ( y ) is required.

Original: Although RLLS is inspired by BBSE , it carries a more robust importance weight estimator as well as generalization guarantees  forboth thelarge and small sample regime.
Revised: Although RLLS is inspired by BBSL , it leads to a more robust importance weight estimator as well as generalization guarantees in particular for the small sample regime, which BBSL does not allow for.

Original: RLLS is also equipped with a sample-size-dependent regularization technique and further improves the classiﬁer in both regimes.
Revised: RLLS is also equipped with a sample-size-dependent regularization technique and further improves the classiﬁer in both regimes.

Original: Note that we consider this  a necessaryﬁrst step in the direction of solving shifts of this type, although the label shift assumption  might not hold in the real world.
Revised: We consider this work a necessary step in the direction of solving shifts of this type, although the label shift assumption itself might be too simpliﬁed in the real world.

Revised: In future work, we plan to also study the setting when it is slightly violated.

Original: In particular, x in practice cannot be solely explained by the wanted label y , but  also depends on attributes z which might not be observable.
Revised: For instance, x in practice cannot be solely explained by the wanted label y , but may also depend on attributes z which might not be observable.

Original: For example, the symptoms might not only depend on the disease but also  the country and living conditions, in the example of disease prediction.
Revised: In the disease prediction task for example, the symptoms might not only depend on the disease but also on the country and living conditions of its population.

Original: In such a case, the label shift assumption only holds in a slightly modiﬁed sense, i.e.that P ( X | Y
Revised: In such a case, the label shift assumption only holds in a slightly modiﬁed sense, i.e. P ( X | Y

Original: =
Revised: =

Original: y, Z = z ) =
Revised: y, Z = z ) =

Original: Q ( X | Y
Revised: Q ( X | Y

Original: =
Revised: =

Original: y, Z = z ) .
Revised: y, Z = z ) .

Original: If the attributes
Revised: If the attributes

Original: Z are observed, then our framework can readily be used to perform importance weights.
Revised: Z are observed, then our framework can readily be used to perform importance weighting.

Original: In the future work, we plan to study the setting where the label shift assumption is slightly violated.

Original: Furthermore, it is not clear whether the ﬁnal predictor is in fact “better” or more robust to shifts just because it achieves a better target accuracy than a vanilla unweighted estimator.
Revised: Furthermore, it is not clear whether the ﬁnal predictor is in fact “better” or more robust to shifts just because it achieves a better target accuracy than a vanilla unweighted estimator.

Original: In fact, there isa reason to believe that under certain shift scenarios, the predictor might learn to use spurious correlations to boost accuracy.
Revised: In fact, there is reason to believe that under certain shift scenarios, the predictor might learn to use spurious correlations to boost accuracy.

Original: Finding a procedure which can both learn a robust model and achieve high accuracies on new target sets remains to be an ongoing challenge.
Revised: Finding a procedure which can both learn a robust model and achieve high accuracies on new target sets remains to be an ongoing challenge.

Original: Moreover, the current choice of regularization depends on the number of samples rather than date driven regularization  is desired.
Revised: Moreover, the current choice of regularization depends on the number of samples rather than data driven regularization which is more desirable.

Original: An interesting direction toward active learning for the same disease-symptoms problem is whenin the country B we also have  a practitioner whom we can ask to diagnose some of the patients.
Revised: An interesting direction towards active learning for the same disease-symptoms scenario is when we also have an expert for diagnosing a limited number of patients in country

Original:  Now the question is which patient we would diagnose?
Revised: B. Now the question is which patients would be most "useful" to diagnose to obtain a high accuracy on the entire target set?

Original: Also, consider the case of high risk, we might be able to send some of the patients tothe country A for further medical diagnosis or treatment, up to some varying cost.
Revised: Furthermore, in the case of high risk, we might be able to send some of the patients to country A for further medical diagnosis or treatment, up to some varying cost.

Original: We plan to extend the current framework to  active learning setting where we actively query the label of certain x ’s (Beygelzimer et al., 2009) as well as the cost-sensitive setting where we also consider the cost of querying labels (Krishnamurthy et al., 2017).
Revised: We plan to extend the current framework to the active learning setting where we actively query the label of certain x ’s (Beygelzimer et al., 2009) as well as the cost-sensitive setting where we also consider the cost of querying labels (Krishnamurthy et al., 2017).

Original: A P ROOFS A. P ROOF OF L EMMA
Revised: A P ROOFS A. P ROOF OF L EMMA

Original: From Thm.
Revised: From Thm.

Original: 3.7 in (Pires & Szepesvári, 2012)
Revised: 3.7 in (Pires & Szepesvári, 2012)

Original: we know that for (cid:101) w as deﬁned in equation (3), if with probability at least 1 − δ , (cid:107) (cid:98) C − C (cid:107) 2 ≤ ∆ C and (cid:107) (cid:98) b − b (cid:107) 2 ≤ ∆ b hold simultaneously, then
Revised: we know that for (cid:98) θ as deﬁned in equation (3), if with probability at least 1 − δ , (cid:107) (cid:98) C − C (cid:107) 2 ≤ ∆ C and (cid:107) (cid:98) b − b (cid:107) 2 ≤ ∆ b hold simultaneously, then

Original: [Equation]
Revised: [Equation]

Original: 7) where we use the shorthand Υ( θ  ) = (cid:107)
Revised: 8) where we use the shorthand Υ( θ (cid:48) ) = (cid:107)

Original: Cθ  − b (cid:107) 2 .
Revised: Cθ (cid:48) − b (cid:107) 2 .

Original: We can get an upper bound on the right hand side of (7) is the inﬁmum by simply choosing a feasible θ (cid:48)
Revised: We can get an upper bound on the right hand side of (8) is the inﬁmum by simply choosing a feasible θ (cid:48)

Original: = θ .
Revised: = θ .

Original: We then have (cid:107)
Revised: We then have (cid:107)

Original: Cθ − b (cid:107) 

Revised: Cθ − b (cid:107)

Original: =
Revised: =

Original: and hence
Revised: and hence

Original: [Equation]
Revised: [Equation]

Original: as a consequence,
Revised: as a consequence,

Original: [Equation]
Revised: [Equation]

Original: Since (cid:107) C (cid:16) (cid:98) θ − θ (cid:17) (cid:107) 2 ≥ σ min ( C ) (cid:107) (cid:98) θ − θ (cid:107) 2 by deﬁnition of the minimum singular value, we thus have
Revised: Since (cid:107) C (cid:16) (cid:98) θ − θ (cid:17) (cid:107) 2 ≥ σ min ( C ) (cid:107) (cid:98) θ − θ (cid:107) 2 by deﬁnition of the minimum singular value, we thus have

Original: [Equation]
Revised: [Equation]

Original: Let us ﬁrst notice that
Revised: Let us ﬁrst notice that

Original: [Equation]
Revised: [Equation]

Original: The mathematical deﬁnition of the ﬁnite sample estimates (cid:98) [Equation] h [Equation]
Revised: The mathematical deﬁnition of the ﬁnite sample estimates (cid:98) C h , (cid:98) b h (in matrix and vector representation) with respect to some hypothesis h are as follows

Original: in matrix and vector representation) with respect to some hypothesis h are as follows

Original: [Equation]
Revised: [Equation]

Original: where m = | D q |
Revised: where m = | D q |

Original: and I is the indicator function.
Revised: and I is the indicator function.

Original: C h , b h can equivalently be expressed with the population over P for C h and over Q for b h respectively.
Revised: C h , b h can equivalently be expressed with the population over P for C h and over Q for b h respectively.

Original: We now use the following concentration
Revised: We now use the following concentration

Original: Lemmas to bound the estimation errors of (cid:98) C, (cid:98) b where we drop the subscript h for ease of notation.
Revised: Lemmas to bound the estimation errors of (cid:98) C, (cid:98) b where we drop the subscript h for ease of notation.

Original: Lemma 2 (Concentration of measurement matrix (cid:98) C )
Revised: Lemma 2 (Concentration of measurement matrix (cid:98) C )

Original: For ﬁnite sample estimate (cid:98) C we have
Revised: For ﬁnite sample estimate (cid:98) C we have

Original: [Equation]
Revised: [Equation]

Original: with probability at least (1 − δ ) .
Revised: with probability at least (1 − δ ) .

Original: Lemma 3 (Concentration of label measurements)
Revised: Lemma 3 (Concentration of label measurements)

Original: For the ﬁnite sample estimate (cid:98) b with respect to any hypothesis h
Revised: For the ﬁnite sample estimate (cid:98) b with respect to any hypothesis h

Original: it holds that
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 −  δfor δ < 0 .
Revised: with probability at least 1 − 2 δ .

Original: By Lemma.
Revised: By Lemma.

Original: 2 for concentration of C and Lemma.
Revised: 2 for concentration of C and Lemma.

Original: 3 for concentration of b we now have with probability at least 1 − δ
Revised: 3 for concentration of b we now have with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: which, considering that O ( 1 √ n ) dominates O ( 1 n ) , yields the statement of the Lemma 1.
Revised: which, considering that O ( 1 √ n ) dominates O ( 1 n ) , yields the statement of the Lemma 1.

Original: A.1. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: We prove this lemma using the theorem 1.4[Matrix Bernstein] and Dilations technique from Tropp (2012).
Revised: We prove this lemma using the theorem 1.4[Matrix Bernstein] and Dilations technique from Tropp (2012).

Original: We can rewrite C h
Revised: We can rewrite C h

Original: = E ( x,y ) ∼ P (cid:2)
Revised: = E ( x,y ) ∼ P (cid:2)

Original: e h ( x ) e (cid:62) y (cid:3) where e ( i ) is the one-hot-encoding of index i .
Revised: e h ( x ) e (cid:62) y (cid:3) where e ( i ) is the one-hot-encoding of index i .

Original: Consider a ﬁnite sequence { Ψ( i ) } of independent random matrices with dimension k .
Revised: Consider a ﬁnite sequence { Ψ( i ) } of independent random matrices with dimension k .

Original: By dilations, lets construct another sequence of self-adjoint random matrices of { (cid:101) Ψ
Revised: By dilations, lets construct another sequence of self-adjoint random matrices of { (cid:101) Ψ

Original: ( i ) } of dimension 2 k ,
Revised: ( i ) } of dimension 2 k ,

Original: such that for all
Revised: such that for all

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: therefore,
Revised: therefore,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: which results in (cid:107) (cid:101)
Revised: which results in (cid:107) (cid:101)

Original: Ψ
Revised: Ψ

Original: ( i ) (cid:107) 

Revised: ( i ) (cid:107)

Original: = (cid:107) Ψ( i ) (cid:107) 2 .
Revised: = (cid:107) Ψ( i ) (cid:107) 2 .

Original: The dilation technique translates the initial sequence of random matrices to the sequence of random self-adjoint matrices where we can apply the Matrix Bernstein theorem which states that, for a ﬁnite sequence of i.i.d.
Revised: The dilation technique translates the initial sequence of random matrices to the sequence of random self-adjoint matrices where we can apply the Matrix Bernstein theorem which states that, for a ﬁnite sequence of i.i.d.

Original: self-adjoint matrices (cid:101) Ψ( i ) , such that, almost surely ∀ i, E (cid:104)(cid:101) Ψ( i ) (cid:105) = 0 and (cid:107) (cid:101) Ψ
Revised: self-adjoint matrices (cid:101) Ψ( i ) , such that, almost surely ∀ i, E (cid:104)(cid:101) Ψ( i ) (cid:105) = 0 and (cid:107) (cid:101) Ψ

Original: ( i ) (cid:107) ≤ R , then for all t ≥ 0 ,(cid:107)t t (cid:88) i = (cid:101) Ψ ( i ) (cid:107) ≤ R log (2 k/δ ) 3 t + (cid:114)
Revised: ( i ) (cid:107) ≤ R , then for all t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: (cid:37) 2 log (2 k/δ ) t with probability at least 1 − δ where (cid:37) 2 : = (cid:107) E (cid:104)(cid:101) Ψ 2 ( i ) (cid:105) (cid:107) 2 , ∀ i which is also (cid:37) 

Revised: t with probability at least 1 − δ where (cid:37) 2 := (cid:107) E (cid:104)(cid:101) Ψ 2 ( i ) (cid:105) (cid:107) 2 , ∀ i which is also (cid:37)

Original: = (cid:107) E (cid:2)
Revised: = (cid:107) E (cid:2)

Original: Ψ 2 ( i ) (cid:3) (cid:107) 2 , ∀ i due to Eq. 8.
Revised: Ψ 2 ( i ) (cid:3) (cid:107) 2 , ∀ i due to Eq. 9.

Original: Therefore, thanks to  dilation trick,
Revised: Therefore, thanks to the dilation trick and theorem 1.4[Matrix Bernstein] in Tropp (2012),

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ .
Revised: with probability at least 1 − δ .

Original: Now, by plugging in Ψ( i ) =
Revised: Now, by plugging in Ψ( i ) =

Original: e h ( x ( i ))
Revised: e h ( x ( i ))

Original: e (cid:62) y
Revised: e (cid:62) y

Original: ( i ) − C , we have E (cid:104)(cid:101) Ψ
Revised: ( i ) − C , we have E (cid:104)(cid:101) Ψ

Original: ( i ) (cid:105)
Revised: ( i ) (cid:105)

Original: = 0 .
Revised: = 0 .

Original: Together with (cid:107) (cid:101) Ψ
Revised: Together with (cid:107) (cid:101) Ψ

Original: ( i ) (cid:107) ≤ 2 as well as (cid:37) 

Revised: ( i ) (cid:107) ≤ 2 as well as (cid:37)

Original: = (cid:107) E (cid:2)
Revised: = (cid:107) E (cid:2)

Original: Ψ 2 ( i ) (cid:3) (cid:107) 2 = 1 and setting t = n , we have
Revised: Ψ 2 ( i ) (cid:3) (cid:107) 2 = 1 and setting t = n , we have

Original: [Equation]
Revised: [Equation]

Original: A.1. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: The proof of this lemma is mainly based on a special case of and appreared at proposition 6 in
Revised: The proof of this lemma is mainly based on a special case of and appreared at proposition 6 in

Original: Azizzadenesheli et al.
Revised: Azizzadenesheli et al.

Original: Lemma F.1 in Anandkumar et al.
Revised: Lemma F.1 in Anandkumar et al.

Original: and Proposition 19 of Hsu et al.
Revised: and Proposition 19 of Hsu et al.

Original: Analogous to the previous section we can rewrite b h
Revised: Analogous to the previous section we can rewrite b h

Original: = E ( x,y ) ∈ Q [ e h ( x ) ] − E ( x
Revised: = E ( x,y ) ∈ Q [ e h ( x ) ] − E ( x

Original: ,y ) ∈ P [ e h ( x ) ] where e ( i ) is the one-hot-encoding of index i .
Revised: ,y ) ∈ P [ e h ( x ) ] where e ( i ) is the one-hot-encoding of index i .

Original: Note that (dropping the subscript h ) we have
Revised: Note that (dropping the subscript h ) we have

Original: [Equation]
Revised: [Equation]

Original: We now bound both estimates of probability vectors separately.
Revised: We now bound both estimates of probability vectors separately.

Original: Consider a ﬁxed multinomial distribution characterized with probability vector of ς ∈ ∆ k − 1 where ∆ k − 1 is a k − 1 dimensional simplex.
Revised: Consider a ﬁxed multinomial distribution characterized with probability vector of ς ∈ ∆ k − 1 where ∆ k − 1 is a k − 1 dimensional simplex.

Original: Further, consider t realization of this multinomial distribution { ς ( i ) } ti =1 where ς ( i ) is the one-hot-encoding of the i ’th sample.
Revised: Further, consider t realization of this multinomial distribution { ς ( i ) } ti =1 where ς ( i ) is the one-hot-encoding of the i ’th sample.

Original: Consider the empirical estimate mean of this distribution through empirical average of the samples; (cid:98) ς
Revised: Consider the empirical estimate mean of this distribution through empirical average of the samples; (cid:98) ς

Original: = 1 t (cid:80) ( i ) t
Revised: = 1 t (cid:80) ( i ) t

Original: ς
Revised: ς

Original: ( i ) , then
Revised: ( i ) , then

Original: [Equation]
Revised: [Equation]

Original: t with probability at least 1 − δ .
Revised: with probability at least 1 − δ .

Original: By plugging in ς = q h , (cid:98) ς = (cid:98) q h with t = n q and ﬁnally { ς ( i ) } n q i =1 = { e h ( x ( i )) }
Revised: By plugging in ς = q h , (cid:98) ς = (cid:98) q h with t = n q and ﬁnally { ς ( i ) } n q i =1 = { e h ( x ( i )) }

Original: ( i ) n q and equivalently for p
Revised: ( i ) n q and equivalently for p

Original: h we obtain for δ < 0 . 5 that
Revised: h we obtain;

Original: [Equation]
Revised: [Equation]

Original: we get the statement in Lemma
Revised: with probability at least 1 − 2 δ , therefore;

Original: [Equation]
Revised: [Equation]

Revised: resulting in the statement in the Lemma 3.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: We want to ultimately bound |L ( (cid:98) h (cid:98) w ) − L ( h (cid:63) ) | .
Revised: We want to ultimately bound |L ( (cid:98) h (cid:98) w ) − L ( h (cid:63) ) | .

Original: By addition and subtraction we have
Revised: By addition and subtraction we have

Original: [Equation]
Revised: [Equation]

Original: where n = βn p
Revised: where n = βn p

Original: and we used optimality of (cid:98) h (cid:98) w .
Revised: and we used optimality of (cid:98) h (cid:98) w .

Original: Here (a) is the weight estimation error and (b) is the ﬁnite sample error.
Revised: Here (a) is the weight estimation error and (b) is the ﬁnite sample error.

Original: Uniform law for bounding (b) We bound (b) using standard results for uniform laws for uniformly bounded functions which holds since (cid:107) w (cid:107) ∞ ≤ d ∞ ( q || p ) and (cid:107) (cid:96) (cid:107) ∞ ≤ 1 .
Revised: Uniform law for bounding (b) We bound (b) using standard results for uniform laws for uniformly bounded functions which holds since (cid:107) w (cid:107) ∞ ≤ d ∞ ( q || p ) and (cid:107) (cid:96) (cid:107) ∞ ≤ 1 .

Original: Since | w ( y ) (cid:96) ( h ( x ) , y ) | ≤ d ∞ ( q || p ) , ∀ x, y ∈
Revised: Since | w ( y ) (cid:96) ( h ( x ) , y ) | ≤ d ∞ ( q || p ) , ∀ x, y ∈

Original: X
Revised: X

Original: × Y , by deploying the McDiarmid’s inequality we then obtain that
Revised: × Y , by deploying the McDiarmid’s inequality we then obtain that

Original: [Equation]
Revised: [Equation]

Original:  where G ( (cid:96), H ) = { g h ( x, y ) =
Revised: n where G ( (cid:96), H ) = { g h ( x, y ) =

Original: w ( y ) (cid:96) ( h ( x ) , y ) : h ∈ H} and the Rademacher complexity is deﬁned asR n ( G ) :
Revised: w ( y ) (cid:96) ( h ( x ) , y ) : h ∈ H} and the Rademacher complexity is deﬁned as

Original: =
Revised: [Equation]

Original: E X ( i ) ,Y ( i ) ∼ P : i ∈ [ n ] (cid:2) E
Revised: (cid:80) ni

Original: [Equation]
Revised: [Equation]

Original: : i
Revised: σ

Original: [Equation]
Revised: [Equation]

Revised: w

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: y

Original: [Equation]
Revised: [Equation]

Original:  ( x
Revised: (cid:96) ( x

Original: [Equation]
Revised: [Equation]

Original:  y 
Revised: h ( y i )

Original: [Equation]
Revised: [Equation]

Original: )] (cid:3) .
Revised: ] (cid:3) .

Original: of the hypothesis class H (see for example Percy Liang notes on Statistical Learning Theory and chapter 4 in Wainwright (2019))
Revised: of the hypothesis class H (see for example Percy Liang notes on Statistical Learning Theory and chapter 4 in Wainwright (2019))

Original: Bounding term (a) Remember that k = |Y| is the cardinality of the ﬁnite domain of Y , or the number of classes.
Revised: Bounding term (a) Remember that k = |Y| is the cardinality of the ﬁnite domain of Y , or the number of classes.

Original: Let us deﬁne (cid:101) (cid:96)
Revised: Let us deﬁne (cid:101) (cid:96)

Original: ∈
Revised: ∈

Original: R k with (cid:101) (cid:96) j = (cid:80) ni
Revised: R k with (cid:101) (cid:96) j = (cid:80) ni

Original: =

Revised: =

Original: I y ( i )=
Revised: I y ( i )=

Original: j (cid:96) ( y ( i ) , h ( x ( i ))) .
Revised: j (cid:96) ( y ( i ) , h ( x ( i ))) .

Original: Notice that by deﬁnition (cid:107) (cid:101) (cid:96) (cid:107) 1 ≤ n and (cid:107) (cid:101) (cid:96) (cid:107) ∞ ≤ n from which it follows by Hoelder’s inequality that (cid:107) (cid:101) (cid:96) (cid:107) 2 ≤ n .
Revised: Notice that by deﬁnition (cid:107) (cid:101) (cid:96) (cid:107) 1 ≤ n and (cid:107) (cid:101) (cid:96) (cid:107) ∞ ≤ n from which it follows by Hoelder’s inequality that (cid:107) (cid:101) (cid:96) (cid:107) 2 ≤ n .

Original: Furthermore, we slightly abuse notation and use w to denote the k -dimensional vector with w( i) = w ( i ) .
Revised: Furthermore, we slightly abuse notation and use w to denote the k -dimensional vector with w i = w ( i ) .

Original: Therefore, for all h we have via the Cauchy Schwarz inequality that
Revised: Therefore, for all h we have via the Cauchy Schwarz inequality that

Original: [Equation]
Revised: [Equation]

Original: It then follows by Lemma 1 that
Revised: It then follows by Lemma 1 that

Original: [Equation]
Revised: [Equation]

Original: Plugging both bounds back into equation (9) concludes the proof of the theorem.

Original: A. G ENERALIZATION FOR FINITE HYPOTHESIS CLASSES
Revised: A. G ENERALIZATION FOR FINITE HYPOTHESIS CLASSES

Original: For ﬁnite hypothesis classes, one may bound (b) in (9) using Bernstein’s inequality.
Revised: For ﬁnite hypothesis classes, one may bound (b) in (10) using Bernstein’s inequality.

Original: We bound (b) by ﬁrst noting that w ( Y ) (cid:96) ( Y, h ( X )) satisﬁes the Bernstein conditions so that Bernstein’s inequality holds
Revised: We bound (b) by ﬁrst noting that w ( Y ) (cid:96) ( Y, h ( X )) satisﬁes the Bernstein conditions so that Bernstein’s inequality holds

Original: [Equation]
Revised: [Equation]

Original: by deﬁnition.
Revised: by deﬁnition.

Original: Because we assume (cid:96) ≤ 1 , we directly have
Revised: Because we assume (cid:96) ≤ 1 , we directly have

Original: [Equation]
Revised: [Equation]

Original: Since we have a bound on the second moment of weighted loss while its ﬁrst moment is L ( h ) we can apply Bernstein’s inequality to obtain for any ﬁxed h that
Revised: Since we have a bound on the second moment of weighted loss while its ﬁrst moment is L ( h ) we can apply Bernstein’s inequality to obtain for any ﬁxed h that

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: For the uniform law for ﬁnite hypothesis classes we can just do union bound on top of Bernstein and could get a slightly better dependence on d ∞ ( q || p ) . The second moment of the importance weighted loss E P (cid:2) ω ( Y )
Revised: For the uniform law for ﬁnite hypothesis classes make the union bound on all the hypotheses;

Original: [Equation]
Revised: [Equation]

Original:  2 ( h ( X ) , Y ) (cid:3) , given a h ∈ H (cid:48) can be bounded for general α ≥ 0 , potentially leading to smaller values than d ( q || p ) :[Equation] =
Revised: n The second moment of the importance weighted loss E P (cid:2) ω ( Y ) 2 (cid:96) 2 ( h ( X ) , Y ) (cid:3) , given a h ∈ H (cid:48) can be bounded for general α ≥ 0 , potentially leading to smaller values than d ( q || p ) :

Original: [Equation]
Revised: [Equation]

Original: (12) where the ﬁrst inequality follows from Hölder’s inequality [Equation] the second one follows from Jensen’s inequality and the fact that the loss is in [0 , 1] as well as the fact that the exponentiation function is convex in this region.
Revised: where the ﬁrst inequality follows from Hölder’s inequality, the second one follows from Jensen’s inequality and the fact that the loss is in [0 , 1] as well as the fact that the exponentiation function is convex in this region.

Original: Moreover, since 1 + 1 α ≥ 1 and upper bound for the loss square, l ( · , · ) 2 ≤ 1 , then;
Revised: Moreover, since 1 + 1 α ≥ 1 and upper bound for the loss square, l ( · , · ) 2 ≤ 1 , then;

Original: [Equation]
Revised: [Equation]

Original: which gives bound on the second moment of weighted loss.
Revised: which gives bound on the second moment of weighted loss.

Revised: A. M ORE EXPERIMENTAL RESULTS

Revised: This section contains more experiments that provide more insights about in which settings the advantage of using RLLS vs. BBSL are more or less pronounced.

Revised: A.6. CIFAR10 E XPERIMENTS UNDER TWEAK - ONE SHIFT AND D IRICHLET SHIFT

Revised: Here we compare weight estimation performance between RLLS and BBSL for different types of shifts including the Tweak-one Shift , for which we randomly choose one class,

Revised: e.g. i and set p ( i ) =

Revised: ρ while all other classes are distributed evenly.

Revised: Figure 5 depicts the the weight estimation performance of RLLS compared to BBSL for a variety of values of ρ and α .

Revised: Note that larger shifts correspond to smaller α and larger ρ .

Revised: In general, one observes that our RLLS estimator has smaller MSE and that as the shift increases, the error of both methods increases.

Revised: For tweak-one shift we can additionally see that as the shift increases, RLLS outperforms BBSL more and more as both in terms of bias and variance.

Revised: [Figure]

Revised: A.6. MNIST E XPERIMENTS UNDER M INORITY -C LASS SOURCE SHIFTS FOR DIFFERENT VALUES OF p

Revised: In order to show weight estimation and classiﬁcation performance under different level of label shifts, we include several additional sets of experiments here in the appendix.

Revised: Figure 6 shows the weight estimation error and accuracy comparison under a minority-class shift with p = 0.001.

Revised: The training and testing sample size is 10000 examples in this case.

Revised: We can see that whenever the weight estimation of RLLS is better, the accuracy is also better, except in the four classes case when both methods are bad in weight estimation.

Revised: Figure 7 demonstrates another case in minority-class shift

Revised: [Figure]

Revised: when p = 0 .

Revised: The black-box classiﬁer is the same two-layers neural network trained on a biased source data set with tweak-one ρ = 0 .

Revised: We observe that when the number of minority class is small like 1 or 2, the weight estimation is similar between two methods, as well as in the classiﬁcation accuracy.

Revised: But when the shift get larger, the weights are worse and the performance in accuracy decreases, getting even worse than the unweighted classiﬁer.

Revised: [Figure]

Revised: A.6. MNIST E XPERIMENT UNDER D IRICHLET S HIFT WITH LOW TARGET SAMPLES

Revised: We show the performance of classiﬁer with different regularization λ under a Dirichlet shift with α = 0 .

Revised: 5 in Figure 8.

Revised: The training has 5000 examples in this case.

Revised: We can see that in this low target sample case, λ

Revised: = 1 only take over after several hundreds example, while some λ value betweenand 1 outperforms it at the beginning.

Revised: Similar as in the paper, we use different black-box classiﬁer that is corrupted in different levels to show the relation between the quality of black-box predictor and the necessary target sample size.

Revised: We use biased source data with tweak-one ρ = 0 , 0 . 2 , 0 .

Revised: 6 to train the black-box classiﬁer.

Revised: We see that we need more target samples for the fully weighted version λ

Revised: = 1 to take over for a more corrupted black-box classiﬁer.

Revised: [Figure]