Original: R EGULARIZED L EARNING FOR D OMAIN A DAPTATION UNDER L ABEL S HIFTS
Revised: R EGULARIZED L EARNING FOR D OMAIN A DAPTATION UNDER L ABEL S HIFTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose Regularized Learning under Label shifts ( RLLS ), a principled and practical domain adaptation approach to correct for shifts in the label distribution between a source and a target domain.
Revised: We propose Regularized Learning under Label shifts ( RLLS ), a principled and practical domain adaptation approach to correct for shifts in the label distribution between a source and a target domain.

Original: We ﬁrst estimate importance weights using both labeled source and unlabeled target data before training a classiﬁer on the weighted source samples.
Revised: We ﬁrst estimate importance weights using both labeled source and unlabeled target data before training a classiﬁer on the weighted source samples.

Original: Depending on the number of source and target samples, we thereby regularize the inﬂuence of these estimated weights.
Revised: Depending on the number of source and target samples, we thereby regularize the inﬂuence of these estimated weights.

Original: We derive a generalization bound for the classiﬁer on the target domain which is independent of the (ambient) data dimensions and instead depends on the complexity of the function class.
Revised: We derive a generalization bound for the classiﬁer on the target domain which is independent of the (ambient) data dimensions and instead depends on the complexity of the function class.

Original: To the best of our knowledge, this is the ﬁrst generalization bound for label-shift problems where the labels in the target domain are unknown.
Revised: To the best of our knowledge, this is the ﬁrst generalization bound for label-shift problems where the labels in the target domain are unknown.

Original: Based on this bound, we propose a regularized estimator for the small-sample regime which accounts for the uncertainty in the estimated weights.
Revised: Based on this bound, we propose a regularized estimator for the small-sample regime which accounts for the uncertainty in the estimated weights.

Original: Experiments on the
Revised: Experiments on the

Original: CIFAR-10 and MNIST datasets show that RLLS improves classiﬁcation accuracy in the low sample and large-shift regimes compared to previous methods.
Revised: CIFAR-10 and MNIST datasets show that RLLS improves classiﬁcation accuracy in the low sample and large-shift regimes compared to previous methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In supervised learning, we usually make predictions on an unlabeled target set by training a predictive model on a labeled source set.
Revised: In supervised learning, we usually make predictions on an unlabeled target set by training a predictive model on a labeled source set.

Original: This is a sensible approach especially when the source and target sets are expected to have i.i.d.
Revised: This is a sensible approach especially when the source and target sets are expected to have i.i.d.

Original: samples from the same distribution.
Revised: samples from the same distribution.

Original: However, this assumption does not hold in many real-world applications.
Revised: However, this assumption does not hold in many real-world applications.

Original: For example, often times predictions for medical diagnostics are based on data from a particular population of patients and machines due to the limited variety of data that is available in the training phase.
Revised: For example, often times predictions for medical diagnostics are based on data from a particular population of patients and machines due to the limited variety of data that is available in the training phase.

Original: Let us consider the following scenario.
Revised: Let us consider the following scenario.

Original: A classiﬁer is trained to predict whether a patient has contracted a severe disease in country A based on the physical measurements and diagnosis (labeled source data).
Revised: A classiﬁer is trained to predict whether a patient has contracted a severe disease in country A based on the physical measurements and diagnosis (labeled source data).

Original: The disease is potentially even more prevalent in country B, where no experts are available to make a reliable diagnosis.
Revised: The disease is potentially even more prevalent in country B, where no experts are available to make a reliable diagnosis.

Original: The following questions could be relevant in this scenario:
Revised: The following questions could be relevant in this scenario:

Original: • If we send a group of volunteers to country B with the preliminary equipment to record measurements of patients, how can the data in country
Revised: • If we send a group of volunteers to country B with the preliminary equipment to record measurements of patients, how can the data in country

Original: A
Revised: A

Original: and the new unlabeled data from country B be used to update the classiﬁer to give good diagnostic predictions for patients in country B?
Revised: and the new unlabeled data from country B be used to update the classiﬁer to give good diagnostic predictions for patients in country B?

Original: • If our group members visit the patients in country B in online and daily manner rather than batch and the disease is deadly, can the classiﬁer be updated after only few patients have been measured, so as to maximize the accuracy of the diagnosis for the following patients?
Revised: • If our group members visit the patients in country B in online and daily manner rather than batch and the disease is deadly, can the classiﬁer be updated after only few patients have been measured, so as to maximize the accuracy of the diagnosis for the following patients?

Original: Similar problems arise in many other domains besides health care, but we will use this medical example throughout the paper for ease of presentation.
Revised: Similar problems arise in many other domains besides health care, but we will use this medical example throughout the paper for ease of presentation.

Original: If we view the data points ( X i , Y i ) in the source and target set as i.i.d.
Revised: If we view the data points ( X i , Y i ) in the source and target set as i.i.d.

Original: samples from a joint distributions P and Q taking values in X × Y , we are dealing with the distributional shift problem when P (cid:54) =
Revised: samples from a joint distributions P and Q taking values in X × Y , we are dealing with the distributional shift problem when P (cid:54) =

Original: Q .
Revised: Q .

Original: There are various ways to model the difference between P and Q , depending on the use case at hand.
Revised: There are various ways to model the difference between P and Q , depending on the use case at hand.

Original: Traditionally, the two following perspectives are considered: (i) the causal setting where the input x explains the label y , and ( ii ) the anti-causal setting, where the label y causes the input x .
Revised: Traditionally, the two following perspectives are considered: (i) the causal setting where the input x explains the label y , and ( ii ) the anti-causal setting, where the label y causes the input x .

Original: For the causal setting, one usually assumes that the marginal distribution of the covariates x shifts between P and Q , whereas the conditional probability mass function of y given x stays constant, which we denote as p ( y | x ) =
Revised: For the causal setting, one usually assumes that the marginal distribution of the covariates x shifts between P and Q , whereas the conditional probability mass function of y given x stays constant, which we denote as p ( y | x ) =

Original: q ( y | x ) for all y ∈ Y and x ∈ X .
Revised: q ( y | x ) for all y ∈ Y and x ∈ X .

Original: Although this is valid in some situations, in many data science applications an inherent ground truth Y causes an observation X .
Revised: Although this is valid in some situations, in many data science applications an inherent ground truth Y causes an observation X .

Original: For example, reported physical symptoms might be approximately similar across all patients that have contracted a particular disease.
Revised: For example, reported physical symptoms might be approximately similar across all patients that have contracted a particular disease.

Original: This simpliﬁed relation can be modeled e.g. by a directed graphical model Y → X .
Revised: This simpliﬁed relation can be modeled e.g. by a directed graphical model Y → X .

Original: It is now possible that the percentage of ill patients in the two countries A and B are very different, i.e. P ( Y = y ) (cid:54) =
Revised: It is now possible that the percentage of ill patients in the two countries A and B are very different, i.e. P ( Y = y ) (cid:54) =

Original: Q ( Y = y )
Revised: Q ( Y = y )

Original: ( p ( y ) (cid:54) =
Revised: ( p ( y ) (cid:54) =

Original: q ( y ) for short) for all y ∈ Y .
Revised: q ( y ) for short) for all y ∈ Y .

Original: Similarly, within one country, the number of ill patients may suddenly change because of an irregular event such as a natural catastrophe.
Revised: Similarly, within one country, the number of ill patients may suddenly change because of an irregular event such as a natural catastrophe.

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: In all of the above cases, there is a shift in the distribution of the label y , while given a ﬁxed label, the conditional distribution of x given y is constant, i.e., p ( x | y )
Revised: In all of the above cases, there is a shift in the distribution of the label y , while given a ﬁxed label, the conditional distribution of x given y is constant, i.e., p ( x | y )

Original: =
Revised: =

Original: q ( x | y ) .
Revised: q ( x | y ) .

Original: By the Bayes rule, label shift causes the covariate shift condition to be violated, i.e. p ( y | x ) (cid:54) =
Revised: By the Bayes rule, label shift causes the covariate shift condition to be violated, i.e. p ( y | x ) (cid:54) =

Original: q ( y | x ) .
Revised: q ( y | x ) .

Original: Despite being an intuitively natural scenario in some cases, even this simpliﬁed model has not yet been sufﬁciently studied in the literature.
Revised: Despite being an intuitively natural scenario in some cases, even this simpliﬁed model has not yet been sufﬁciently studied in the literature.

Original: This paper aims to ﬁll the gap in terms of both theoretical understanding and practical methods for the label shift setting and thereby move a step closer towards more complicated and realistic distributional shift models.
Revised: This paper aims to ﬁll the gap in terms of both theoretical understanding and practical methods for the label shift setting and thereby move a step closer towards more complicated and realistic distributional shift models.

Original: In particular, our goal was to ﬁnd an efﬁcient method which is scalable to large-scale data where conventional approaches such as ones based on kernel mean embeddings, may become computationally feasible.
Revised: In particular, our goal was to ﬁnd an efﬁcient method which is scalable to large-scale data where conventional approaches such as ones based on kernel mean embeddings, may become computationally feasible.

Original: Lipton et al.
Revised: Lipton et al.

Original: (2018) introduced an importance-weight estimator, which has access to the labeled source data set and a large batch of unlabeled target data, and demonstrated promising performance on real data.
Revised: (2018) introduced an importance-weight estimator, which has access to the labeled source data set and a large batch of unlabeled target data, and demonstrated promising performance on real data.

Original: Using a black-box classiﬁer which can be biased, uncalibrated and inaccurate, they ﬁrst estimate importance weights q ( y ) /p
Revised: Using a black-box classiﬁer which can be biased, uncalibrated and inaccurate, they ﬁrst estimate importance weights q ( y ) /p

Original: ( y ) for the source samples and train a classiﬁer on the weighted data.
Revised: ( y ) for the source samples and train a classiﬁer on the weighted data.

Original: In the following we refer to the procedure as black box shift learning ( BBSL ) which the authors proved to be good for large enough sample size.
Revised: In the following we refer to the procedure as black box shift learning ( BBSL ) which the authors proved to be good for large enough sample size.

Original: There are two relevant questions which remain unanswered by their work: What are the generalization guarantees for the ﬁnal predictor which uses the weighted samples?
Revised: There are two relevant questions which remain unanswered by their work: What are the generalization guarantees for the ﬁnal predictor which uses the weighted samples?

Original: How do we deal with the uncertainty of the weight estimation when only few samples are available?
Revised: How do we deal with the uncertainty of the weight estimation when only few samples are available?

Original: In this work, our ﬁrst contribution is to propose an efﬁcient weight estimator for which we can obtain good statistical guarantees without a requirement on the problem-dependent minimum sample complexity as necessary for BBSL , which in their case can be arbitrarily large for small number of samples.
Revised: In this work, our ﬁrst contribution is to propose an efﬁcient weight estimator for which we can obtain good statistical guarantees without a requirement on the problem-dependent minimum sample complexity as necessary for BBSL , which in their case can be arbitrarily large for small number of samples.

Original: We then derive a dimension-independent generalization bound for the ﬁnal Regularized
Revised: We then derive a dimension-independent generalization bound for the ﬁnal Regularized

Original: Learning under Label Shift ( RLLS ) classiﬁer trained on the reweighted samples.
Revised: Learning under Label Shift ( RLLS ) classiﬁer trained on the reweighted samples.

Original: Our method improves the weight estimation error and the excess risk of the classiﬁer on reweighted samples by a factor of k log( k ) , where k is the number of classes, i.e. the cardinality of Y .
Revised: Our method improves the weight estimation error and the excess risk of the classiﬁer on reweighted samples by a factor of k log( k ) , where k is the number of classes, i.e. the cardinality of Y .

Original: It also follows from our analysis that explicitly controlling the inﬂuence of our regularized weight estimates could help in practice.
Revised: It also follows from our analysis that explicitly controlling the inﬂuence of our regularized weight estimates could help in practice.

Original: In particular, it suggests that when the number of target samples is below a certain threshold, the classiﬁer would obtain higher accuracy if the weight estimates are not trusted because of high uncertainty and the samples stay unweighted.
Revised: In particular, it suggests that when the number of target samples is below a certain threshold, the classiﬁer would obtain higher accuracy if the weight estimates are not trusted because of high uncertainty and the samples stay unweighted.

Original: We empirically study the performance of RLLS and show weight estimation as well as prediction accuracy for a variety of shifts on the CIFAR-10 and MNIST datasets, including shifting source data and shifting target data with various parameters, different sample sizes and regularization parameters.
Revised: We empirically study the performance of RLLS and show weight estimation as well as prediction accuracy for a variety of shifts on the CIFAR-10 and MNIST datasets, including shifting source data and shifting target data with various parameters, different sample sizes and regularization parameters.

Original: We achieve an order of magnitude smaller weight estimation error than baseline methods in large shift cases and enjoy 20% higher accuracy and F-1 score in corresponding predictive tasks.
Revised: We achieve an order of magnitude smaller weight estimation error than baseline methods in large shift cases and enjoy 20% higher accuracy and F-1 score in corresponding predictive tasks.

Original: We also see at least 10% accuracy improvement using our regularized estimator when faced with small target sample sizes.
Revised: We also see at least 10% accuracy improvement using our regularized estimator when faced with small target sample sizes.

Original:  R EGULARIZED LEARNING OF LABEL SHIFTS ( RLLS )
Revised:  R EGULARIZED LEARNING OF LABEL SHIFTS ( RLLS )

Original: Formally let us the short hand for the marginal probability mass functions of Y on ﬁnite Y with respect to P , Q as p, q : [ k ] → [0 , 1] with p ( i )
Revised: Formally let us the short hand for the marginal probability mass functions of Y on ﬁnite Y with respect to P , Q as p, q : [ k ] → [0 , 1] with p ( i )

Original: =
Revised: =

Original: P ( Y = i ) , and q ( i ) =
Revised: P ( Y = i ) , and q ( i ) =

Original: Q ( Y = i ) for all i ∈ [ k ] , representable by vectors in R k + which sum to one.
Revised: Q ( Y = i ) for all i ∈ [ k ] , representable by vectors in R k + which sum to one.

Original: In the label shift setting, we deﬁne the importance weight vector w ∈ R k between these two domains as w ( i ) =
Revised: In the label shift setting, we deﬁne the importance weight vector w ∈ R k between these two domains as w ( i ) =

Original: q ( i ) p ( i ) .
Revised: q ( i ) p ( i ) .

Original: We quantify the shift using the exponent of the inﬁnite and second order Renyi divergence as follows
Revised: We quantify the shift using the exponent of the inﬁnite and second order Renyi divergence as follows

Original: [Equation]
Revised: [Equation]

Original: Given a hypothesis class H and a loss function (cid:96) : Y × Y → [0 , 1] , our aim is to ﬁnd the hypothesis h ∈ H which minimizes
Revised: Given a hypothesis class H and a loss function (cid:96) : Y × Y → [0 , 1] , our aim is to ﬁnd the hypothesis h ∈ H which minimizes

Original: [Equation]
Revised: [Equation]

Original: In the usual ﬁnite sample setting however, L unknown and we observe samples { ( x j , y j ) } nj =1 from P instead.
Revised: In the usual ﬁnite sample setting however, L unknown and we observe samples { ( x j , y j ) } nj =1 from P instead.

Original: If we are given the vector of importance weights w we could then minimize the empirical loss with importance weighted samples deﬁned as
Revised: If we are given the vector of importance weights w we could then minimize the empirical loss with importance weighted samples deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where n is the number of available observations drawn from P used to learn the classiﬁer h .
Revised: where n is the number of available observations drawn from P used to learn the classiﬁer h .

Original: As w is unknown in practice, we have to ﬁnd the minimizer of the empirical loss with estimated importance weights
Revised: As w is unknown in practice, we have to ﬁnd the minimizer of the empirical loss with estimated importance weights

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) w are estimates of w .
Revised: where (cid:98) w are estimates of w .

Original: Given a set D p of n p samples from the source distribution P , we ﬁrst divide it into two sets where we use (1 − β ) n p samples in set D weight p to compute the estimate (cid:98) w and the remaining n = βn p in the set D class p to ﬁnd the classiﬁer which minimizes the loss (1), i.e.
Revised: Given a set D p of n p samples from the source distribution P , we ﬁrst divide it into two sets where we use (1 − β ) n p samples in set D weight p to compute the estimate (cid:98) w and the remaining n = βn p in the set D class p to ﬁnd the classiﬁer which minimizes the loss (1), i.e.

Original: (cid:98) h (cid:98) w = arg min
Revised: (cid:98) h (cid:98) w = arg min

Original: h ∈H L n ( h ; (cid:98) w ) .
Revised: h ∈H L n ( h ; (cid:98) w ) .

Original: In the following, we describe how to estimate the weights (cid:98) w and provide guarantees for the resulting estimator (cid:98) h (cid:98) w .
Revised: In the following, we describe how to estimate the weights (cid:98) w and provide guarantees for the resulting estimator (cid:98) h (cid:98) w .

Original: Plug-in weight estimation
Revised: Plug-in weight estimation

Original: The following simple correlation between the label distributions p, q was noted in Lipton et al.
Revised: The following simple correlation between the label distributions p, q was noted in Lipton et al.

Original: (2018): for a ﬁxed hypothesis h , if for all y ∈ Y it holds that q ( y ) ≥ 

Revised: (2018): for a ﬁxed hypothesis h , if for all y ∈ Y it holds that q ( y ) ≥

Original: = ⇒ p
Revised: = ⇒ p

Original: ( y ) ≥ 0 , we have
Revised: ( y ) ≥ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: for all i, j ∈ Y .
Revised: for all i, j ∈ Y .

Original: This can equivalently be written in matrix vector notation as
Revised: This can equivalently be written in matrix vector notation as

Original: [Equation]
Revised: [Equation]

Original: where C h is the confusion matrix with [ C h ] i,j = p ( h ( X )
Revised: where C h is the confusion matrix with [ C h ] i,j = P ( h ( X )

Original: =
Revised: =

Original: i, Y = j ) and q h is the vector which represents the probability mass function of h ( X ) under distribution Q .
Revised: i, Y = j ) and q h is the vector which represents the probability mass function of h ( X ) under distribution Q .

Original: The requirement q ( y ) ≥ 0 = ⇒ p
Revised: The requirement q ( y ) ≥ 0 = ⇒ p

Original: ( y ) ≥ 0 is a reasonable condition since without any prior knowledge, there is no way to properly reason about a class in the target domain that is not represented in the source domain.
Revised: ( y ) ≥ 0 is a reasonable condition since without any prior knowledge, there is no way to properly reason about a class in the target domain that is not represented in the source domain.

Original: In reality, both q h and C h can only be estimated by the corresponding ﬁnite sample averages (cid:98) q h , (cid:98) C h .
Revised: In reality, both q h and C h can only be estimated by the corresponding ﬁnite sample averages (cid:98) q h , (cid:98) C h .

Original: Lipton et al.
Revised: Lipton et al.

Original: (2018) simply compute the inverse of the estimated confusion matrix (cid:98) C h in order to estimate the importance weight, i.e. (cid:98)
Revised: (2018) simply compute the inverse of the estimated confusion matrix (cid:98) C h in order to estimate the importance weight, i.e. (cid:98)

Original: w = (cid:98) C − 1 h (cid:98) q h .
Revised: w = (cid:98) C − 1 h (cid:98) q h .

Original: While C − 1 h (cid:98) q h is a statistically efﬁcient estimator, (cid:98) w with estimated (cid:98) C − 1 h can be arbitrarily bad since (cid:98) C − 1 h can be arbitrary close to a singular matrix especially for small sample sizes and small minimum singular value of the confusion matrix.
Revised: While C − 1 h (cid:98) q h is a statistically efﬁcient estimator, (cid:98) w with estimated (cid:98) C − 1 h can be arbitrarily bad since (cid:98) C − 1 h can be arbitrary close to a singular matrix especially for small sample sizes and small minimum singular value of the confusion matrix.

Original: Intuitively, when there are very few samples, the weight estimation will have high variance in which case it might be better to avoid importance weighting altogether.
Revised: Intuitively, when there are very few samples, the weight estimation will have high variance in which case it might be better to avoid importance weighting altogether.

Original: Furthermore, even when the sample complexity in Lipton et al.
Revised: Furthermore, even when the sample complexity in Lipton et al.

Original: (2018), unknown in practice, is met, the resulting error of this estimator is linear in k which is problematic for large k .
Revised: (2018), unknown in practice, is met, the resulting error of this estimator is linear in k which is problematic for large k .

Original: We therefore aim to address these shortcomings by proposing the following two-step procedure to compute importance weights.
Revised: We therefore aim to address these shortcomings by proposing the following two-step procedure to compute importance weights.

Original: In the case of no shift we have w = 1 so that we deﬁne the amount of weight shift as θ = w − 1 .
Revised: In the case of no shift we have w = 1 so that we deﬁne the amount of weight shift as θ = w − 1 .

Original: Given a “decent” black box estimator which we denote by h 0 , we make the ﬁnal classiﬁer less sensitive to the estimation performance of C (i.e. regularize the weight estimate) by
Revised: Given a “decent” black box estimator which we denote by h 0 , we make the ﬁnal classiﬁer less sensitive to the estimation performance of C (i.e. regularize the weight estimate) by

Original: calculating the measurement error adjusted (cid:98) θ (described in Section 2.1 for h 0 ) and 2.
Revised: calculating the measurement error adjusted (cid:98) θ (described in Section 2.1 for h 0 ) and 2.

Original: computing the regularized weight (cid:98) w =
Revised: computing the regularized weight (cid:98) w =

Original: + λ (cid:98) θ where λ depends on the sample size (1 − β ) n p .
Revised: + λ (cid:98) θ where λ depends on the sample size (1 − β ) n p .

Original: By "decent" we refer to a classiﬁer h 0 which yields a full rank confusion matrix C h 0 .
Revised: By "decent" we refer to a classiﬁer h 0 which yields a full rank confusion matrix C h 0 .

Original: A trivial example for a non-”decent” classiﬁer h 0 is one that always outputs a ﬁxed class.
Revised: A trivial example for a non-”decent” classiﬁer h 0 is one that always outputs a ﬁxed class.

Original: As it does not capture any characteristics of the data, there is no hope to gain any statistical information without any prior information.
Revised: As it does not capture any characteristics of the data, there is no hope to gain any statistical information without any prior information.

Original: 2. E STIMATOR CORRECTING FOR FINITE SAMPLE ERRORS
Revised: 2. E STIMATOR CORRECTING FOR FINITE SAMPLE ERRORS

Original: Both the confusion matrix C h 0 and the label distribution q h 0 on the target for the black box hypothesis h 0 are unknown and we are instead only given access to ﬁnite sample estimates (cid:98) C h 0 , (cid:98)
Revised: Both the confusion matrix C h 0 and the label distribution q h 0 on the target for the black box hypothesis h 0 are unknown and we are instead only given access to ﬁnite sample estimates (cid:98) C h 0 , (cid:98)

Original: q h 0 .
Revised: q h 0 .

Original: In what follows all empirical and population confusion matrices, as well as label distributions, are deﬁned with respect to the hypothesis h
Revised: In what follows all empirical and population confusion matrices, as well as label distributions, are deﬁned with respect to the hypothesis h

Original: = h 0 .
Revised: = h 0 .

Original: For notation simplicity, we thus drop the subscript h 0 in what follows.
Revised: For notation simplicity, we thus drop the subscript h 0 in what follows.

Original: The reparameterized linear model (2) with respect to θ then reads
Revised: The reparameterized linear model (2) with respect to θ then reads

Original: [Equation]
Revised: [Equation]

Original: with corresponding ﬁnite sample quantity (cid:98) b = (cid:98) q − (cid:98) C 1 .
Revised: with corresponding ﬁnite sample quantity (cid:98) b = (cid:98) q − (cid:98) C 1 .

Original: When (cid:98) C is near singular, the estimation of θ becomes unstable.
Revised: When (cid:98) C is near singular, the estimation of θ becomes unstable.

Original: Furthermore, large values in the true shift θ result in large variances.
Revised: Furthermore, large values in the true shift θ result in large variances.

Original: We address this problem by adding a regularizing (cid:96) 2 penalty term to the usual loss and thus push the amount of shift towards 0 , a method that has been proposed in (Pires & Szepesvári, 2012).
Revised: We address this problem by adding a regularizing (cid:96) 2 penalty term to the usual loss and thus push the amount of shift towards 0 , a method that has been proposed in (Pires & Szepesvári, 2012).

Original: In particular, we compute
Revised: In particular, we compute

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) θ γ is deﬁned by
Revised: where (cid:98) θ γ is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: Here, ∆ C , ∆ b are parameters which will eventually be high probability upper bounds for (cid:107) (cid:98) C − C (cid:107)and (cid:107) (cid:98) b − b (cid:107) 2 respectively, where (cid:107) · (cid:107) 2 denotes the spectral norm.
Revised: Here, ∆ C , ∆ b are parameters which will eventually be high probability upper bounds for (cid:107) (cid:98) C − C (cid:107)and (cid:107) (cid:98) b − b (cid:107) 2 respectively, where (cid:107) · (cid:107) 2 denotes the spectral norm.

Original: Let σ min be the minimum singular value of the true confusion matrix C .
Revised: Let σ min be the minimum singular value of the true confusion matrix C .

Original: For the estimate (cid:98) θ , we then have the following upper bound
Revised: For the estimate (cid:98) θ , we then have the following upper bound

Original: Lemma 1 For (cid:98) θ as deﬁned in equation (3) , we have with probability at least 1 − δ that(cid:107)
Revised: Lemma 1 For (cid:98) θ as deﬁned in equation (3) , we have with probability at least 1 − δ that(cid:107)

Original: (cid:98) θ − θ (cid:107) 2 ≤ (cid:15) θ ( n p , n q , (cid:107) θ (cid:107) 2 , δ ) :
Revised: (cid:98) θ − θ (cid:107) 2 ≤ (cid:15) θ ( n p , n q , (cid:107) θ (cid:107) 2 , δ ) :

Original: = O
Revised: = O

Original: [Equation]
Revised: [Equation]

Original: The proof of this lemma can be found in Appendix A.1.
Revised: The proof of this lemma can be found in Appendix A.1.

Original: A couple of remarks are in order at this point.
Revised: A couple of remarks are in order at this point.

Original: First of all, notice that the weight estimation procedure (3) does not require a minimum sample complexity which is in the order of σ − 2 min to obtain the guarantees for BBSL .
Revised: First of all, notice that the weight estimation procedure (3) does not require a minimum sample complexity which is in the order of σ − 2 min to obtain the guarantees for BBSL .

Original: This is due to the fact that errors in the covariates are accounted for.
Revised: This is due to the fact that errors in the covariates are accounted for.

Original: In order to directly see the improvements in the upper bound of Lemma 1 compared to Theorem 3 in Lipton et al.
Revised: In order to directly see the improvements in the upper bound of Lemma 1 compared to Theorem 3 in Lipton et al.

Original: (2018), ﬁrst observe that in order to obtain their upper bound with a probability of at least 1 − δ , it is necessary that
Revised: (2018), ﬁrst observe that in order to obtain their upper bound with a probability of at least 1 − δ , it is necessary that

Original: [Equation]
Revised: [Equation]

Original: ≤ δ .
Revised: ≤ δ .

Original: As a consequence, the upper bound in Theorem 3 of Lipton et al.
Revised: As a consequence, the upper bound in Theorem 3 of Lipton et al.

Original: (2018) is bigger than
Revised: (2018) is bigger than

Original: [Equation]
Revised: [Equation]

Original: Thus Lemma 1 improves upon the previous upper bound by a factor of k .
Revised: Thus Lemma 1 improves upon the previous upper bound by a factor of k .

Original: Furthermore, as in Lipton et al.
Revised: Furthermore, as in Lipton et al.

Original: (2018), this result holds for any black box estimator h 0 which enters the bound via σ min ( C h 0 ) .
Revised: (2018), this result holds for any black box estimator h 0 which enters the bound via σ min ( C h 0 ) .

Original: We can directly see how a good choice of h 0 helps to decrease the upper bound in Lemma 1.
Revised: We can directly see how a good choice of h 0 helps to decrease the upper bound in Lemma 1.

Original: In particular, if h 0 is an ideal estimator, and the source set is balanced, C is the unit matrix with σ min = 1 /k .
Revised: In particular, if h 0 is an ideal estimator, and the source set is balanced, C is the unit matrix with σ min = 1 /k .

Original: In contrast, when the model h 0 is uncertain, the singular value σ min is close to zero.
Revised: In contrast, when the model h 0 is uncertain, the singular value σ min is close to zero.

Original: Moreover, for least square problems with Gaussian measurement errors in both input and target variables, it is standard to use regularized total least squares approaches which requires a singular
Revised: Moreover, for least square problems with Gaussian measurement errors in both input and target variables, it is standard to use regularized total least squares approaches which requires a singular

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: value decomposition.
Revised: value decomposition.

Original: Finally, our choice for the alternative estimator in Eq.
Revised: Finally, our choice for the alternative estimator in Eq.

Original: 3 with norm instead of norm squared regularization is motivated by the cases with large shifts θ , where using the squared norm may shrink the estimate (cid:98) θ too much and away from the true θ .
Revised: 3 with norm instead of norm squared regularization is motivated by the cases with large shifts θ , where using the squared norm may shrink the estimate (cid:98) θ too much and away from the true θ .

Original: 2. R EGULARIZED ESTIMATOR AND GENERALIZATION BOUND
Revised: 2. R EGULARIZED ESTIMATOR AND GENERALIZATION BOUND

Original: When a few samples from the target set are available or the label shift is mild, the estimated weights might be too uncertain to be applied.
Revised: When a few samples from the target set are available or the label shift is mild, the estimated weights might be too uncertain to be applied.

Original: We therefore propose a regularized estimator deﬁned as follows
Revised: We therefore propose a regularized estimator deﬁned as follows

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that (cid:98) w implicitly depends on λ , and β .
Revised: Note that (cid:98) w implicitly depends on λ , and β .

Original: By rewriting (cid:98) w
Revised: By rewriting (cid:98) w

Original: =
Revised: =

Original: (1 − λ ) 

Revised: (1 − λ )

Original: + λ ( 1 + (cid:98) θ ) , we see that intuitively λ closer to 1 the more reason there is to believe that 1 + (cid:98) θ is in fact the true weight.
Revised: + λ ( 1 + (cid:98) θ ) , we see that intuitively λ closer to 1 the more reason there is to believe that 1 + (cid:98) θ is in fact the true weight.

Original: Deﬁne the set G ( (cid:96), H ) =
Revised: Deﬁne the set G ( (cid:96), H ) =

Original: { g h ( x, y ) =
Revised: { g h ( x, y ) =

Original: w ( y ) (cid:96) ( h ( x ) , y ) : h ∈ H} and its Rademacher complexity measure
Revised: w ( y ) (cid:96) ( h ( x ) , y ) : h ∈ H} and its Rademacher complexity measure

Original: [Equation]
Revised: [Equation]

Original: with ξ i , ∀ i as the Rademacher random variables (see e.g. Bartlett & Mendelson (2002)).
Revised: with ξ i , ∀ i as the Rademacher random variables (see e.g. Bartlett & Mendelson (2002)).

Original: We can now state a generalization bound for the classiﬁer (cid:98) h (cid:98) w in a general hypothesis class H , which is trained on source data with the estimated weights deﬁned in equation (4).
Revised: We can now state a generalization bound for the classiﬁer (cid:98) h (cid:98) w in a general hypothesis class H , which is trained on source data with the estimated weights deﬁned in equation (4).

Original: Theorem 1 (Generalization bound for (cid:98) h (cid:98) w ) Given n p samples from the source data set and n q samples from the target set, a hypothesis class H and loss function (cid:96) , for (cid:15) G ( n p , δ ) :
Revised: Theorem 1 (Generalization bound for (cid:98) h (cid:98) w ) Given n p samples from the source data set and n q samples from the target set, a hypothesis class H and loss function (cid:96) , for (cid:15) G ( n p , δ ) :

Original: = 2 R n ( G ) + d ∞ ( q || p ) (cid:113) log(2 /δ )
Revised: = 2 R n ( G ) + d ∞ ( q || p ) (cid:113) log(2 /δ )

Original: βn p
Revised: βn p

Original: the following upper bound holds with probability at least 1 − δ
Revised: the following upper bound holds with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: The proof can be found in Appendix A.4.
Revised: The proof can be found in Appendix A.4.

Original: Additionally, we derive the analysis also for ﬁnite hypothesis classes in Appendix A.5 to provide more insight into the proof of general hypothesis classes.
Revised: Additionally, we derive the analysis also for ﬁnite hypothesis classes in Appendix A.5 to provide more insight into the proof of general hypothesis classes.

Original: The size of R n ( G ) is determined by the structure of the function class H and the loss (cid:96) .
Revised: The size of R n ( G ) is determined by the structure of the function class H and the loss (cid:96) .

Original: For example for the 0 / 1 loss, the VC dimension of H can be deployed to upper bound the Rademacher complexity.
Revised: For example for the 0 / 1 loss, the VC dimension of H can be deployed to upper bound the Rademacher complexity.

Original: The bound (5) in Theorem 1 holds for all choices of λ .
Revised: The bound (5) in Theorem 1 holds for all choices of λ .

Original: In order to exploit the possibility of choosing λ and β to have an improved accuracy depending on the sample sizes, we ﬁrst let the user deﬁne a set of shifts θ against which we want to be robust against, i.e. all shifts with (cid:107) θ (cid:107) 2 ≤ θ max .
Revised: In order to exploit the possibility of choosing λ and β to have an improved accuracy depending on the sample sizes, we ﬁrst let the user deﬁne a set of shifts θ against which we want to be robust against, i.e. all shifts with (cid:107) θ (cid:107) 2 ≤ θ max .

Original: For these shifts, we obtain the following upper bound
Revised: For these shifts, we obtain the following upper bound

Original: [Equation]
Revised: [Equation]

Original: The bound in equation (6) suggests using Algorithm 1 as our ultimate label shift correction procedure.
Revised: The bound in equation (6) suggests using Algorithm 1 as our ultimate label shift correction procedure.

Original: where for step 2 of the algorithm, we choose λ (cid:63) = 1 whenever n q ≥
Revised: where for step 2 of the algorithm, we choose λ (cid:63) = 1 whenever n q ≥

Original: [Equation]
Revised: [Equation]

Original: hereby neglecting the log factors and thus dependencies on k ) and 0 else.
Revised: neglecting the log factors and thus dependencies on k ) and 0 else.

Original: When using this rule, we obtain [Equation]
Revised: When using this rule, we obtain L ( (cid:98) h (cid:98) w ) − L ( h ∗ ) ≤ (cid:15) G ( n p , δ ) + min { θ max , (cid:15) θ ( n p , n q , θ max , δ ) } which is smaller than the unregularized bound for small n q , n p .

Original: Notice that in practice, we do not know σ min in advance so that in Algorithm 1 we need to use an estimate of σ min , which could e.g. be the minimum eigenvalue of the empirical confusion matrix (cid:98) C with an additional computational complexity of at most O ( k 3 ) .1.0 θ max 10101010n q σ
Revised: Notice that in practice, we do not know σ min in advance so that in Algorithm 1 we need to use an estimate of σ min , which could e.g. be the minimum eigenvalue of the empirical confusion matrix (cid:98) C with an additional computational complexity of at most O ( k 3 ) .

Original: [Figure]
Revised: [Figure]

Original: Figure 1 shows how the oracle thresholds vary with n q and σmin when n p is kept ﬁx.
Revised: Figure 1 shows how the oracle thresholds vary with n q and σ when n p is kept ﬁx.

Original: When the parameters are above the curves for ﬁxed n p , λ should be chosen as 1 otherwise the samples should be unweighted, i.e. λ = 0 .
Revised: When the parameters are above the curves for ﬁxed n p , λ should be chosen as 1 otherwise the samples should be unweighted, i.e. λ = 0 .

Original: This ﬁgure illustrates that when the confusion matrix has small singular values, the estimated weights should only be trusted for rather high n q and high believed shifts θ max .
Revised: This ﬁgure illustrates that when the confusion matrix has small singular values, the estimated weights should only be trusted for rather high n q and high believed shifts θ max .

Original: Although the overall statistical rate of the excess risk of the classiﬁer does not change as a function of the sample sizes, θ max could be signiﬁcantly smaller than (cid:15) θ when σ min is very small and thus the accuracy in this regime could improve.
Revised: Although the overall statistical rate of the excess risk of the classiﬁer does not change as a function of the sample sizes, θ max could be signiﬁcantly smaller than (cid:15) θ when σ min is very small and thus the accuracy in this regime could improve.

Original: Indeed we observe this to be the case empirically in Section 3.3.
Revised: Indeed we observe this to be the case empirically in Section 3.3.

Original:  EXPERIMENTS
Revised:  EXPERIMENTS

Original: In this section we illustrate the theoretical analysis by running RLLS on a variety of artiﬁcially generated shifts on the MNIST (LeCun & Cortes, 2010) and CIFAR10 (Krizhevsky & Hinton, 2009) datasets.
Revised: In this section we illustrate the theoretical analysis by running RLLS on a variety of artiﬁcially generated shifts on the MNIST (LeCun & Cortes, 2010) and CIFAR10 (Krizhevsky & Hinton, 2009) datasets.

Original: We ﬁrst randomly separate the entire dataset into two sets (source and target pool) of the same size.
Revised: We ﬁrst randomly separate the entire dataset into two sets (source and target pool) of the same size.

Original: Then we sample, unless speciﬁed otherwise, the same number of data points from each pool to form the source and target set respectively.
Revised: Then we sample, unless speciﬁed otherwise, the same number of data points from each pool to form the source and target set respectively.

Original: We chose to have equal sample sizes to allow for fair comparisons across shifts.
Revised: We chose to have equal sample sizes to allow for fair comparisons across shifts.

Original: There are various kinds of shifts which we consider in our experiments.
Revised: There are various kinds of shifts which we consider in our experiments.

Original: In general we assume one of the source or target datasets to have uniform distribution over the labels.
Revised: In general we assume one of the source or target datasets to have uniform distribution over the labels.

Original: Within the non-uniform set, we consider two types of sampling strategies in the main text: the Minority-Class Shift refers to the case where we set a ﬁxed number of classes m to have probability p < 0 .
Revised: Within the non-uniform set, we consider two types of sampling strategies in the main text: the Minority-Class Shift refers to the case where we set a ﬁxed number of classes m to have probability p < 0 .

Original: 1 , while the distribution over the rest of the classes is uniform.
Revised: 1 , while the distribution over the rest of the classes is uniform.

Original: For the Dirichlet shift , we draw a probability vector p from the
Revised: For the Dirichlet shift , we draw a probability vector p from the

Original: Dirichlet distribution with concentration parameter set to α for all classes, before including sample points which correspond to the multinomial label variable according to p .
Revised: Dirichlet distribution with concentration parameter set to α for all classes, before including sample points which correspond to the multinomial label variable according to p .

Original: Results for the tweak-one shift strategy as in Lipton et al.
Revised: Results for the tweak-one shift strategy as in Lipton et al.

Original: (2018) can be found in Section A.6.1.
Revised: (2018) can be found in Section A.6.1.

Original: After artiﬁcially shifting the label distribution in one of the source and target sets, we then follow algorithm 1, where we choose the black box predictor h 0 to be a two-layer fully connected neural network trained on the source dataset.
Revised: After artiﬁcially shifting the label distribution in one of the source and target sets, we then follow algorithm 1, where we choose the black box predictor h 0 to be a two-layer fully connected neural network trained on the source dataset.

Original: Note that any black box predictor could be employed here, though the higher the accuracy, the more likely weight estimation will be precise.
Revised: Note that any black box predictor could be employed here, though the higher the accuracy, the more likely weight estimation will be precise.

Original: In order to compute (cid:98) ω
Revised: In order to compute (cid:98) ω

Original: = 1 + (cid:98) θ in Eq.
Revised: = 1 + (cid:98) θ in Eq.

Original: (3), performing a grid search over γ on the entire set of natural numbers is not feasible.
Revised: (3), performing a grid search over γ on the entire set of natural numbers is not feasible.

Original: Instead, we call a built-in solver to directly solve the low dimensional problem min θ (cid:107) (cid:98)
Revised: Instead, we call a built-in solver to directly solve the low dimensional problem min θ (cid:107) (cid:98)

Original: Cθ − (cid:98) b (cid:107) 2 + γ 3∆ C (cid:107) θ (cid:107) 2 where the choice of γ = 0 .
Revised: Cθ − (cid:98) b (cid:107) 2 + γ 3∆ C (cid:107) θ (cid:107) 2 where the choice of γ = 0 .

Original: 01 is observed to yield a low loss in the ﬁrst line of Eq.
Revised: 01 is observed to yield a low loss in the ﬁrst line of Eq.

Original: (3) on various levels of label shift pre-computed beforehand.
Revised: (3) on various levels of label shift pre-computed beforehand.

Original: We thus treat it as a hyperparameter that can be chosen using standard cross validation methods.
Revised: We thus treat it as a hyperparameter that can be chosen using standard cross validation methods.

Original: Finally, we train a classiﬁer on the source samples weighted by (cid:98) ω , where we use a two-layer fully connected neural network for MNIST and a ResNet-18 (He et al., 2016) for CIFAR10.
Revised: Finally, we train a classiﬁer on the source samples weighted by (cid:98) ω , where we use a two-layer fully connected neural network for MNIST and a ResNet-18 (He et al., 2016) for CIFAR10.

Original: We sample 20 datasets with the label distributions for each shift parameter.
Revised: We sample 20 datasets with the label distributions for each shift parameter.

Original: to evaluate the empirical mean square estimation error (MSE) and variance of the estimated weights E (cid:107) (cid:98) w − w (cid:107) 

Revised: to evaluate the empirical mean square estimation error (MSE) and variance of the estimated weights E (cid:107) (cid:98) w − w (cid:107)

Original: and the predictive accuracy on the target set.
Revised: and the predictive accuracy on the target set.

Original: We use these measures to compare our procedure with the black box shift learning method (BBSL) in Lipton et al.
Revised: We use these measures to compare our procedure with the black box shift learning method (BBSL) in Lipton et al.

Original: Notice that although KMM methods (Zhang et al., 2013) would be another standard baseline to compare with, it is not scalable to large sample size regimes for n p , n q above n = 8000 as mentioned by Lipton et al.
Revised: Notice that although KMM methods (Zhang et al., 2013) would be another standard baseline to compare with, it is not scalable to large sample size regimes for n p , n q above n = 8000 as mentioned by Lipton et al.

Original: 3. W EIGHT E STIMATION AND PREDICTIVE PERFORMANCE FOR SOURCE SHIFT
Revised: 3. W EIGHT E STIMATION AND PREDICTIVE PERFORMANCE FOR SOURCE SHIFT

Original: In this set of experiments on the MNIST dataset, we illustrate our weight estimation and prediction performance for Minority-Class source shifts and compare it with BBSL .
Revised: In this set of experiments on the MNIST dataset, we illustrate our weight estimation and prediction performance for Minority-Class source shifts and compare it with BBSL .

Original: For this set of experiments, we set the number of data points in both source and target set to 1000 and sample from the two pools without replacement.
Revised: For this set of experiments, we set the number of data points in both source and target set to 1000 and sample from the two pools without replacement.

Original: Figure 2 illustrates the weight estimation alongside ﬁnal classiﬁcation performance for Minority-Class source shift of MNIST.
Revised: Figure 2 illustrates the weight estimation alongside ﬁnal classiﬁcation performance for Minority-Class source shift of MNIST.

Original: We created large shifts of three or more minority classes with p = 0 .
Revised: We created large shifts of three or more minority classes with p = 0 .

Original: We use a ﬁxed black-box classiﬁer that is trained on biased source data, with tweak-one ρ = 0 . 5 .
Revised: We use a ﬁxed black-box classiﬁer that is trained on biased source data, with tweak-one ρ = 0 . 5 .

Original: Observe that the MSE in weight estimation is relatively large and RLLS outperforms BBSL as the number of minority classes increases.
Revised: Observe that the MSE in weight estimation is relatively large and RLLS outperforms BBSL as the number of minority classes increases.

Original: As the shift increases the performance for all methods deteriorates.
Revised: As the shift increases the performance for all methods deteriorates.

Original: Furthermore, Figure 2 (b) illustrates how the advantage of RLLS over the unweighted classiﬁer increases as the shift increases.
Revised: Furthermore, Figure 2 (b) illustrates how the advantage of RLLS over the unweighted classiﬁer increases as the shift increases.

Original: Across all shifts, the RLLS based classiﬁer yields higher accuracy than the one based on BBSL .
Revised: Across all shifts, the RLLS based classiﬁer yields higher accuracy than the one based on BBSL .

Original: Results for smaller and larger values of p can be found in Section A.6.2.
Revised: Results for smaller and larger values of p can be found in Section A.6.2.

Original: [Figure]
Revised: [Figure]

Original: 3. W EIGHT ESTIMATION AND PREDICTIVE PERFORMANCE FOR TARGET SHIFT
Revised: 3. W EIGHT ESTIMATION AND PREDICTIVE PERFORMANCE FOR TARGET SHIFT

Original: In this section, we compare the predictive performances between a classiﬁer trained on unweighted source data and the classiﬁers trained on weighted loss obtained by the RLLS and BBSL procedure on CIFAR10.
Revised: In this section, we compare the predictive performances between a classiﬁer trained on unweighted source data and the classiﬁers trained on weighted loss obtained by the RLLS and BBSL procedure on CIFAR10.

Original: The target set is shifted using the Dirichlet shift with parameters α =
Revised: The target set is shifted using the Dirichlet shift with parameters α =

Original: The number of data points in both source and target set is 10000 .
Revised: The number of data points in both source and target set is 10000 .

Original: In the case of target shifts, larger shifts actually make the predictive task easier, such that even a constant majority class vote would give high accuracy.
Revised: In the case of target shifts, larger shifts actually make the predictive task easier, such that even a constant majority class vote would give high accuracy.

Original: However it would have zero accuracy on all but one class.
Revised: However it would have zero accuracy on all but one class.

Original: Therefore, in order to allow for a more comprehensive performance performance between the methods, we also compute the macro-averaged F-1 score by averaging the per-class quantity 2( precision · recall ) /
Revised: Therefore, in order to allow for a more comprehensive performance performance between the methods, we also compute the macro-averaged F-1 score by averaging the per-class quantity 2( precision · recall ) /

Original: ( precision + recall ) over all classes.
Revised: ( precision + recall ) over all classes.

Original: For a class i , precision is the percentage of correct predictions among all samples predicted to have label i , while recall is the proportion of correctly predicted labels over the number of samples with true label i .
Revised: For a class i , precision is the percentage of correct predictions among all samples predicted to have label i , while recall is the proportion of correctly predicted labels over the number of samples with true label i .

Original: This measure gives higher weight to the accuracies of minority classes which have no effect on the total accuracy.
Revised: This measure gives higher weight to the accuracies of minority classes which have no effect on the total accuracy.

Original: [Figure]
Revised: [Figure]

Original: Figure 3 depicts the MSE of the weight estimation (a), the corresponding performance comparison on accuracy (b) and F-1 score (c).
Revised: Figure 3 depicts the MSE of the weight estimation (a), the corresponding performance comparison on accuracy (b) and F-1 score (c).

Original: Recall that the accuracy performance for low shifts is not comparable with standard CIFAR10 benchmark results because of the overall lower sample size chosen for the comparability between shifts.
Revised: Recall that the accuracy performance for low shifts is not comparable with standard CIFAR10 benchmark results because of the overall lower sample size chosen for the comparability between shifts.

Original: We can see that in the large target shift case for α = 0 .
Revised: We can see that in the large target shift case for α = 0 .

Original: 01 , the Fscore for BBSL and the unweighted classiﬁer is rather low compared to RLLS while the accuracy is high.
Revised: 01 , the Fscore for BBSL and the unweighted classiﬁer is rather low compared to RLLS while the accuracy is high.

Original: As mentioned before, the reason for this observation and why in Figure 3 (b) the accuracy is higher when the shift is larger, is that the predictive task actually becomes easier with higher shift.
Revised: As mentioned before, the reason for this observation and why in Figure 3 (b) the accuracy is higher when the shift is larger, is that the predictive task actually becomes easier with higher shift.

Original: In conclusion, our proposed RLLS procedure yields better weight estimation and predictive performance for a whole range of source and target shifts, while the improvement is bigger for bigger shifts.
Revised: In conclusion, our proposed RLLS procedure yields better weight estimation and predictive performance for a whole range of source and target shifts, while the improvement is bigger for bigger shifts.

Original: 3. R EGULARIZED WEIGHTS IN THE LOW SAMPLE REGIME FOR SOURCE SHIFT
Revised: 3. R EGULARIZED WEIGHTS IN THE LOW SAMPLE REGIME FOR SOURCE SHIFT

Original: In the following, we present the average accuracy of RLLS in Figure 4 as a function of the number of target samples n q for different values of λ for small n q .
Revised: In the following, we present the average accuracy of RLLS in Figure 4 as a function of the number of target samples n q for different values of λ for small n q .

Original: Here we ﬁx the sample size in the source set to n p = 1000 and investigate a Minority-Class source shift with ﬁxed p = 0 .
Revised: Here we ﬁx the sample size in the source set to n p = 1000 and investigate a Minority-Class source shift with ﬁxed p = 0 .

Original: 01 and ﬁve minority classes.
Revised: 01 and ﬁve minority classes.

Original: A motivation to use intermediate λ is discussed in Section 2.2, as λ in equation (4) may be chosen according to θ max , σ min .
Revised: A motivation to use intermediate λ is discussed in Section 2.2, as λ in equation (4) may be chosen according to θ max , σ min .

Original: In practice, since θ max is just an upper bound on the true amount of shift (cid:107) θ (cid:107) 2 , in some cases λ should in fact ideally be 0 when
Revised: In practice, since θ max is just an upper bound on the true amount of shift (cid:107) θ (cid:107) 2 , in some cases λ should in fact ideally be 0 when

Original: [Figure]
Revised: [Figure]

Original: Thus for target sample sizes n q that are a little bit above the threshold (depending on the certainty of the belief how close to θ max the norm of the shift is believed to be), it could be sensible to use an intermediate value λ ∈ (0 , 1) .
Revised: Thus for target sample sizes n q that are a little bit above the threshold (depending on the certainty of the belief how close to θ max the norm of the shift is believed to be), it could be sensible to use an intermediate value λ ∈ (0 , 1) .

Original: [Figure]
Revised: [Figure]

Original:  Figure 4 suggests that unweighted samples (red) yield the best classiﬁer for very few samples n q , while for 10 ≤ n q ≤ 500 an intermediate λ ∈ (0 , 1) (purple) has the highest accuracy and for n q > 1000 , the weight estimation is certain enough for the fully weighted classiﬁer (yellow) to have the best performance (see also the corresponding data points in Figure 2).
Revised: 0 with ρ = 0 . 8 . Figure 4 suggests that unweighted samples (red) yield the best classiﬁer for very few samples n q , while for 10 ≤ n q ≤ 500 an intermediate λ ∈ (0 , 1) (purple) has the highest accuracy and for n q > 1000 , the weight estimation is certain enough for the fully weighted classiﬁer (yellow) to have the best performance (see also the corresponding data points in Figure 2).

Original: The unweighted BBSL classiﬁer is also shown for completeness.
Revised: The unweighted BBSL classiﬁer is also shown for completeness.

Original: We can conclude that regularizing the inﬂuence of the estimated weights allows us to adjust to the uncertainty on importance weights and generalize well for a wide range of target sample sizes.
Revised: We can conclude that regularizing the inﬂuence of the estimated weights allows us to adjust to the uncertainty on importance weights and generalize well for a wide range of target sample sizes.

Original: Furthermore, the different plots in Figure 4 correspond to black-box predictors h 0 for weight estimation which are trained on more or less corrupted data, i.e. have a better or worse conditioned confusion matrix.
Revised: Furthermore, the different plots in Figure 4 correspond to black-box predictors h 0 for weight estimation which are trained on more or less corrupted data, i.e. have a better or worse conditioned confusion matrix.

Original: The fully weighted methods with λ = 1 achieve the best performance faster with a better trained black-box classiﬁer (a), while it takes longer for it to improve with a corrupted one (c).
Revised: The fully weighted methods with λ = 1 achieve the best performance faster with a better trained black-box classiﬁer (a), while it takes longer for it to improve with a corrupted one (c).

Original: Furthermore, this reﬂects the relation between eigenvalue of confusion matrix σ min and target sample size n q in Theorem 1.
Revised: Furthermore, this reﬂects the relation between eigenvalue of confusion matrix σ min and target sample size n q in Theorem 1.

Original: In other words, we need more samples from the target data to compensate a bad predictor in weight estimation.
Revised: In other words, we need more samples from the target data to compensate a bad predictor in weight estimation.

Original: So the generalization error decreases faster with an increasing number of samples for good predictors.
Revised: So the generalization error decreases faster with an increasing number of samples for good predictors.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The task of being robust against distributional shifts has been widely studied in the literature.
Revised: The task of being robust against distributional shifts has been widely studied in the literature.

Original: One line of work focuses on distributional robustness (see e.g. Esfahani & Kuhn; Namkoong & Duchi (2016)) without explicitly modeling the shift by making assumptions on the distribution.
Revised: One line of work focuses on distributional robustness (see e.g. Esfahani & Kuhn; Namkoong & Duchi (2016)) without explicitly modeling the shift by making assumptions on the distribution.

Original: Our theoretical results adapt to any shift θ .
Revised: Our theoretical results adapt to any shift θ .

Original: However, these methods achieve robustness only against tiny changes in the distribution and are often impractical for large-scale high dimensional data.
Revised: However, these methods achieve robustness only against tiny changes in the distribution and are often impractical for large-scale high dimensional data.

Original: Another line of work has addressed the causal setting where the conditional distribution of Y given
Revised: Another line of work has addressed the causal setting where the conditional distribution of Y given

Original: X is assumed to be constant.
Revised: X is assumed to be constant.

Original: This assumption is sensible when there is reason to believe that there is a true optimal mapping from X to Y which does not change if the distribution of X changes.
Revised: This assumption is sensible when there is reason to believe that there is a true optimal mapping from X to Y which does not change if the distribution of X changes.

Original: In our example, if a given set of symptoms correctly predicts a disease for the “right reasons”, the predictive function should be the same, no matter which country you are moving to.
Revised: In our example, if a given set of symptoms correctly predicts a disease for the “right reasons”, the predictive function should be the same, no matter which country you are moving to.

Original: Among the various methods to correct for covariate shift, the majority uses the concept of importance weights q ( x ) /p
Revised: Among the various methods to correct for covariate shift, the majority uses the concept of importance weights q ( x ) /p

Original: ( x ) (Zadrozny, 2004; Cortes et al., 2010; Cortes & Mohri, 2014; Shimodaira, 2000), which are unknown but can be estimated for example via kernel embeddings (Huang et al., 2007; Gretton et al., 2009; 2012;
Revised: ( x ) (Zadrozny, 2004; Cortes et al., 2010; Cortes & Mohri, 2014; Shimodaira, 2000), which are unknown but can be estimated for example via kernel embeddings (Huang et al., 2007; Gretton et al., 2009; 2012;

Original: Zhang et al., 2013; Zaremba et al., 2013) or by learning a binary discriminative classiﬁer between source and target (Lopez-Paz & Oquab, 2016; Liu et al., 2017).
Revised: Zhang et al., 2013; Zaremba et al., 2013) or by learning a binary discriminative classiﬁer between source and target (Lopez-Paz & Oquab, 2016; Liu et al., 2017).

Original: Finally, Sanderson & Scott (2014); Ramaswamy et al.
Revised: Finally, Sanderson & Scott (2014); Ramaswamy et al.

Original: (2016) formulate the label shift problem as a mixture of the class conditional covariate distributions with unknown mixture weights.
Revised: (2016) formulate the label shift problem as a mixture of the class conditional covariate distributions with unknown mixture weights.

Original: Under the pairwise mutual irreducibility (Scott et al., 2013) assumption on the class conditional covariate distributions, they deploy the Neyman-Pearson criterion (Blanchard et al., 2010) to estimate the class distribution q ( y ) .
Revised: Under the pairwise mutual irreducibility (Scott et al., 2013) assumption on the class conditional covariate distributions, they deploy the Neyman-Pearson criterion (Blanchard et al., 2010) to estimate the class distribution q ( y ) .

Original: Common issues shared by these methods is that they either result in a massive computational burden for large scale problems or cannot be deployed for neural networks.
Revised: Common issues shared by these methods is that they either result in a massive computational burden for large scale problems or cannot be deployed for neural networks.

Original: Despite the wide applicability of label shift, approaches with global guarantees in high dimensional data regimes remain under-explored.
Revised: Despite the wide applicability of label shift, approaches with global guarantees in high dimensional data regimes remain under-explored.

Original: The correction of label shift mainly requires to estimate the importance weights q ( y ) /p
Revised: The correction of label shift mainly requires to estimate the importance weights q ( y ) /p

Original: ( y ) over the labels which typically live in a very low-dimensional space.
Revised: ( y ) over the labels which typically live in a very low-dimensional space.

Original: Bayesian approaches are studied when a prior over the marginal label distribution is assumed (Storkey, 2009; Chan & Ng, 2005).
Revised: Bayesian approaches are studied when a prior over the marginal label distribution is assumed (Storkey, 2009; Chan & Ng, 2005).

Original: However, these methods often require explicitly computing the posterior distribution of y and suffer from the curse of dimensionality.
Revised: However, these methods often require explicitly computing the posterior distribution of y and suffer from the curse of dimensionality.

Original: Recently, for the label shift setting,
Revised: Recently, for the label shift setting,

Original: Lipton et al.
Revised: Lipton et al.

Original: (2018) proposed Black Box Shift Estimation ( BBSL ) which can be applied to large scale data.
Revised: (2018) proposed Black Box Shift Estimation ( BBSL ) which can be applied to large scale data.

Original: This approach is related to Buck et al.
Revised: This approach is related to Buck et al.

Original: (1966); Forman (2008);
Revised: (1966); Forman (2008);

Original: Saerens et al.
Revised: Saerens et al.

Original: in the low dimensional setting but lacks guarantees for the excess risk.
Revised: in the low dimensional setting but lacks guarantees for the excess risk.

Original: On the other hand, generalization theory, usually used to bound the excess loss, has historically been developed for the case when P = Q (see e.g. Vapnik (1999)
Revised: On the other hand, generalization theory, usually used to bound the excess loss, has historically been developed for the case when P = Q (see e.g. Vapnik (1999)

Original: ; Bartlett & Mendelson (2002);
Revised: ; Bartlett & Mendelson (2002);

Original: Kakade et al.
Revised: Kakade et al.

Original: ; Wainwright (2019)).
Revised: ; Wainwright (2019)).

Original: Ben-David et al.
Revised: Ben-David et al.

Original: (2010) provides theoretical analysis and generalization guarantees for distribution shifts when the H-divergence between joint distributions is considered, whereas Crammer et al.
Revised: (2010) provides theoretical analysis and generalization guarantees for distribution shifts when the H-divergence between joint distributions is considered, whereas Crammer et al.

Original: (2008) proves generalization bounds for learning from multiple sources.
Revised: (2008) proves generalization bounds for learning from multiple sources.

Original: For the covariate shift setting, Cortes et al.
Revised: For the covariate shift setting, Cortes et al.

Original: (2010) provides a generalization bound when q ( x ) /p
Revised: (2010) provides a generalization bound when q ( x ) /p

Original: ( x ) is known which however does not apply in practice.
Revised: ( x ) is known which however does not apply in practice.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this work, we establish the ﬁrst generalization guarantee for the label shift setting and propose an importance weighting procedure for which no prior knowledge of q ( y ) /p
Revised: In this work, we establish the ﬁrst generalization guarantee for the label shift setting and propose an importance weighting procedure for which no prior knowledge of q ( y ) /p

Original: ( y ) is required.
Revised: ( y ) is required.

Original: Although
Revised: Although

Original: RLLS is inspired by BBSL , it leads to a more robust importance weight estimator as well as generalization guarantees in particular for the small sample regime, which BBSL does not allow for.
Revised: RLLS is inspired by BBSL , it leads to a more robust importance weight estimator as well as generalization guarantees in particular for the small sample regime, which BBSL does not allow for.

Original: RLLS is also equipped with a sample-size-dependent regularization technique and further improves the classiﬁer in both regimes.
Revised: RLLS is also equipped with a sample-size-dependent regularization technique and further improves the classiﬁer in both regimes.

Original: We consider this work a necessary step in the direction of solving shifts of this type, although the label shift assumption itself might be too simpliﬁed in the real world.
Revised: We consider this work a necessary step in the direction of solving shifts of this type, although the label shift assumption itself might be too simpliﬁed in the real world.

Original: In future work, we plan to also study the setting when it is slightly violated.
Revised: In future work, we plan to also study the setting when it is slightly violated.

Original: For instance, x in practice cannot be solely explained by the wanted label y , but may also depend on attributes z which might not be observable.
Revised: For instance, x in practice cannot be solely explained by the wanted label y , but may also depend on attributes z which might not be observable.

Original: In the disease prediction task for example, the symptoms might not only depend on the disease but also on the country and living conditions of its population.
Revised: In the disease prediction task for example, the symptoms might not only depend on the disease but also on the country and living conditions of its population.

Original: In such a case, the label shift assumption only holds in a slightly modiﬁed sense, i.e. P ( X | Y
Revised: In such a case, the label shift assumption only holds in a slightly modiﬁed sense, i.e. P ( X | Y

Original: =
Revised: =

Original: y, Z = z ) =
Revised: y, Z = z ) =

Original: Q ( X | Y
Revised: Q ( X | Y

Original: =
Revised: =

Original: y, Z = z ) .
Revised: y, Z = z ) .

Original: If the attributes
Revised: If the attributes

Original: Z are observed, then our framework can readily be used to perform importance weighting.
Revised: Z are observed, then our framework can readily be used to perform importance weighting.

Original: Furthermore, it is not clear whether the ﬁnal predictor is in fact “better” or more robust to shifts just because it achieves a better target accuracy than a vanilla unweighted estimator.
Revised: Furthermore, it is not clear whether the ﬁnal predictor is in fact “better” or more robust to shifts just because it achieves a better target accuracy than a vanilla unweighted estimator.

Original: In fact, there is reason to believe that under certain shift scenarios, the predictor might learn to use spurious correlations to boost accuracy.
Revised: In fact, there is reason to believe that under certain shift scenarios, the predictor might learn to use spurious correlations to boost accuracy.

Original: Finding a procedure which can both learn a robust model and achieve high accuracies on new target sets remains to be an ongoing challenge.
Revised: Finding a procedure which can both learn a robust model and achieve high accuracies on new target sets remains to be an ongoing challenge.

Original: Moreover, the current choice of regularization depends on the number of samples rather than data driven regularization which is more desirable.
Revised: Moreover, the current choice of regularization depends on the number of samples rather than data driven regularization which is more desirable.

Original: An interesting direction towards active learning for the same disease-symptoms scenario is when we also have an expert for diagnosing a limited number of patients in country
Revised: An interesting direction towards active learning for the same disease-symptoms scenario is when we also have an expert for diagnosing a limited number of patients in country

Original: B. Now the question is which patients would be most "useful" to diagnose to obtain a high accuracy on the entire target set?
Revised: B. Now the question is which patients would be most "useful" to diagnose to obtain a high accuracy on the entire target set?

Original: Furthermore, in the case of high risk, we might be able to send some of the patients to country
Revised: Furthermore, in the case of high risk, we might be able to send some of the patients to country

Original: A for further medical diagnosis or treatment, up to some varying cost.
Revised: A for further medical diagnosis or treatment, up to some varying cost.

Original: We plan to extend the current framework to the active learning setting where we actively query the label of certain x ’s (Beygelzimer et al., 2009) as well as the cost-sensitive setting where we also consider the cost of querying labels (Krishnamurthy et al., 2017).
Revised: We plan to extend the current framework to the active learning setting where we actively query the label of certain x ’s (Beygelzimer et al., 2009) as well as the cost-sensitive setting where we also consider the cost of querying labels (Krishnamurthy et al., 2017).

Original: A P ROOFS A. P ROOF OF L EMMA
Revised: A P ROOFS A. P ROOF OF L EMMA

Original: From Thm.
Revised: From Thm.

Original: 3.7 in (Pires & Szepesvári, 2012)
Revised: 3.7 in (Pires & Szepesvári, 2012)

Original: we know that for (cid:98) θ as deﬁned in equation (3), if with probability at least 1 − δ , (cid:107) (cid:98) C − C (cid:107) 2 ≤ ∆ C and (cid:107) (cid:98) b − b (cid:107) 2 ≤ ∆ b hold simultaneously, then
Revised: we know that for (cid:98) θ as deﬁned in equation (3), if with probability at least 1 − δ , (cid:107) (cid:98) C − C (cid:107) 2 ≤ ∆ C and (cid:107) (cid:98) b − b (cid:107) 2 ≤ ∆ b hold simultaneously, then

Original: [Equation]
Revised: [Equation]

Original: 8) where we use the shorthand Υ( θ (cid:48) )
Revised: 7) where we use the shorthand Υ( θ (cid:48) )

Original: = (cid:107)
Revised: = (cid:107)

Original: Cθ (cid:48) − b (cid:107) 2 .
Revised: Cθ (cid:48) − b (cid:107) 2 .

Original: We can get an upper bound on the right hand side of (8) is the inﬁmum by simply choosing a feasible θ (cid:48)
Revised: We can get an upper bound on the right hand side of (7) is the inﬁmum by simply choosing a feasible θ (cid:48)

Original: = θ .
Revised: = θ .

Original: We then have (cid:107)
Revised: We then have (cid:107)

Original: Cθ − b (cid:107) 

Revised: Cθ − b (cid:107)

Original: =
Revised: =

Original: and hence
Revised: and hence

Original: [Equation]
Revised: [Equation]

Original: as a consequence,
Revised: as a consequence,

Original: [Equation]
Revised: [Equation]

Original: Since (cid:107) C (cid:16) (cid:98) θ − θ (cid:17) (cid:107) 2 ≥ σ min ( C ) (cid:107) (cid:98) θ − θ (cid:107) 2 by deﬁnition of the minimum singular value, we thus have
Revised: Since (cid:107) C (cid:16) (cid:98) θ − θ (cid:17) (cid:107) 2 ≥ σ min ( C ) (cid:107) (cid:98) θ − θ (cid:107) 2 by deﬁnition of the minimum singular value, we thus have

Original: [Equation]
Revised: [Equation]

Original: Let us ﬁrst notice that
Revised: Let us ﬁrst notice that

Original: [Equation]
Revised: [Equation]

Original: The mathematical deﬁnition of the ﬁnite sample estimates (cid:98) C h , (cid:98) b h (in matrix and vector representation) with respect to some hypothesis h are as follows
Revised: The mathematical deﬁnition of the ﬁnite sample estimates (cid:98) C h , (cid:98) b h (in matrix and vector representation) with respect to some hypothesis h are as follows

Original: [Equation]
Revised: [Equation]

Original: where m = | D q |
Revised: where m = | D q |

Original: and I is the indicator function.
Revised: and I is the indicator function.

Original: C h , b h can equivalently be expressed with the population over P for C h and over Q for b h respectively.
Revised: C h , b h can equivalently be expressed with the population over P for C h and over Q for b h respectively.

Original: We now use the following concentration
Revised: We now use the following concentration

Original: Lemmas to bound the estimation errors of (cid:98) C, (cid:98) b where we drop the subscript h for ease of notation.
Revised: Lemmas to bound the estimation errors of (cid:98) C, (cid:98) b where we drop the subscript h for ease of notation.

Original: Lemma 2 (Concentration of measurement matrix (cid:98) C )
Revised: Lemma 2 (Concentration of measurement matrix (cid:98) C )

Original: For ﬁnite sample estimate (cid:98) C we have
Revised: For ﬁnite sample estimate (cid:98) C we have

Original: [Equation]
Revised: [Equation]

Original: with probability at least (1 − δ ) .
Revised: with probability at least (1 − δ ) .

Original: Lemma 3 (Concentration of label measurements)
Revised: Lemma 3 (Concentration of label measurements)

Original: For the ﬁnite sample estimate (cid:98) b with respect to any hypothesis h
Revised: For the ﬁnite sample estimate (cid:98) b with respect to any hypothesis h

Original: it holds that
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 2 δ .
Revised: with probability at least 1 − 2 δ .

Original: By Lemma.
Revised: By Lemma.

Original: 2 for concentration of C and Lemma.
Revised: 2 for concentration of C and Lemma.

Original: 3 for concentration of b we now have with probability at least 1 − δ
Revised: 3 for concentration of b we now have with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: which, considering that O ( 1 √ n ) dominates O ( 1 n ) , yields the statement of the Lemma 1.
Revised: which, considering that O ( 1 √ n ) dominates O ( 1 n ) , yields the statement of the Lemma 1.

Original: A. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: We prove this lemma using the theorem 1.4[Matrix Bernstein] and Dilations technique from Tropp (2012).
Revised: We prove this lemma using the theorem 1.4[Matrix Bernstein] and Dilations technique from Tropp (2012).

Original: We can rewrite C h
Revised: We can rewrite C h

Original: = E ( x,y ) ∼ P (cid:2)
Revised: = E ( x,y ) ∼ P (cid:2)

Original: e h ( x ) e (cid:62) y (cid:3) where e ( i ) is the one-hot-encoding of index i .
Revised: e h ( x ) e (cid:62) y (cid:3) where e ( i ) is the one-hot-encoding of index i .

Original: Consider a ﬁnite sequence { Ψ( i ) } of independent random matrices with dimension k .
Revised: Consider a ﬁnite sequence { Ψ( i ) } of independent random matrices with dimension k .

Original: By dilations, lets construct another sequence of self-adjoint random matrices of { (cid:101) Ψ
Revised: By dilations, lets construct another sequence of self-adjoint random matrices of { (cid:101) Ψ

Original: ( i ) } of dimension 2 k ,
Revised: ( i ) } of dimension 2 k ,

Original: such that for all
Revised: such that for all

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: therefore,
Revised: therefore,

Original: [Equation]
Revised: [Equation]

Original: which results in (cid:107) (cid:101)
Revised: which results in (cid:107) (cid:101)

Original: Ψ
Revised: Ψ

Original: ( i ) (cid:107) 

Revised: ( i ) (cid:107)

Original: = (cid:107) Ψ( i ) (cid:107) 2 .
Revised: = (cid:107) Ψ( i ) (cid:107) 2 .

Original: The dilation technique translates the initial sequence of random matrices to the sequence of random self-adjoint matrices where we can apply the Matrix Bernstein theorem which states that, for a ﬁnite sequence of i.i.d.
Revised: The dilation technique translates the initial sequence of random matrices to the sequence of random self-adjoint matrices where we can apply the Matrix Bernstein theorem which states that, for a ﬁnite sequence of i.i.d.

Original: self-adjoint matrices (cid:101) Ψ( i ) , such that, almost surely ∀ i, E (cid:104)(cid:101) Ψ( i ) (cid:105) = 0 and (cid:107) (cid:101) Ψ
Revised: self-adjoint matrices (cid:101) Ψ( i ) , such that, almost surely ∀ i, E (cid:104)(cid:101) Ψ( i ) (cid:105) = 0 and (cid:107) (cid:101) Ψ

Original: ( i ) (cid:107) ≤ R , then for all t ≥ 0 ,
Revised: ( i ) (cid:107) ≤ R , then for all t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: t with probability at least 1 − δ where (cid:37) 2 := (cid:107) E (cid:104)(cid:101) Ψ 2 ( i ) (cid:105) (cid:107) 2 , ∀ i which is also (cid:37) 

Revised: t with probability at least 1 − δ where (cid:37) 2 := (cid:107) E (cid:104)(cid:101) Ψ 2 ( i ) (cid:105) (cid:107) 2 , ∀ i which is also (cid:37)

Original: = (cid:107) E (cid:2)
Revised: = (cid:107) E (cid:2)

Original: Ψ 2 ( i ) (cid:3) (cid:107) 2 , ∀ i due to Eq. 9.
Revised: Ψ 2 ( i ) (cid:3) (cid:107) 2 , ∀ i due to Eq. 8.

Original: Therefore, thanks to the dilation trick and theorem 1.4[Matrix Bernstein] in Tropp (2012),
Revised: Therefore, thanks to the dilation trick and theorem 1.4[Matrix Bernstein] in Tropp (2012),

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ .
Revised: with probability at least 1 − δ .

Original: Now, by plugging in Ψ( i ) =
Revised: Now, by plugging in Ψ( i ) =

Original: e h ( x ( i ))
Revised: e h ( x ( i ))

Original: e (cid:62) y
Revised: e (cid:62) y

Original: ( i ) − C , we have E (cid:104)(cid:101) Ψ
Revised: ( i ) − C , we have E (cid:104)(cid:101) Ψ

Original: ( i ) (cid:105)
Revised: ( i ) (cid:105)

Original: = 0 .
Revised: = 0 .

Original: Together with (cid:107) (cid:101) Ψ
Revised: Together with (cid:107) (cid:101) Ψ

Original: ( i ) (cid:107) ≤ 2 as well as (cid:37) 

Revised: ( i ) (cid:107) ≤ 2 as well as (cid:37)

Original: = (cid:107) E (cid:2)
Revised: = (cid:107) E (cid:2)

Original: Ψ 2 ( i ) (cid:3) (cid:107) 2 = 1 and setting t = n , we have
Revised: Ψ 2 ( i ) (cid:3) (cid:107) 2 = 1 and setting t = n , we have

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: The proof of this lemma is mainly based on a special case of and appreared at proposition 6 in
Revised: The proof of this lemma is mainly based on a special case of and appreared at proposition 6 in

Original: Azizzadenesheli et al.
Revised: Azizzadenesheli et al.

Original: Lemma F.1 in Anandkumar et al.
Revised: Lemma F.1 in Anandkumar et al.

Original: and Proposition 19 of Hsu et al.
Revised: and Proposition 19 of Hsu et al.

Original: Analogous to the previous section we can rewrite b h
Revised: Analogous to the previous section we can rewrite b h

Original: = E ( x,y ) ∈ Q [ e h ( x ) ] − E ( x
Revised: = E ( x,y ) ∈ Q [ e h ( x ) ] − E ( x

Original: ,y ) ∈ P [ e h ( x ) ] where e ( i ) is the one-hot-encoding of index i .
Revised: ,y ) ∈ P [ e h ( x ) ] where e ( i ) is the one-hot-encoding of index i .

Original: Note that (dropping the subscript h ) we have
Revised: Note that (dropping the subscript h ) we have

Original: [Equation]
Revised: [Equation]

Original: We now bound both estimates of probability vectors separately.
Revised: We now bound both estimates of probability vectors separately.

Original: Consider a ﬁxed multinomial distribution characterized with probability vector of ς ∈ ∆ k − 1 where ∆ k − 1 is a k − 1 dimensional simplex.
Revised: Consider a ﬁxed multinomial distribution characterized with probability vector of ς ∈ ∆ k − 1 where ∆ k − 1 is a k − 1 dimensional simplex.

Original: Further, consider t realization of this multinomial distribution { ς ( i ) } ti =1 where ς ( i ) is the one-hot-encoding of the i ’th sample.
Revised: Further, consider t realization of this multinomial distribution { ς ( i ) } ti =1 where ς ( i ) is the one-hot-encoding of the i ’th sample.

Original: Consider the empirical estimate mean of this distribution through empirical average of the samples; (cid:98) ς
Revised: Consider the empirical estimate mean of this distribution through empirical average of the samples; (cid:98) ς

Original: = 1 t (cid:80) ( i ) t
Revised: = 1 t (cid:80) ( i ) t

Original: ς
Revised: ς

Original: ( i ) , then
Revised: ( i ) , then

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ .
Revised: with probability at least 1 − δ .

Original: By plugging in ς = q h , (cid:98) ς = (cid:98) q h with t = n q and ﬁnally { ς ( i ) } n q i =1 = { e h ( x ( i )) }
Revised: By plugging in ς = q h , (cid:98) ς = (cid:98) q h with t = n q and ﬁnally { ς ( i ) } n q i =1 = { e h ( x ( i )) }

Original: ( i ) n q and equivalently for p
Revised: ( i ) n q and equivalently for p

Original: h we obtain;
Revised: h we obtain;

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 2 δ , therefore;
Revised: with probability at least 1 − 2 δ , therefore;

Original: [Equation]
Revised: [Equation]

Original: resulting in the statement in the Lemma 3.
Revised: resulting in the statement in the Lemma 3.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: We want to ultimately bound |L ( (cid:98) h (cid:98) w ) − L ( h (cid:63) ) | .
Revised: We want to ultimately bound |L ( (cid:98) h (cid:98) w ) − L ( h (cid:63) ) | .

Original: By addition and subtraction we have
Revised: By addition and subtraction we have

Original: [Equation]
Revised: [Equation]

Original: where n = βn p
Revised: where n = βn p

Original: and we used optimality of (cid:98) h (cid:98) w .
Revised: and we used optimality of (cid:98) h (cid:98) w .

Original: Here (a) is the weight estimation error and (b) is the ﬁnite sample error.
Revised: Here (a) is the weight estimation error and (b) is the ﬁnite sample error.

Original: Uniform law for bounding (b) We bound (b) using standard results for uniform laws for uniformly bounded functions which holds since (cid:107) w (cid:107) ∞ ≤ d ∞ ( q || p ) and (cid:107) (cid:96) (cid:107) ∞ ≤ 1 .
Revised: Uniform law for bounding (b) We bound (b) using standard results for uniform laws for uniformly bounded functions which holds since (cid:107) w (cid:107) ∞ ≤ d ∞ ( q || p ) and (cid:107) (cid:96) (cid:107) ∞ ≤ 1 .

Original: Since | w ( y ) (cid:96) ( h ( x ) , y ) | ≤ d ∞ ( q || p ) , ∀ x, y ∈
Revised: Since | w ( y ) (cid:96) ( h ( x ) , y ) | ≤ d ∞ ( q || p ) , ∀ x, y ∈

Original: X
Revised: X

Original: × Y , by deploying the McDiarmid’s inequality we then obtain that
Revised: × Y , by deploying the McDiarmid’s inequality we then obtain that

Original: [Equation]
Revised: [Equation]

Original: n where G ( (cid:96), H )
Revised: n where G ( (cid:96), H )

Original: = { g h ( x, y ) =
Revised: = { g h ( x, y ) =

Original: w ( y ) (cid:96) ( h ( x ) , y ) : h ∈ H} and the Rademacher complexity is deﬁned as
Revised: w ( y ) (cid:96) ( h ( x ) , y ) : h ∈ H} and the Rademacher complexity is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: (cid:80) ni
Revised: (cid:80) ni

Original: [Equation]
Revised: [Equation]

Original: σ
Revised: σ

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: (cid:96) ( x
Revised: (cid:96) ( x

Original: [Equation]
Revised: [Equation]

Original: h ( y i )
Revised: h ( y i )

Original: [Equation]
Revised: [Equation]

Original: ] (cid:3) .
Revised: ] (cid:3) .

Original: of the hypothesis class H (see for example Percy Liang notes on Statistical Learning Theory and chapter 4 in Wainwright (2019))
Revised: of the hypothesis class H (see for example Percy Liang notes on Statistical Learning Theory and chapter 4 in Wainwright (2019))

Original: Bounding term (a) Remember that k = |Y| is the cardinality of the ﬁnite domain of Y , or the number of classes.
Revised: Bounding term (a) Remember that k = |Y| is the cardinality of the ﬁnite domain of Y , or the number of classes.

Original: Let us deﬁne (cid:101) (cid:96)
Revised: Let us deﬁne (cid:101) (cid:96)

Original: ∈
Revised: ∈

Original: R k with (cid:101) (cid:96) j = (cid:80) ni
Revised: R k with (cid:101) (cid:96) j = (cid:80) ni

Original: =

Revised: =

Original: I y ( i )=
Revised: I y ( i )=

Original: j (cid:96) ( y ( i ) , h ( x ( i ))) .
Revised: j (cid:96) ( y ( i ) , h ( x ( i ))) .

Original: Notice that by deﬁnition (cid:107) (cid:101) (cid:96) (cid:107) 1 ≤ n and (cid:107) (cid:101) (cid:96) (cid:107) ∞ ≤ n from which it follows by Hoelder’s inequality that (cid:107) (cid:101) (cid:96) (cid:107) 2 ≤ n .
Revised: Notice that by deﬁnition (cid:107) (cid:101) (cid:96) (cid:107) 1 ≤ n and (cid:107) (cid:101) (cid:96) (cid:107) ∞ ≤ n from which it follows by Hoelder’s inequality that (cid:107) (cid:101) (cid:96) (cid:107) 2 ≤ n .

Original: Furthermore, we slightly abuse notation and use w to denote the k -dimensional vector with w i = w ( i ) .
Revised: Furthermore, we slightly abuse notation and use w to denote the k -dimensional vector with w i = w ( i ) .

Original: Therefore, for all h we have via the Cauchy Schwarz inequality that
Revised: Therefore, for all h we have via the Cauchy Schwarz inequality that

Original: [Equation]
Revised: [Equation]

Original: It then follows by Lemma 1 that
Revised: It then follows by Lemma 1 that

Original: [Equation]
Revised: [Equation]

Original: A. G ENERALIZATION FOR FINITE HYPOTHESIS CLASSES
Revised: A. G ENERALIZATION FOR FINITE HYPOTHESIS CLASSES

Original: For ﬁnite hypothesis classes, one may bound (b) in (10) using Bernstein’s inequality.
Revised: For ﬁnite hypothesis classes, one may bound (b) in (9) using Bernstein’s inequality.

Original: We bound (b) by ﬁrst noting that w ( Y ) (cid:96) ( Y, h ( X )) satisﬁes the Bernstein conditions so that Bernstein’s inequality holds
Revised: We bound (b) by ﬁrst noting that w ( Y ) (cid:96) ( Y, h ( X )) satisﬁes the Bernstein conditions so that Bernstein’s inequality holds

Original: [Equation]
Revised: [Equation]

Original: by deﬁnition.
Revised: by deﬁnition.

Original: Because we assume (cid:96) ≤ 1 , we directly have
Revised: Because we assume (cid:96) ≤ 1 , we directly have

Original: [Equation]
Revised: [Equation]

Original: Since we have a bound on the second moment of weighted loss while its ﬁrst moment is L ( h ) we can apply Bernstein’s inequality to obtain for any ﬁxed h that
Revised: Since we have a bound on the second moment of weighted loss while its ﬁrst moment is L ( h ) we can apply Bernstein’s inequality to obtain for any ﬁxed h that

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: For the uniform law for ﬁnite hypothesis classes make the union bound on all the hypotheses;
Revised: For the uniform law for ﬁnite hypothesis classes make the union bound on all the hypotheses;

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: The second moment of the importance weighted loss E P (cid:2) ω ( Y ) 2 (cid:96) 2 ( h ( X ) , Y ) (cid:3) , given a h ∈ H (cid:48) can be bounded for general α ≥ 0 , potentially leading to smaller values than d ( q || p ) :
Revised: The second moment of the importance weighted loss E P (cid:2) ω ( Y ) 2 (cid:96) 2 ( h ( X ) , Y ) (cid:3) , given a h ∈ H (cid:48) can be bounded for general α ≥ 0 , potentially leading to smaller values than d ( q || p ) :

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality follows from Hölder’s inequality, the second one follows from Jensen’s inequality and the fact that the loss is in [0 , 1] as well as the fact that the exponentiation function is convex in this region.
Revised: where the ﬁrst inequality follows from Hölder’s inequality, the second one follows from Jensen’s inequality and the fact that the loss is in [0 , 1] as well as the fact that the exponentiation function is convex in this region.

Original: Moreover, since 1 + 1 α ≥ 1 and upper bound for the loss square, l ( · , · ) 2 ≤ 1 , then;
Revised: Moreover, since 1 + 1 α ≥ 1 and upper bound for the loss square, l ( · , · ) 2 ≤ 1 , then;

Original: [Equation]
Revised: [Equation]

Original: which gives bound on the second moment of weighted loss.
Revised: which gives bound on the second moment of weighted loss.

Original: A. M ORE EXPERIMENTAL RESULTS
Revised: A. M ORE EXPERIMENTAL RESULTS

Original: This section contains more experiments that provide more insights about in which settings the advantage of using RLLS vs. BBSL are more or less pronounced.
Revised: This section contains more experiments that provide more insights about in which settings the advantage of using RLLS vs. BBSL are more or less pronounced.

Original: A.6. CIFAR10 E XPERIMENTS UNDER TWEAK - ONE SHIFT AND D IRICHLET SHIFT
Revised: A.6. CIFAR10 E XPERIMENTS UNDER TWEAK - ONE SHIFT AND D IRICHLET SHIFT

Original: Here we compare weight estimation performance between RLLS and BBSL for different types of shifts including the Tweak-one Shift , for which we randomly choose one class,
Revised: Here we compare weight estimation performance between RLLS and BBSL for different types of shifts including the Tweak-one Shift , for which we randomly choose one class,

Original: e.g. i and set p ( i ) =
Revised: e.g. i and set p ( i ) =

Original: ρ while all other classes are distributed evenly.
Revised: ρ while all other classes are distributed evenly.

Original: Figure 5 depicts the the weight estimation performance of RLLS compared to BBSL for a variety of values of ρ and α .
Revised: Figure 5 depicts the the weight estimation performance of RLLS compared to BBSL for a variety of values of ρ and α .

Original: Note that larger shifts correspond to smaller α and larger ρ .
Revised: Note that larger shifts correspond to smaller α and larger ρ .

Original: In general, one observes that our RLLS estimator has smaller MSE and that as the shift increases, the error of both methods increases.
Revised: In general, one observes that our RLLS estimator has smaller MSE and that as the shift increases, the error of both methods increases.

Original: For tweak-one shift we can additionally see that as the shift increases, RLLS outperforms BBSL more and more as both in terms of bias and variance.
Revised: For tweak-one shift we can additionally see that as the shift increases, RLLS outperforms BBSL more and more as both in terms of bias and variance.

Original: [Figure]
Revised: [Figure]

Original: A.6. MNIST E XPERIMENTS UNDER M INORITY -C LASS SOURCE SHIFTS FOR DIFFERENT VALUES OF p
Revised: A.6. MNIST E XPERIMENTS UNDER M INORITY -C LASS SOURCE SHIFTS FOR DIFFERENT VALUES OF p

Original: In order to show weight estimation and classiﬁcation performance under different level of label shifts, we include several additional sets of experiments here in the appendix.
Revised: In order to show weight estimation and classiﬁcation performance under different level of label shifts, we include several additional sets of experiments here in the appendix.

Original: Figure 6 shows the weight estimation error and accuracy comparison under a minority-class shift with p = 0.001.
Revised: Figure 6 shows the weight estimation error and accuracy comparison under a minority-class shift with p = 0.001.

Original: The training and testing sample size is 10000 examples in this case.
Revised: The training and testing sample size is 10000 examples in this case.

Original: We can see that whenever the weight estimation of RLLS is better, the accuracy is also better, except in the four classes case when both methods are bad in weight estimation.
Revised: We can see that whenever the weight estimation of RLLS is better, the accuracy is also better, except in the four classes case when both methods are bad in weight estimation.

Original: Figure 7 demonstrates another case in minority-class shift
Revised: Figure 7 demonstrates another case in minority-class shift

Original: [Figure]
Revised: [Figure]

Original: when p = 0 .
Revised: when p = 0 .

Original: The black-box classiﬁer is the same two-layers neural network trained on a biased source data set with tweak-one ρ = 0 .
Revised: The black-box classiﬁer is the same two-layers neural network trained on a biased source data set with tweak-one ρ = 0 .

Original: We observe that when the number of minority class is small like 1 or 2, the weight estimation is similar between two methods, as well as in the classiﬁcation accuracy.
Revised: We observe that when the number of minority class is small like 1 or 2, the weight estimation is similar between two methods, as well as in the classiﬁcation accuracy.

Original: But when the shift get larger, the weights are worse and the performance in accuracy decreases, getting even worse than the unweighted classiﬁer.
Revised: But when the shift get larger, the weights are worse and the performance in accuracy decreases, getting even worse than the unweighted classiﬁer.

Original: [Figure]
Revised: [Figure]

Original: A.6. MNIST E XPERIMENT UNDER D IRICHLET S HIFT WITH LOW TARGET SAMPLES
Revised: A.6. MNIST E XPERIMENT UNDER D IRICHLET S HIFT WITH LOW TARGET SAMPLES

Original: We show the performance of classiﬁer with different regularization λ under a Dirichlet shift with α = 0 .
Revised: We show the performance of classiﬁer with different regularization λ under a Dirichlet shift with α = 0 .

Original: 5 in Figure 8.
Revised: 5 in Figure 8.

Original: The training has 5000 examples in this case.
Revised: The training has 5000 examples in this case.

Original: We can see that in this low target sample case, λ
Revised: We can see that in this low target sample case, λ

Original: = 1 only take over after several hundreds example, while some λ value betweenand 1 outperforms it at the beginning.
Revised: = 1 only take over after several hundreds example, while some λ value betweenand 1 outperforms it at the beginning.

Original: Similar as in the paper, we use different black-box classiﬁer that is corrupted in different levels to show the relation between the quality of black-box predictor and the necessary target sample size.
Revised: Similar as in the paper, we use different black-box classiﬁer that is corrupted in different levels to show the relation between the quality of black-box predictor and the necessary target sample size.

Original: We use biased source data with tweak-one ρ = 0 , 0 . 2 , 0 .
Revised: We use biased source data with tweak-one ρ = 0 , 0 . 2 , 0 .

Original: 6 to train the black-box classiﬁer.
Revised: 6 to train the black-box classiﬁer.

Original: We see that we need more target samples for the fully weighted version λ
Revised: We see that we need more target samples for the fully weighted version λ

Original: = 1 to take over for a more corrupted black-box classiﬁer.
Revised: = 1 to take over for a more corrupted black-box classiﬁer.

Original: [Figure]
Revised: [Figure]