T RACKING THE RISK OF A DEPLOYED MODEL AND DETECTING HARMFUL DISTRIBUTION SHIFTS
Anonymous authors
Paper under double-blind review
A BSTRACT
When deployed in the real world, machine learning models inevitably encounter changes in the data distribution, and certain—but not all—distribution shifts could result in signiﬁcant performance degradation.
In practice, it may make sense to ignore benign shifts, under which the performance of a deployed model does not degrade substantially, making interventions by a human expert (or model retraining) unnecessary.
While several works have developed tests for distribution shifts, these typically either use non-sequential methods, or detect arbitrary shifts (benign or harmful), or both.
We argue that a sensible method for ﬁring off a warning has to both (a) detect harmful shifts while ignoring benign ones, and (b) allow continuous monitoring of model performance without increasing the false alarm rate.
In this work, we design simple sequential tools for testing if the difference between source (training) and target (test) distributions leads to a signiﬁcant increase in a risk function of interest, like accuracy or calibration.
Recent advances in constructing time-uniform conﬁdence sequences allow efﬁcient aggregation of statistical evidence accumulated during the tracking process.
The designed framework is applicable in settings where (some) true labels are revealed after the prediction is performed, or when batches of labels become available in a delayed fashion.
We demonstrate the efﬁcacy of the proposed framework through an extensive empirical study on a collection of simulated and real datasets.
I NTRODUCTION
Developing a machine learning system usually involves data splitting where one of the labeled folds is used to assess its generalization properties.
Under the assumption that the incoming test instances (target) are sampled independently from the same underlying distribution as the training data (source), estimators of various performance metrics, such as accuracy, reliably reﬂect the truth.
However, a model deployed in the real world inevitably encounters variability in the input distribution, a phenomenon referred to as dataset shift ; see the book by Quionero-Candela et al.
Special cases include covariate shift (Shimodaira, 2000) and label shift (Saerens et al., 2002).
While testing whether a distribution shift is present has been studied both in ofﬂine (Rabanser et al., 2019;
Gretton et al., 2012; Hu & Lei, 2020) and online (Vovk et al., 2005; Vovk, 2020a;b) settings, a natural question is whether an intervention is required once there is evidence that a shift has occurred.
A trustworthy machine learning system has to be supplemented with a set of tools designed to raise alarms whenever critical changes to the environment take place.
Vovk et al.
(2021) propose retraining once an i.i.d.
assumption becomes violated and design corresponding online testing protocols.
However, naively testing for the presence of distribution shift is not fully practical since it does not take into account the malignancy of a shift (Rabanser et al., 2019).
To elaborate, users are typically interested in how a model performs according to certain prespeciﬁed metrics.
Deﬁne a benign shift as one in which model performance is not signiﬁcantly affected despite there being a distribution shift.
In practice, raising unnecessary alarms in benign scenarios might lead to delays and substantial increase in the cost of model deployment.
The recent approach by Vovk et al.
based on conformal test martingales does not always differentiate between benign and harmful shifts.
The exact behavior depends on the choice of conformity score, but in general the methodology raises an alarm whenever a deviation from i.i.d.
is detected, even when the deviation is not necessarily harmful (see Appendix A.2).
[Figure]
(cid:0) x 21 + x 22 + ε ≥ 1 /(cid:1)
Sometimes, structured shifts can be handled in a post-hoc fashion without performing expensive actions, such as model retraining.
One example arises within the context of distribution-free uncertainty quantiﬁcation where the goal is to supplement predictions of a model with a measure of uncertainty valid under minimal assumptions.
Recent works (Tibshirani et al., 2019; Gupta et al., 2020; Podkopaev & Ramdas, 2021) describe ways of correcting for covariate and label shifts that do not require labeled data from the target.
However, both aforementioned shifts impose restrictive assumptions on the possible changes in the underlying probability distribution, assuming that either that
P ( X ) changes but P ( Y | X ) stays unchanged (covariate shift assumption) or P ( Y ) changes P ( X | Y ) stays unchanged (label shift assumption)
Thinking of distribution shifts in terms of covariate shift or label shift has two drawbacks: such (unveriﬁable) assumptions often may be unrealistic, and even if they were plausible, such shifts may be benign and thus could be ignored.
To elaborate on the ﬁrst point, it is evident that while distribution shifts constantly occur in practice, they may generally have a more complex nature.
For example, in medical diagnosis, P ( Y ) and P ( X
| Y =
y ) could describe the prevalence of certain diseases in the population and the corresponding symptoms of disease y .
One might reasonably expect not only the former to change over time (say during ﬂu season or epidemics) but also the latter (due to potential mutations and partially-effective drugs/vaccines), thus violating both the covariate and label shift assumptions.
Regarding the second point, a model capable of separating classes sufﬁciently well on the source distribution can sometimes generalize well to the target.
We illustrate benign covariate and label shifts on Figures 1a and 1b respectively.
In this paper, we do not make any covariate or label shift assumption, and allow for arbitrary shifts.
In our view, the important distinction—from the point of view of raising alarms—is between harmful and benign shifts, not between covariate and label shifts.
A related question is whether labeled data from one distribution can be used for training a model in a way that it generalizes well to another distribution where it is hard to obtain labeled examples.
Importance-weighted risk minimization can yield models with good generalization properties on the target domain, but the corresponding statistical guarantees typically become vacuous if the importance weights are unbounded, which happens when the source distribution’s support fails to cover the target support.
Adversarial training schemes (Ganin et al., 2016; Wu et al., 2019) for deep learning models often yield models with reasonable performance on some types of distribution shifts, but certainly some unanticipated shifts could still degrade performance.
This paper does not deal with how to train a model if a particular type of shift is anticipated; it answers the question of when one should consider retraining (or re-evaluating) a trained and currently deployed model.
We argue for triggering a warning once the non-regularities in the data generating distribution lead to a statistically signiﬁcant increase in the risk function of interest.
We design theoretically tractable tools for nonparametric sequential testing for an unfavorable change in a chosen risk function of any black-box model.
The procedure can be deployed in settings where (some) true target labels can be obtained, immediately after prediction or in a delayed fashion.
During the preparation of our paper, we noticed a very recent preprint (Kamulete, 2021) on broadly the same topic.
While they also advise against testing naively for the presence of a shift, their approach is different from ours as it (a) it is based on measuring the malignancy of a shift based on outlier scores (while we suggest measuring malignancy via a drop in test accuracy or another prespeciﬁed loss), and arguably more importantly (b) their procedure is non-sequential (it is designed to be performed once at the end of the year and cannot be continuously monitored, but ours is designed to ﬂag alarms at any moment that a harmful shift is detected).
Adapting ﬁxed-time testing to the sequential settings requires performing corrections for multiple testing: if the correction is not performed, the procedure is no longer valid, but if naively performed, the procedure becomes too conservative, due to the dependence among the tests being ignored (see Appendix 4).
We reduce the testing problem to performing sequential estimation that allows us to accumulate evidence over time, without throwing away any data or the necessity of performing explicit corrections for multiple testing; these are implicitly handled in an efﬁcient manner by the martingale methods that underpin the sequential estimation procedures (Howard et al., 2021; Waudby-Smith & Ramdas, 2021).
In summary, the main contributions of this work are:
We deviate from the literature on detecting covariate or label shifts
and instead of focus on differentiating harmful and benign shifts.
We pose the latter problem as a nonparametric sequential hypothesis test, and we differentiate between malignant and benign shifts by measuring changes in a user-speciﬁed risk metric.
We utilize recent progress in sequential estimation to develop tests that provably control the false alarm rate despite the multiple testing issues caused by continuously monitoring the deployed model (Section 2.2), and without constraining the form of allowed distribution shifts.
For example, we do not require the target data to itself be i.i.d.
our methods are provably valid even if the target distribution is itself shifting or drifting over time.
We evaluate the framework on both simulated (Section 3.1)
and real data (Section 3.2), illustrating its promising empirical performance.
In addition to traditional losses, we also study several generalizations of the Brier score (Brier, 1950) to multiclass classiﬁcation.
S EQUENTIAL TESTING FOR A SIGNIFICANT RISK INCREASE
Let X and Y denote the covariate and label spaces respectively.
Consider predictors f : X → Y .
Let (cid:96) ( · , · ) be the loss function chosen to be monitored, with R ( f ) :
=
E [ (cid:96) ( f ( X ) , Y )] denoting the corresponding expected loss, called the risk of f .
We assume from here onwards that (cid:96) is bounded, which is the only restriction made.
This is needed to quantify how far the empirical target risk is from the true target risk without making assumptions on
P ( X, Y ) .
This is not a restriction of the current paper only, but appears broadly in statistical learning theory (or the study of concentration inequalities): if one picks an unbounded loss function—say the logarithmic loss—which can take on inﬁnite values, then it is impossible to say how far the true and empirical risks are without further assumptions, for example that the distribution of P ( Y | X ) is light tailed.
We prefer not to make such assumptions in this paper, to keep it as assumption-light as possible and thus generally applicable in various domains.
The restriction to bounded losses is not heavy since examples abound in the machine learning literature; see some examples in Appendix B.
Remark 1 .
For classiﬁcation, sometimes one does not predict a single label, but a distribution over labels.
In that case the range of f would be ∆ |Y| .
This poses no issue, and it is common to use bounded loss functions and risks such as the Brier score, as exempliﬁed in Section 3.
On a different note, for regression, the loss (like squared error) is bounded only if the observations are.
This is reasonable in some contexts (predicting rain or snow) but possibly not in others (ﬁnancial losses).
2. C ASTING THE DETECTION OF RISK INCREASE AS A SEQUENTIAL HYPOTHESIS TEST
We aim to trigger a warning whenever the risk on the target domain exceeds the risk on the source by a non-negligible amount speciﬁed in advance.
For example, alerting could happen once it is possible to conclude with certain conﬁdence that the accuracy has decreased by 10%.
Shifts that lead to a decrease or an insigniﬁcant increase in the risk are then treated as benign.
Formally, we wish to design a sequential test for the following pair of hypotheses:
[Equation]
where ε tol ≥ 0 is an acceptable tolerance level, and R S ( f ) and R T ( f ) stand for the risk of f on the source and target domains respectively.
Assume that one observes a sequence of data points
Z 1 , Z 2 , . . . .
At each time point t , a sequential test takes the ﬁrst t elements of this sequence and output either a 0 (continue) or 1 (reject the null and stop).
The resulting sequence of 0s and 1s satisﬁes the property that if the null H 0 is true, the probability that it ever outputs a 1 and stops (false alarm) is at most δ .
In our context, this means that if there is no harm distribution shift, then with high probability, the test will never stop and output a 1, and thus runs forever.
Formally, a level- δ sequential test Φ deﬁned as a mapping
[Equation]
must satisfy: P
[Equation] [Equation]
Φ
[Equation]
= 1) ≤ δ .
Note that the sequential nature of a test is critical here as we aim to develop a framework capable of continuously updating inference as data from the target is collected, making it suitable for many practical scenarios.
We distinguish between two important settings when we assume that the target data either satisﬁes (a) an i.i.d.
assumption (under the same or different distribution as the source) or (b) only an independence assumption.
While the i.i.d.
assumption may be arguably reasonable on the source, it is usually less realistic on the target, since in practice, one may expect the distribution to drift slowly in a non-i.i.d.
fashion instead of shifting sharply but staying i.i.d.
˙Under setting (b), the quantity of interest on the target is the running risk :
[Equation]
where the expected value is taken with respect to the joint distribution of ( X (cid:48)
i , Y (cid:48) i ) , possibly different for each test point i .
The goal transforms into designing a test for the following pair of hypotheses:
[Equation]
[Equation]
When considering other notions of risk beyond the misclassiﬁcation error, one could also be interested in relative changes in the risk, and thus a sequential test for the following pair of hypotheses:
[Equation]
The proposed framework handles all of the aforementioned settings as we discuss next.
The most classical approach for sequential testing is the sequential probability ratio test (SPRT) due to Wald (1945).
However, it can only be applied, even for a point null and a point alternative, when the relevant underlying distributions are known.
While extensions of the SPRT exist to the composite null and alternative (our setting above), these also require knowledge of the distributions of the test statistics (e.g., empirical risk) under the null and alternative, and being able to maximize the likelihood.
Clearly, we make no distributional assumptions and so we require a more nonparametric approach.
We perform sequential testing via the dual problem of nonparametric sequential estimation, a problem for which there has been much recent progress to draw from.
2. S EQUENTIAL TESTING VIA SEQUENTIAL ESTIMATION
When addressing a particular prediction problem, the true risk on neither the source nor the target domains is known.
Performance of a model on the source domain is usually assessed through a labeled held-out source sample of a ﬁxed size n S : { ( X i , Y i ) } n S i =1 .
We can write:
[Equation]
where
[Equation]
=
[Equation]
n S i =1 (cid:96) ( f ( X i ) , Y i ))
/n S .
For any ﬁxed tolerance level δ S ∈ (0 , 1) , classic concentration results can be used to obtain an upper conﬁdence bound ε appr on the difference
R S ( f ) − (cid:98)
R S ( f ) , and thus to conclude that with probability at least 1 − δ S :
[Equation]
, where
[Equation]
For example, by Hoeffding’s inequality, n S =
O
(1 /ε 2 appr ) points for the above guarantee, but that bound can be quite loose when the individual losses (cid:96) ( f ( X i ) , Y i ) have low variance.
In such settings, recent variance-adaptive conﬁdence bounds (Waudby-Smith & Ramdas, 2021; Howard et al., 2021) are tighter.
This then translates to an increase in the power of the framework, effectively detecting harmful shifts much earlier, while still controlling the false alarm rate at a prespeciﬁed level.
In contrast, the estimator of the target risk has to be updated as losses on test instances are observed.
While the classic concentration results require specifying in advance the size of a sample used for estimation, time-uniform conﬁdence sequences retain validity under adaptive data collection settings.
For any chosen δ T ∈ (0 , 1) , those yield a time-uniform lower conﬁdence bound on R T ( f ) :
[Equation]
where
[Equation]
L ( t ) T ( f ) is the bound constructed after processing t test points.
We typically set δ S = δ T
= δ/ 2 , where δ refers to the desired type I error.
Under the independence assumption (equation 2), the form of the drift in the distribution of the data is allowed to change with time.
From the technical perspective, the difference with the i.i.d.
setting is given by the applicability of particular concentration results.
While the betting-based approach of (Waudby-Smith & Ramdas, 2021) necessitates assuming that random variables share a common mean, proceeding with the conjugate-mixture empirical-Bernstein (Howard et al., 2021) allows us to lift the common-mean assumption and handle a time-varying mean.
We summarize the testing protocol in Algorithm 1 and refer the reader to
Appendix E for a review of the concentration results used in this work.
One can easily adapt the framework to proceed with a ﬁxed, absolute threshold on the risk, rather than a relative threshold
R S ( f ) + ε tol , e.g., we can raise a warning once accuracy drops below 80%, rather than 5% below the training accuracy.
Input: Predictor f , loss (cid:96) , tolerance level ε tol , sample from the source { ( X i , Y i ) } n S i = 1: procedure 2:
Compute the upper conﬁdence bound on the source risk (cid:98)
U S ( f ) ; 3: for t = 1 , 2 , . . .
do 4:
Compute the lower conﬁdence bound on the target risk (cid:98)
L ( t ) T ( f ) ; 5: if (cid:98)
L ( t ) T ( f ) > (cid:98)
U S ( f ) + ε tol then 6:
Reject H 0 (equation 1) and ﬁre off a warning.
Testing for a relative increase in the risk is performed by replacing the line 5 in the Algorithm 1 by the condition
[Equation]
U S
[Equation]
f ) .
For both cases, the proposed test controls type I error as formally stated next.
The proof is presented in Appendix D.
Proposition 1.
Fix any δ ∈ (0 , 1) .
Let δ S , δ T ∈ (0 , 1) be chosen in a way such that δ S + δ T
= δ .
Let (cid:98)
L ( t ) T ( f ) deﬁne a time-uniform lower conﬁdence bound on R ( t ) T ( f ) at level δ T after processing t data points ( t ≥ 1 ), and let
[Equation]
U S ( f ) deﬁne an upper conﬁdence bound on R S ( f ) at level δ S .
Then:
[Equation]
that is, the procedure described in Algorithm 1 controls the type I error for testing the hypotheses
H 0 (equation 1 and equation 2) and H (cid:48)
(equation 3).
Remark 2 .
Both the testing protocol (Algorithm 1) and the corresponding guarantee (Proposition 1) are stated in a form which requires the lower bound on the target risk to be recomputed after processing each test point.
More generally, test data could be processed in minibatches of size m ≥ 1 .
Similarly, the initial sample size could be replaced with some other initial size t 0 ≥ 1 .
Remark 3 .
The above type I error guarantee (equation 5) holds under continuous monitoring.
This goes beyond standard guarantees for a ﬁxed-time test, for which type I error control only holds when the sample size is ﬁxed in advance, and not under continuous monitoring.
We can deﬁne a stopping time of a sequential test in Algorithm 1 as:
[Equation]
and restate the guarantee in equation 5 as: P H 0 ( N ( δ ) < ∞ ) ≤ δ , that is the probability of ever raising a false alarm is at most δ .
From sequential testing to changepoint detection.
Results due to Lorden (1971) allow to transform a valid sequential test into a changepoint detection procedure with certain guarantees.
The key characteristics of changepoint detection procedures are average run length (ARL), or average time to a false alarm, and average detection delay (ADD).
One way to convert a sequential test into a detection procedure is by running a separate test starting at each time point t = 1 , 2 , . . . , and claiming a change whenever the ﬁrst one of the tests rejects the null.
Subsequently, these tests yield a sequence of stopping variables
N 1 ( δ ) , N 2 ( δ ) , . . . .
The corresponding stopping time is deﬁned as:
[Equation]
Lorden (1971) established a lower bound on the (worst-case) ARL of such changepoint detection procedure of the form: E H
[ N (cid:63) ( δ )] ≥ 1 /δ .
The (worst-case) average detection delay is deﬁned as:
[Equation]
where E m denotes expectation under P m , the distribution of a sequence ( X (cid:48) 1 , Y (cid:48) 1 ) , ( X (cid:48) 1 , Y (cid:48) 1 ) , . . .
under which ( X (cid:48) m , Y (cid:48) m ) is the ﬁrst term from a shifted distribution.
E XPERIMENTS
In Section 3.1, we study the performance of the testing procedure on a collection of simulated datasets.
First, we consider the setting where the i.i.d.
assumption on the target is satisﬁed, and then relax it to the independence assumption.
In Section 3.2, we evaluate the framework on real data.
For simulations, we consider classiﬁcation problems.
We study misclassiﬁcation loss, several versions of the Brier score and miscoverage loss for set-valued predictors.
Due to space limitations, we refer the reader to Appendix B for a detailed review of the loss functions considered in the this work.
3. S IMULATED DATA
Tracking the risk under the i.i.d.
assumption.
Here we induce label shift on the target domain and emulate a setting where it noticeably harms the accuracy of a predictor by modifying the setup from Section 1 through updating the class centers to: µ
=
( − 1 , 0) (cid:62) , µ 1 = (1 , 0) (cid:62) , making the classes largely overlap.
The (oracle) Bayes-optimal predictor on the source domain is:
[Equation]
where ϕ ( x ; µ i , I 2 ) denotes the probability density function of a Gaussian random vector with mean µ i , i ∈ { 0 , 1 } and an identity covariate matrix.
Let (cid:96) be the 0-1 loss, and thus the misclassiﬁcation risk R T ( f ∗ ) on the target is:
[Equation]
For three values of π S 1 , the source marginal probability of class 1, we illustrate how label shift affects the misclassiﬁcation risk of the Bayes-optimal predictor on Figure 2a, noting that it is linear in π T 1 .
Importantly, whether label shift hurts or helps depends on the value of π S 1 .
Next, ﬁx π S 1 = 0 .
and use the corresponding Bayes-optimal rule.
On Figure 2b, we compare upper conﬁdence bounds on the source risk due to different concentration results, against the size of the source holdout set.
The variance-adaptive upper conﬁdence bounds, predictably-mixed empiricalBernstein (PM-EB) and betting-based (see Appendix E), are much tighter than the non-adaptive
Hoeffding’s bound.
Going forward, we use a source heldout set of 1000 points to compute the upper conﬁdence bound on the source risk, where (cid:15) appr from equation 4 is about 0.025.
Next, we ﬁx ε tol = 0 .
05 , that is, a 5% drop in accuracy is treated as signiﬁcant.
For 20 values of π T 1 , evenly spaced in the interval [0 . 1 , 0 . 9] , we sample data from the target distribution in batches of
[Figure]
r i s
[Figure]
50 points, with maximum number of points taken to be 2000.
On Figure 3a, we track the proportion of null rejections after repeating the process 250 times.
On Figure 3b we illustrate the average size of a sample from the target that was needed to reject the null.
Stronger label shift is more harmful to accuracy and the results conﬁrm that tighter bounds yield better detection procedures and generally require less samples to do so, with the most powerful test utilizing the betting-based bounds based (Waudby-Smith & Ramdas, 2021).
We also perform a similar analysis when the
Brier score (Brier, 1950) is used as a target metric (Appendix F) and study the performance of the framework under the covariate shift setting (Appendix H).
[Figure]
P r o p o r t i o n o f r e j ec t i o n s
[Figure]
Tracking beyond the i.i.d.
setting (distribution drift ).
Here we consider testing whether the running risk on the target increases signiﬁcantly (equation 2).
First, we keep the data generation pipeline as before, that is, the target data are still sampled in an i.i.d.
fashion, and ﬁx π T 1 = 0 .
We compare lower conﬁdence bounds on the target risk studied before against the conjugate-mixture empiricalBernstein (CM-EB) bound due to Howard et al.
We note that this lower bound on the running mean of the random variables does not have a closed-form and has to be computed numerically.
On
Figure 4a, we illustrate that the lower conﬁdence bounds based on betting are generally tighter.
Similar results hold for the Brier score as a target metric (Appendix F.1).
Next, we lift the i.i.d.
assumption on the target which makes CM-EB the only valid lower bound on the running risk on the target.
We modify the data generation pipeline the following way: starting with π T 1 = 0 .
25 , we increase π T 1 by 0.1 after sampling each 200 instances, until it reaches the value 0.85.
The results of running the framework for this setting are presented on Figure 4b.
[Figure]
Betting-based LCB is only tighter than conjugate-mixture empirical-Bernstein (CM-EB) for a small number of samples.
(b) Under distribution drift, only CM-EB performs estimation of the running risk.
The resulting test consistently detects a harmful increase in the running risk.
3. R EAL DATA
Next, we evaluate the testing framework on real data.
In deep learning, out-of-distribution robustness is often assessed based on a model performance gap between the original data (used for training) and data to which various perturbations have been applied.
We focus on two image classiﬁcation datasets with induced corruptions: MNIST-C (Mu & Gilmer, 2019) and CIFAR-10-C (Krizhevsky, 2009; Hendrycks & Dietterich, 2019).
We illustrate an example of a clean MNIST image on Figure 5a along with its corrupted versions after applying motion blur , blur along a random direction (Figure 5b), translate , afﬁne transformation along a random direction (Figure 5c), and zigzag , randomly oriented zigzag over an image (Figure 5d).
For CIFAR-10-C, we consider corrupting original images by applying the fog effect with 3 levels of severity as illustrated on the bottom row of Figure 5.
While corruptions are clearly visible to a human eye but are not expected to hurt classiﬁcation performance signiﬁcantly, it is different when such images passed to a predictive model.
In both cases, we consider the settings when either clean test samples or ones with induced corruptions are passed as input to networks trained on clean datasets.
For the simulations that follow, we refer to the betting-based upper conﬁdence bounds on the risk for the source, and to the lower conﬁdence bounds on the running risk that are due to conjugate-mixture empirical-Bernstein for the target.
[Figure]
Tracking the risk of a point predictor on MNIST-C dataset.
We train a shallow CNN on clean
MNIST data, and run the framework testing whether the misclassiﬁcation risk increases by 10%, feeding the network with data in batches of 50 points either from the original or shifted distributions.
Details regarding the network architecture and the training process are given in Appendix G.
On
Figure 6a, we illustrate the results after running the procedure 50 times for each of the settings.
The horizontal dashed line deﬁnes the rejection threshold that has been computed using the source data: once the lower bound on the target risk (solid lines) exceeds this value, the null hypothesis is rejected.
When passing clean MNIST data as input, we do not observe degrading performance.
Further, applying different corruptions leads to both benign and harmful shifts.
While to a human eye the translate effect is arguably the least harmful one, it is the most harmful to the performance of a network.
Such observation is also consistent with ﬁndings of Mu & Gilmer (2019) who observe that if trained on a clean MNIST data without any data augmentation, CNNs tend to fail under this corruption.
We validate this observation by retraining the network several times in Appendix G.
Tracking the risk of a set-valued predictor on CIFAR-10-C dataset.
For high-consequence settings, building accurate models only can be insufﬁcient as it is crucial to quantify uncertainty in predictions.
There is a number of recent works on post-processing point predictors that targets covering the correct label of a test point with high probability (Vovk et al., 2005) or controlling other notions of risk (Bates et al., 2021).
We follow Bates et al.
(2021) who design a procedure that uses a held-out set for tuning the parameters of a wrapper built on top of the original model which, under the i.i.d.
assumption, is guaranteed to have low risk with high probability (see Appendix G for details).
For the simulation that follows we build a wrapper around a trained ResNet-32 model that controls the miscoverage risk (equation 14) at level 0 .
1 with probability at least 0 .
For each run,
CIFAR-10 test set is split at random into three parts used for: (a) transforming a network into a setvalued predictor (1000 points), (b) estimating an upper conﬁdence bound on the miscoverage risk on the source (1000 points), and (c) evaluation purposes on either clean or corrupted images.
We take ε tol = 0 .
05 , that is raising an alarm happens whenever coverage of the prediction sets on the target drops by 5%.
Figure 6b illustrates that only the most severe version of the fog effect consistently leads to a signiﬁcant increase in miscoverage.
In Appendix G, we also consider transforming the network into a set-valued predictor by setting a lower prescribed error level (0.05) for the set-valued predictor.
In particular, it results in larger prediction sets, which partially mitigates the effect of the fog on the drop in coverage.
[Figure]
D ISCUSSION
An important component of building reliable machine learning systems is enabling them to alarm a user when potentially unsafe behavior is observed, instead of allowing them to fail silently.
In this work, we considered one particular failure scenario which deployed models encounter in the real world — presence of distribution shifts.
We developed a set of tools for testing in a data-adaptive way whether the performance of a model on the test data becomes signiﬁcantly worse than the performance on the training data.
The proposed framework is based on performing sequential estimation, and thus requires observing true labels for test data (possibly, in a delayed fashion).
Across various types of distribution shifts considered in this work, it demonstrated promising empirical performance when differentiating between harmful and benign ones.
Ethics Statement.
We do not foresee any negative impacts of this work in and of itself; there may be societal impacts of the underlying classiﬁcation or regression task but these are extraneous to the aims and contributions of this paper.
Reproducibility statement.
In order to ensure reproducibility of the results in this paper, we include the following to the supplementary materials: (a) relevant source code for all simulations that have been performed, (b) clear explanations of all data generation pipelines and preprocessing steps that have been performed, (c) complete proofs for all established theoretical results.
A I SSUES WITH EXISTING TESTS FOR DISTRIBUTION SHIFTS / DRIFTS A. N ON - SEQUENTIAL TESTS HAVE HIGHLY INFLATED FALSE ALARM RATES WHEN DEPLOYED IN SEQUENTIAL SETTINGS WITH CONTINUOUS MONITORING
In this work, we propose a framework that utilizes conﬁdence sequences (CSs), and thus allows for continuous monitoring of model performance.
On the other hand, traditional (ﬁxed-time) testing procedures are not valid under sequential settings, unless corrections for multiple testing are performed.
First, we illustrate that deploying ﬁxed-time detection procedures under sequential settings necessarily leads to raising false alarms.
Then, we illustrate that naive corrections for multiple testing — without taking advantage of the dependence between the tests — lead to losses of power of the resulting procedure.
Deploying ﬁxed-time tests without corrections for multiple testing.
Under the i.i.d.
setting, our framework reduces to testing whether the means corresponding to two unknown distributions are signiﬁcantly different.
Here we consider a simpliﬁed setting: assume that one observes a sequence
Z 1 , Z 2 , . . .
of bounded i.i.d.
random variables, and the goal is to construct a lower conﬁdence bound for the corresponding mean µ .
In this case, a natural alternative to conﬁdence sequences is a lower bound obtained by invoking the Central Limit Theorem:
[Equation]
where z δ is (1 − δ ) -quantile of the standard Gaussian random variable and Z t , (cid:98) σ t denote the sample average and sample standard deviation respectively computed using ﬁrst t instances Z 1 , . .
, Z t .
For the study below, we use the critical (signiﬁcance) level δ = 0 .
We sample the data as: Z t ∼
Ber(0 . 6) , t = 1 , 2 , . .
. , and consider 100 possible sample sizes, evenly spaced between 20 andon a logarithmic scale.
Next, we compare the CLT lower bound with the betting-based one (which underlies the framework proposed in this work) under the following settings:
Fixed-time monitoring.
For a given sample size t , we sample the sequence Z 1 , . . .
, Z t , compute the lower bounds and check whether the true mean is covered only once.
For each value of the sample size, we resample data 100 times and record the miscoverage rate, that is, the fraction of times the true mean is miscovered.
Continuous monitoring.
Here, the lower bound is recomputed once new data become available.
We resample the whole data sequence Z 1 , . . .
, Z 1000 1000 times, and for each value of the sample size, we track the cumulative miscoverage rate, that is, the fraction of times the true mean has been miscovered at some time up to t .
Under ﬁxed-time monitoring (Figure 7a), the false alarm rate is controlled at prespeciﬁed level δ by both procedures.
However, under continuous monitoring (Figure 7b), deploying the CLT lower bound leads to raising false alarms.
At the same time, the betting-based lower bound controls the false alarm rate under both types of monitoring.
Deploying ﬁxed-time tests with corrections for multiple testing.
Next, we illustrate that adapting ﬁxed-time tests to sequential settings via performing corrections for multiple testing comes at the price of signiﬁcant power losses.
Performing the Bonferroni correction requires splitting the available error budget δ among the times when testing is performed.
In particular, we consider:
Power correction
[Equation]
Note that the second option is preferable as the terms in the sum decrease at a slower rate, thus allowing for a narrower sequence of intervals.
Proceeding under the setup considered in the beginning of this section (data points are sampled from Ber(0 . 6) ), we consider two scenarios:
We recompute the CLT lower bound each time a batch of 25 samples is received and perform the Bonferroni correction (utilizing both ways of splitting the error budget described
[Figure]
in equation 7).
We present the lower bounds on Figure 8a (the results have been aggregated over 100 data draws).
Observe that: • While the sequence of intervals shrinks in size with growing number of samples under the polynomial correction, this is not the case under the power correction.
• Not only the betting conﬁdence sequence is uniformly tighter than the CLT-based over all considered sample sizes, it also allows for monitoring at arbitrary stopping times.
Note that the CLT lower bound allows for monitoring only at certain times (marked with stars on Figure 8a
For a fairer comparison with the betting-based bound, we also consider recomputing the CLT bound each time a new sample is received.
Since utilizing the power correction quickly leads to numerical overﬂows, we utilize only the polynomial correction.
We present the lower bounds on Figure 8b (the results have been aggregated over 100 data draws).
While now the CLT lower bound can be monitored at arbitrary stopping times, it is substantially lower (thus more conservative) than the betting-based.
[Figure]
A. C ONFORMAL TEST MARTINGALES DO NOT IN GENERAL DIFFERENTIATE BETWEEN HARMFUL AND BENIGN DISTRIBUTION SHIFTS
Testing the i.i.d.
assumption online can be performed using conformal test martingales (Vovk et al., 2021; Vovk, 2020b;a).
Below, we review building blocks underlying a conformal test martingale.
First, one has to pick a conformity score .
Assigning a lower score to a sample indicates abnormal behavior.
Vovk et al.
(2021) consider the regression setting and, like us, use scores that depends on true labels: for a point ( x i , y i ) , let (cid:98)
y i denote the output of a predictor on input x i .
The authors propose a score of the form:
[Equation]
Note that lower scores deﬁned in equation 8 clearly reﬂect degrading performance of a predictor (possibly due to the presence of distribution drift).
Under the classiﬁcation setting, we propose to consider the following conformity score:
[Equation]
which is a rescaled estimated probability mass of all the labels that are more likely than the true one (here, we assume that predictor f outputs an element of ∆ |Y| ).
Rescaling in equation 9 is used to ensure that this score represent the conformity score, that is, the higher the value is, the better a given data point conforms.
Note that if for a given data point, the true label happens to be top-ranked by a predictor f , then such point receives the largest conformity score equal to one.
This score is inspired by recent works in conformal classiﬁcation (Romano et al., 2020; Podkopaev & Ramdas, 2021).
After processing n data points, a transducer transforms a collection of conformity scores into a conformal p-value:
[Equation]
where u ∼ Unif([0 , 1])
Conformal p-values are i.i.d.
uniform on [0 , 1] when the data points are i.i.d.
(or more generally, exchangeable; see (Vovk et al., 2021)).
Note that the design of conformal p-value P n ensures it takes small value when the conformity score α n is small, that is, when abnormal behavior is being observed in a sequence.
A betting martingale is used to gamble again the null hypothesis that a sequence of random variables is distributed uniformly and independently on [0 , 1] .
Formally, a betting martingale is a measurable function F : [0 , 1] (cid:63) → [0 , ∞ ] such that F ( (cid:3) )
=
1 ( (cid:3) deﬁnes an empty sequence and Z (cid:63) stands for the set of all ﬁnite sequences of elements of Z ) and for each sequence ( u 1 , . .
, u n − 1 ) ∈ [0 , 1] n − 1 and any n ≥ 1 :
[Equation]
The simplest example is given by the product of simple bets:
[Equation]
but more sophisticated options are available (Vovk et al., 2005).
For the simulations that follow, we use simple mixture martingale which is obtained by integrating equation 10 over ε ∈ [0 , 1] .
Conformal test martingale S n is obtained by plugging in the sequence of conformal pvalues P 1 , . .
, P n into the betting martingale.
The test starts with S
= 1 and it rejects at the ﬁrst time n when S n exceeds 1 /α .
They type I error control for this test is justiﬁed by Ville’s inequality which states that for any nonnegative martingale (which S n is, under the i.i.d.
null), the entire process S n stays below 1 /α with probability at least 1 − α .
Mathematically:
[Equation]
To study conformal test martingales, we consider the label shift setting described in Section 3.1.
Recall that for this setting we know exactly when a shift in label proportions becomes harmful to misclassiﬁcation risk of the Bayes-optimal rule on the source distribution (see Figures 2a and 3a).
For the simulations that follow, we assume that the marginal probability of class 1 on the source is π S 1 = 0 .
25 , and use the corresponding optimal rule.
We analyze conformal test martingales under several distribution drift scenarios differing in their severity and rate, and start with the settings where a sharp shift is present.
Harmful distribution shift with cold start .
Here, the data are sampled i.i.d.
from a shifted distribution corresponding to π T 1 = 0 .
We illustrate 50 runs of the procedure on Figure 9a.
Recall that when the data are sampled i.i.d.
the conformal p-values are i.i.d.
uniform on [0 , 1] .
Under the (exchangeability) null, conformal test martingales are not growing, and thus are not able to detect that a present shift, even though it corresponds to a harmful setting.
Harmful distribution shift with warm start .
For a fairer comparison, we also consider a warm start setting when the ﬁrst 100 points are sampled i.i.d.
from the source distribution ( π T 1 = 0 . 25 ), followed by the data sampled i.i.d.
from a shifted distribution ( π T 1 = 0 . 75 ).
We illustrate 50 runs of the procedure on Figure 9b.
In this case, conformal test martingales demonstrate better detection properties.
However, a large fraction of conformal test martingales still is incapable of detecting a shift.
[Figure]
The simulations above illustrate that conformal test martingales are inferior to the framework proposed in this work whenever a sharp distribution shift happens
in the early stage of a model deployment, even when such shift is harmful.
Next, we consider several settings where instead of changing sharply, the distribution drifts gradually.
Slow and benign distribution drift .
Starting with the marginal probability of class 1, π T 1 = 0 .
1 , we keep increasing π T 1 by 0.05 each time a batch of 75 data points is sampled until it reaches the value 0.45.
Recall from Section 3.1 that those values of π T 1 correspond to a benign setting where the risk of the predictor on the target domain does not exceed substantially the source risk.
We illustrate 50 runs of the procedure on Figure 10a. 4.
Slow and harmful distribution drift .
Starting with the marginal probability of classπ T 1 = 0 .
5 , we keep increasing π T 1 by 0.05 each time a batch of 75 data points is sampled until it reaches the value 0.85.
Recall from Section 3.1 that those values of π T 1 correspond to a harmful setting where the risk of the predictor on the target domain is substantially larger than the source risk.
We illustrate 50 runs of the procedure on Figure 10b.
Sharp and harmful distribution drift .
Starting with the marginal probability of class 1, π T 1 = 0 .
1 , we keep increasing π T 1 by 0.2 each time a batch of 150 data points is sampled until it reaches the value 0 .
We illustrate 50 runs of the procedure on Figure 10c.
The settings where the distribution drifts gradually illustrate several shortcomings of conformal test martingales:
• Conformal test martingales consistently detect only sharp distribution drifts.
Recall from Section 3.1 that increasing π T 1 from 0.1 to 0.9 results in more than 20% accuracy drop.
When a drift is slow (Figures 10a and 10b), conformal test martingales demonstrate much less power.
• Inspired by the ideas of Vovk et al.
(2021) who assumed, like us, that (some) true data labels are observed, we designed a conformity score that reﬂects decrease in performance.
[Figure]
On Figures 10a and 10b, conformal test martingales illustrate similar behavior but the corresponding settings are drastically different.
Only one corresponds to a benign drift when the risk does not become signiﬁcantly worse than the source risk.
Thus, even though it is possible to make conformal test martingale reﬂect degrading performance, it is hard to incorporate evaluation of the malignancy of a change in an interpretable way , like a decrease of an important metric.
• Another problem of using conformal test martingales to be aware of is that after some time the corresponding values of the test martingale (larger implies more evidence for a shift) could start to decrease as the shifted distribution becomes the ‘new normal’ (Vovk et al., 2021).
This is because they measure deviations from iid data, not degradations in performance from some benchmark (like source accuracy).
B L OSS FUNCTIONS
For our simulations, we consider the following bounded losses.
Below, let
[Equation]
x ) denote the label prediction of a model f on a given input x ∈ X .
Multiclass losses.
The most common example is arguably the misclassiﬁcation loss and its generalization that allows for a label-dependent cost of a mistake:
[Equation]
where { (cid:96) k } k ∈Y is a collection of per-class costs and L = max k ∈Y (cid:96) k .
The loss (cid:96) w-mis is more relevant to high-stakes decision settings and imbalanced classiﬁcation.
However, high accuracy alone can often be insufﬁcient.
The Brier score (squared error), introduced initially for the binary setting (Brier, 1950), is commonly employed to encourage calibration of probabilistic classiﬁers.
For multiclass problems, one could consider the mean-squared error of the whole output vector:
[Equation]
(11) where h : Y → { 0 , 1 } |Y| is a one-hot label encoder:
h y (cid:48)
( y )
=
1 { y (cid:48)
=
y } for y, y (cid:48) ∈ Y .
Top-label calibration (Gupta & Ramdas, 2021) restricts attention to the entry corresponding to the top-ranked label.
A closely related loss function, which we call the top-label Brier score , is the following:
[Equation]
Alternatively, instead of the top-ranked label, one could focus only on the entry corresponding to the true class.
It gives rise to another loss function which we call the true-class Brier score:
[Equation]
The loss functions (cid:96) brier , (cid:96) brier-top , (cid:96) brier-true trivially reduce to the same loss function in the binary setting.
In Appendix C, we present a more detailed study of the Brier score in the multiclass setting with several illustrative examples.
Set-valued predictors.
The proposed framework can be used for set-valued predictors that output a subset of Y as a prediction.
Such predictors naturally arise in multilabel classiﬁcation, where more than a single label can be the correct one, or as a result of post-processing point predictors.
Postprocessing could target covering the correct label of a test point with high probability (Vovk et al., 2005) or controlling other notions of risk (Bates et al., 2021) like the miscoverage loss:
[Equation]
14) where S ( x ) denotes the output of a set-valued predictor on any given input x ∈ X .
When considering multilabel classiﬁcation, relevant loss functions include the symmetric difference between the output and the set of true labels, false negative rate and false discovery rate.
C B RIER SCORE IN THE MULTICLASS SETTING
This section contains derivations of decompositions stated in Section 2 and comparisons between introduced versions of the Brier score.
Brier score decompositions.
The expected Brier score for the case when the whole output vector is considered satisﬁes the following decomposition:
[Equation]
sharpness
[Equation]
Above, (a) follows by conditioning on f ( X ) for the cross-term and recalling that E [ h ( Y ) | f
( X )]
= c ( X ) , (b) also follows by conditioning on f ( X ) and noticing that E [ h ( Y )]
=
E [ c ( X )] .
Now, recall that a predictor is (canonically) calibrated if f ( X ) a.s. =
c ( X ) , in which case the calibration error term is simply zero.
Next, we consider the top-label Brier score (cid:96) brier-top .
Following essentially the same argument as before, we get that:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Note that in contrast to the classic Brier score decomposition, the last term in this decomposition depends only on the top-class prediction of the underlying predictor, and thus on its accuracy.
Comparison of the scores in multiclass setting.
Recall that the difference between three versions of the Brier score arises when one moves beyond the binary classiﬁcation setting.
We illustrate the difference by considering a 4-class classiﬁcation problem where the data represent a balanced (that is all classes are equally likely) mixture of 4 Gaussians with identity covariance matrix and mean vectors being the vertices of a 2-dimensional unit cube.
One such sample is presented on Figure 11a.
Next, we analyze locally the Brier scores when the Bayes-optimal rule is used as an underlying predictor, that is we split the area into small rectangles and estimate the mean score within each rectangle by a sample average.
The results are presented on Figures 11b, 11c and 11d.
Note that the difference between the assigned scores is mostly observed for the points that lie at the intersection of 4 classes where the support of the corresponding output vectors is large.
[Figure] [Figure]
(c)
[Figure] [Figure]
Brier scores under label shift.
Here we consider the case when label shift on the target domain is present.
First, introduce the label likelihood ratios, also known as the importance weights, w y :=
π Ty /π
Sy , y ∈ Y .
For measuring the strength of the shift, we introduce the condition number : κ = sup y w y / inf y : w y (cid:54) =
w y .
Note that when the shift is not present, the condition number κ = 1 .
To evaluate the sensitivity of the losses to the presence of label shift, we proceed as follows: ﬁrst, the class proportions for both source and target domains are sampled from the Dirichlet distribution (to avoid extreme class proportion, we perform truncation at levels 0.15 and 0.85 and subsequent renormalization).
Then we use the Bayes-optimal rule for the source domain to perform predictions on the target and compute the corresponding losses.
On Figure 12, we illustrate relative increase in the average Brier scores plotted against the corresponding condition number when all data points are considered (Figure 12a) and when attention is restricted to the area where classes highly intersect (Figure 12b).
In general, all three versions of the Brier score suffer similarly on average, but in the localized area where classes highly intersect, the top-label Brier score does not increase signiﬁcantly under label shift.
[Figure]
Condition number − 0 .0 .0 .0 .
R e l a t i v e i n c r e a s e : b R T ( f ) b R S
( f ) −
ClassicBrierscoreTop-labelBrierscoreTrue-classBrierscore (
[Figure]
[Figure]
Condition number − 0 .0 .0 .
R e l a t i v e i n c r e a s e : b R T ( f ) b R S
( f )
[Figure]
ClassicBrierscoreTop
[Figure]
D P ROOFS
Proof of Proposition 1.
For brevity, we omit writing f for the source/target risks and the corresponding bound upper/lower conﬁdence bounds.
Starting with the absolute change and the null
H 0 : R T − R S ≤ ε tol , we have:
[Equation]
Note that ∃ t ≥ 1 : (
[Equation]
t )
[Equation]
− R T
[Equation]
− ( (cid:98)
U S − R S )
> 0 implies that either ∃ t ≥ 1 : (cid:98)
L ( t ) T − R T > 0 or (cid:98)
U S − R S < 0 .
Thus, invoking union bound yields:
[Equation]
by construction and validity guarantees for (cid:98)
L ( t ) T and (cid:98)
U S .
Similarly, considering the relative change, i.e., the null: H (cid:48) 0 : R T ≤ (1 + ε (cid:48) tol ) R S , we have:
[Equation]
(cid:98)
L ( t ) T − R T (cid:17) − (1 + ε (cid:48) tol ) (cid:16) (cid:98)
U S − R S (cid:17) > (1 + ε (cid:48) tol ) R S − R T (cid:17) ≤ P H (cid:48)(cid:16) ∃ t ≥ 1 : (cid:16) (cid:98)
L ( t ) T − R T (cid:17) − (1 + ε (cid:48) tol ) (cid:16) (cid:98)
U S − R S (cid:17) >(cid:17) .
Similarly, note that ∃ t ≥ 1 : ( (cid:98)
L ( t ) T − R T ) − (1 + ε (cid:48) tol )( (cid:98)
U S − R S )
> 0 implies that either ∃ t ≥ 1 : (cid:98)
L ( t ) T − R T > 0 or (cid:98)
U S − R S < 0 .
Thus, invoking union bound yields the desired result.
E P RIMER ON THE UPPER AND LOWER CONFIDENCE BOUNDS
This section contains the details for the concentration results used in this work.
Results presented in this section are not new were developed in a series of recent works (Waudby-Smith & Ramdas, 2021;
Howard et al., 2021).
We follow the notation from Waudby-Smith & Ramdas (2021) for consistency and use the superscript ( t ) when referring to conﬁdence sequences (CS) and ( n ) when referring to conﬁdence intervals (CI).
Predictably-mixed Hoeffding’s (PM-H) conﬁdence sequence.
Then the upper and lower endpoints of the predictably-mixed Hoeffding’s (PM-H) conﬁdence sequence are given by:
[Equation]
where ψ H ( λ ) :
= λ 2 / 8 and λ 1 , λ 2 , . . .
is a predictable mixture.
We use a particular predictable mixture given by:
[Equation]
When approximating the risk on the source domain, one would typically have a held-out sample of a ﬁxed size n , and so one could either use the classic upper limit of the Hoeffding’s conﬁdence interval, which is recovered by taking equal λ i
= λ =
[Equation]
/n , i = 1 , . . . , n , in which case the upper and lower limits simplify to:
[Equation]
(cid:33) , or by considering running intersection of the predictably mixed Hoeffding’s conﬁdence sequence: (min t ≤ n U ( t ) PM-H , max t ≤ n L ( t ) PM-H ) .
Predictably-mixed empirical-Bernstein (PM-EB) conﬁdence sequence.
The upper and lower endpoints of the predictably-mixed empirical-Bernstein (PM-EB) conﬁdence sequence are given by:
[Equation]
where v i := 4 ( X i − (cid:98) µ i − 1 ) 2 , and ψ E ( λ ) :
= ( − log(1 − λ ) − λ ) / 4 , for λ ∈ [0 , 1) .
One particular choice of a predictable mixture ( λ PM-EB t ) ∞ t =1 is given by:
[Equation]
t
[Equation]
for some c ∈ (0 , 1) .
We use c = 1 / 2 and also set
[Equation]
σ 0 = 1 / 4 .
If given a sample of a ﬁxed size n , we consider running intersection along with the predictable sequence given by:
[Equation]
Betting-based conﬁdence sequence.
Tighter conﬁdence intervals/sequences can be obtained by invoking tools from martingale analysis and deploying betting strategies for conﬁdence intervals/sequences construction proposed in (Waudby-Smith & Ramdas, 2021).
While those can not be computed in closed-form, empirically they tend to outperform previously considered conﬁdence intervals/sequences.
Recall that we are primarily interested in one-sided results, and for simplicity we discuss.
For any m ∈ [0 , 1] , introduce a capital (wealth) process:
[Equation]
where (cid:8) λ
+
[Equation]
m
[Equation]
t =1 and
[Equation]
λ − t
[Equation]
m
[Equation]
∞ t =1 are [0 , 1 /m ] -valued and [0 , 1 / (1 − m )]
-valued predictable sequences respectively.
A particular example of such predictable sequences we use is given by:
[Equation]
where, for example, c = 1 / 2 or 3 / 4 and ˙ λ ± t do not depend on m .
Such choice guarantees, in particular, that the resulting martingale is nonnegative.
For example, the wealth process K + t ( m ) incorporates a belief that the true mean µ is larger than m and the converse belief is incorporated in K − t ( m ) , that is, the wealth is expected to be accumulated under the corresponding belief (e.g., consider m =
and the corresponding K + t (0) with a high value, and m = 1 and the corresponding
K + t (1) with a low value).
Using that K + t ( m ) is non-increasing in m , i.e., m 2 ≥ m 1 , then K + t ( m 2 ) ≤
K + t ( m 1 ) , we thus can use grid search (up to speciﬁed approximation error ∆ grid ) to efﬁciently approximate L ( t ) Bet = inf B + t , where
[Equation]
that is, the collection of all m for which that the capital wasn’t accumulated.
Then we can consider
L ( n ) Bet = max t ≤ n L ( t ) Bet .
When m = µ is considered, none of K ± t ( µ ) is expected to be large, since by
Ville’s inequality:
[Equation]
and thus we know that with high probability the true mean is larger than L ( n ) Bet .
That is,
[Equation]
By a similar argument, we get that with high probability, the true mean is less than U ( n ) Bet = min t ≤ n sup
[Equation]
Conjugate-mixture empirical-Bernstein (CM-EB) conﬁdence sequence.
Below, we present a shortened description of CM-EB and refer the reader to Howard et al.
(2021) for more details.
Assume that one observes a sequence of random variables Z t , bounded in [ a, b ] almost surely for all t , and the goal is to construct a conﬁdence sequence for µ t := t −(cid:80) ti =1 E
i − 1 Z i , the average conditional expectation.
Theorem 4 in Howard et al.
(2021) states that for any ( (cid:98)
Z t ) , [ a, b ] -valued predictable sequence, and any u , the sub-exponential uniform boundary with crossing probability α for scale c = b − a , it holds that:
[Equation]
where a reasonable choice for the predictable sequence ( (cid:98)
Z t ) is given by
[Equation]
The key challenge which is addressed by conjugate mixtures is obtaining sublinear uniform boundaries that allows the radius of the conﬁdence sequences to shrink to zero asymptotically.
When a closed form of the conﬁdence sequence is not required, the gamma-exponential mixture generally yields the tightest bounds.
The procedure relies on the following mixing result (Lemma 2, Howard et al.
(2021)) which states that for any α ∈ (0 , 1) and any chosen probability distribution F on [0 , λ max ) :
[Equation]
yields a sub- ψ uniform boundary with crossing probability α .
When the gamma distribution is used for mixing, m ( s, v ) has a closed form given in [Proposition 9, Howard et al.
Subsequently, the resulting gamma-exponential mixture boundary u CM α ( v ) is computed by numerically solving the equation m ( s, v )
= 1 /α in s .
Howard et al.
(2021) provide the packages for computing the corresponding conﬁdence sequence.
F E XPERIMENTS ON SIMULATED DATA
Figure 13 illustrates data samples for two settings where the presence of label shift is not expected to cause degradation in model performance (measured in terms of absolute increase in misclassiﬁcation risk) for the ﬁrst one ( µ 0 = ( − 2 , 0) (cid:62) , µ 1 = (2 , 0) (cid:62) ), but label shift may potentially degrade performance for the second ( µ 0 =
( − 1 , 0) (cid:62) , µ 1 = (1 , 0) (cid:62) ).
For both cases, samples follow the same data generation pipeline as in Section 1 with only changes in class centers.
[Figure]
F. B RIER SCORE AS A TARGET METRIC
Here we replicate the empirical study from Section 3.1 but use the Brier score as a target metric.
Recall that all three multiclass versions of the Brier score discussed in this work reduce to the same loss in the binary setting.
First, we compare upper conﬁdence bounds for the Brier score computed by invoking different concentration results on Figure 14.
Similar to the misclassiﬁcation risk, variance-adaptive conﬁdence bounds exploit the low-variance structure and are tighter when compared against the non-adaptive one.
Next, we perform empirical analysis of the power of the testing framework.
We take ε tol =
.which corresponds to testing for a 10% relative increase in the Brier score.
We take n S = 1000 data points from the source distribution to compute the upper conﬁdence bound on the source risk (cid:98)
U S ( f ) .
Subsequently, we sample the data from the target distribution in batches of 50, with maximum number of samples from the target set to be 2000.
On Figure 14b, we present the proportion of cases when the null hypothesis is rejected out of 250 simulations performed for each candidate class 1 probability.
On Figure 14c, we illustrate average sample size from the target domain that was needed to reject the null hypothesis.
When a stronger and more harmful label shift is present, less samples are required to reject the null, and moreover, the most powerful tests utilize upper/lower conﬁdence bounds obtained via the betting approach.
On Figure 14d, we present the comparison of different time-uniform lower conﬁdence bounds.
G E XPERIMENTS ON REAL DATASETS G. MNIST-C SIMULATION
Architecture and training.
For MNIST-C dataset, we train a shallow CNN with two convolutional layers (each with 3 × 3 kernel matrices), each followed by max-pooling layers.
Subsequently, the result is ﬂattened and followed by a dropout layer ( p = 0 . 5 ), a fully-connected layer with
[Figure]
neurons and an output layer.
Note that the network is trained on original (clean) MNIST data, which is split split into two folds with 10% of data used for validation purposes.
All images are scaled to [0 , 1] range before the training is performed.
Training multiple networks.
To validate observations regarding shift malignancy from Section 3.2, we train 5 different networks (following the same training protocol) and report aggregated (over 25 random ordering of the data from the target) results on Figure 15.
The observation that applying translation to the MNIST images represents a harmful shift is consistent across all networks.
[Figure]
G. CIFAR
10-C SIMULATION
Architecture and training.
The model underlying a set-valued predictor is a standard ResNet-32.
It is trained for 50 epochs on the original (clean) CIFAR-10 dataset, without data augmentation, using 10% of data for validation purposes.
All images are scaled to [0 , 1] range before the training is performed.
The accuracy of the resulting network is ≈ 80 .
Transforming a point predictor into a set-valued one.
To transform a point predictor into a setvalued one, we consider a sequence of candidate prediction sets S λ ( x ) , parameterized by univariate parameter λ , with larger λ leading to larger prediction sets.
Under the considered setting, the underlying predictor is an estimator of the true conditional probabilities π y ( x ) =
P ( Y = y
| X = x ) .
Given a learnt predictor f , we can deﬁne
[Equation]
to represent estimated probability mass of the labels that are more likely than y .
Subsequently, we can consider the following sequence of set-valued predictors:
[Equation]
that is the sequence is based on the estimated density level sets, starting by including the most likely labels according to the predictions of f .
To tune the parameter λ , we follow Bates et al.
(2021): we keep a labeled held-out calibration set , and use it to pick:
[Equation]
where (cid:98)
R + ( λ (cid:48) ) is an upper conﬁdence bound for the risk function at level β .
The resulting set-valued predictor is then ( β, γ ) -RCPS, that is,
[Equation]
under the i.i.d.
assumption.
More details and validity guarantees can be found in Bates et al.
Set-valued predictor when β
= 0 .
05 is used as a prescribed error level.
In contrast to β = 0 .
1 used in the main paper, we also consider decreasing β to 0.05, which in words, corresponds to increasing a desired coverage level of the resulting set-valued predictor.
Figure 16a compares average sizes of the prediction sets for two candidate values: β 1 = 0 .
05 and β 2 = 0 .
1 , when the set-valued predictor is passes either clean CIFAR-10 data, or images to which fog corruption has been applied.
As expected, decreasing β leads to larger prediction sets on average, with the size increasing when corrupted images are passes as input, that the size reﬂects uncertainty in prediction.
In Figure 16b, we observe that when we run the testing framework for the set-valued predictor corresponding to β = 0 .
05 , only the most severe version of corruptions by adding fog is consistently marked as harmful, and thus raising an alarm.
Similar to Section 3.2, we also use ε tol = 0 .
H T ESTING FOR HARMFUL COVARIATE SHIFT
In this section, we consider a case when covariate shift is present on the target domain, that is, when the marginal distribution P ( X ) changes but P ( Y | X ) does not.
Consider the binary classiﬁcation setting with accuracy being a target metric.
It is known that the optimal decision rule for this case is the Bayes-optimal rule:
[Equation]
which minimizes the probability of misclassifying a new data point
Then one might expect that a change in the marginal distribution of X does not necessarily call for retraining if a learnt predictor f is ‘close’ to f (cid:63) (which itself could be a serious assumption to make).
However, a change in the marginal distribution of X could indicate, in particular, that one should reconsider certain design choices made during training a model.
To illustrate when it could be useful, we consider the following data generation pipeline:
Initially, each point is assigned either to the origin or to a circle of radius 1 centered at the origin with probability 1 / 2 .
[Figure]
v e r a g
[Figure]
p r e d i c t
i
[Figure]
n s e t s i
z e
[Figure]
For points assigned to the origin, the coordinates are sampled from multivariate Gaussian distribution with zero mean and rescaled identity covariance matrix: 136 I 2 .
For points assigned to the circle, the coordinates are sampled from multivariate Gaussian distribution with the same covariance matrix but with the mean vector: (cid:26)
[Equation]
where ϕ S ∼ Unif([ − π/ 3 , π/ 3]) on the source domain and ϕ T ∼ Unif([0 , 2 π ]) on the target domain (see Figure 17 for a visualization).
Then points are assigned the corresponding labels according to:
[Equation]
It is easy to see that a linear predictor, e.g., logistic regression, can achieve high accuracy if deployed on the data sampled from the source distribution.
However, it will clearly fail to recover the true relationship between the features and responses.
In this case, a change in the marginal distribution of X might indicate that updating a functional class could be necessary.
On this data, we also run the proposed framework testing for a 10% increase in misclassiﬁcation risk ( ε tol = 0 . 1 ).
At each run, we use 200 points to train a logistic regression and 100 points to estimate the betting-based upper conﬁdence bound on the source risk.
On the target domain, we use the lower conﬁdence bound due to conjugate-mixture empirical-Bernstein (CM-EB).
The results presented on Figure 17c (which have been aggregated over 100 random data draws) illustrate that the framework successfully detects a harmful shift, requiring only a small number of samples to do so.
[Figure]