Sequential algorithmic modiﬁcation with test data reuse
Abstract
After initial release of a machine learning algorithm, the model can be ﬁne-tuned by retraining on subsequently gathered data, adding newly discovered features, or more.
Each modiﬁcation introduces a risk of deteriorating performance and must be validated on a test dataset.
It may not always be practical to assemble a new dataset for testing each modiﬁcation, especially when most modiﬁcations are minor or are implemented in rapid succession.
Recent work has shown how one can repeatedly test modiﬁcations on the same dataset and protect against overﬁtting by (i) discretizing test results along a grid and (ii) applying a Bonferroni correction to adjust for the total number of modiﬁcations considered by an adaptive developer.
However, the standard Bonferroni correction is overly conservative when most modiﬁcations are beneﬁcial and/or highly correlated.
This work investigates more powerful approaches using alpha-recycling and sequentially-rejective graphical procedures (SRGPs).
We introduce two novel extensions that account for correlation between adaptively chosen algorithmic modiﬁcations: the ﬁrst leverages the correlation between consecutive modiﬁcations using ﬂexible ﬁxed sequence tests, and the second leverages the correlation between the proposed modiﬁcations and those generated by a hypothetical prespeciﬁed model updating procedure.
In empirical analyses, both SRGPs control the error rate of approving deleterious modiﬁcations and approve signiﬁcantly more beneﬁcial modiﬁcations than previous approaches.
INTRODUCTION
Before a machine learning (ML) algorithm is approved for deployment, its performance is usually evaluated on an independent test dataset.
If the ML algorithm is modiﬁed over time, its performance may change.
There are no guarantees on how the performance may evolve when the model developer is allowed to introduce modiﬁcations in an unconstrained manner.
For instance, algorithmic modiﬁcations that reduce computational costs may negatively impact model accuracy or precision, and improvements along an aggregate performance metric may come at the cost of lower performance for certain minority subgroups and exacerbate issues of algorithmic fairness.
To check that the performance of a proposed modiﬁcation is acceptable for deployment, the current approach is to run a hypothesis test on a new test dataset, separate from the original one (Feng et al., 2020).
The null hypothesis is that the modiﬁcation is not acceptable; a modiﬁcation is approved if we successfully reject the null.
Nevertheless, large high-quality test datasets are often hard to acquire, particularly in the medical setting.
A major motivation for this work comes from the FDA’s recent interest in letting medical device developers update ML-based software, while still ensuring its safety and effectiveness (U.S. FDA, 2019).
As outlined in this discussion paper, there are many modiﬁcations that may be of interest to software vendors, including the addition/removal of features, changing the model class, and retraining the model on new data.
In this work, we consider procedures for testing modiﬁcations that are agnostic to the model class and how the modiﬁcation was trained, e.g. by an online learning algorithm versus a human expert.
When labeled data are expensive and/or difﬁcult to collect, it is tempting to reuse an existing test dataset for determining the acceptability of an algorithmic modiﬁcation.
The danger of test data reuse is that the model developer can learn aspects of the test data when it is used in a sequential and adaptive manner, creating dependencies between algorithmic modiﬁcations and the holdout data.
For instance, the model developer may inadvertently incorporate spurious correlations in the test data to attain over-optimistic performance estimates.
This feedback loop introduces bias to the performance evaluation procedure, and adaptively deﬁned hypothesis tests can have drastically inﬂated Type I error rates (Gelman and Loken, 2017; Thompson et al., 2020).
Recent work tries to protect against inappropriate test data reuse and overﬁtting by reducing the amount of information released by the testing procedure (Russo and Zou, 2016).
The two main approaches are to either coarsen the test outputs along a grid of values (Blum and Hardt, 2015; Rogers et al., 2019) or to perturb the test results with random noise using differential privacy techniques (Dwork et al., 2015a;
Feldman and Steinke, 2018).
However, existing methods require immensely large datasets to provide protection against overﬁtting with theoretical guarantees (Rogers et al., 2019).
Our aim is to design valid test data reuse procedures for smaller sample sizes that still have sufﬁciently high power to approve good algorithmic modiﬁcations.
Our focus is on methods that coarsen the test results.
In fact, we consider the extreme case of coarsening where the procedure releases a single bit of information, e.g. whether or not the modiﬁcation was approved.
When test results are coarsened, the adaptive modiﬁcation strategy can be described as a tree.
As such, one can view test data reuse as a multiple hypothesis testing problem: If we control the family-wise error rate across the entire tree, we control the probability of approving one or more unacceptable modiﬁcations.
Existing procedures perform a Bonferroni correction with respect to the size of this tree (Blum and Hardt, 2015; Rogers et al., 2019).
Nevertheless, the Bonferroni correction is known to be conservative.
Instead, we can gain signiﬁcant power using alpha-recycling (Burman et al., 2009) and accounting for correlation between test statistics (Westfall and Stanley Young, 1993).
Indeed, we expect algorithmic modiﬁcations to be highly correlated when there is signiﬁcant overlap between their training data and similarities in their training procedures.
In this paper, we design valid test data reuse procedures based on sequentially rejective graphical procedures (SRGPs)
(Bretz et al., 2009, 2011a,b).
Although SRGPs are a well-established technique for testing many pre-speciﬁed hypotheses, many of these procedures cannot be applied when the hypotheses are adaptively deﬁned in sequence .
The main challenge is that many nodes in the tree of hypotheses are not observed.
As such, we introduce two novel SRGPs that are able to account for correlation between adaptivelydeﬁned algorithmic modiﬁcations without needing to observe these “counterfactual” hypotheses.
The ﬁrst SRGP accounts for correlation between observed nodes in the tree using a ﬁxed-sequence testing procedure.
The second SRGP is based on the fact that analysts are not adversarial in practice, i.e. they will not purposefully use prior results to overﬁt to the test data (Mania et al., 2019; Zrnic and Hardt, 2019).
To leverage this fact, this second procedure requires the model developer to pre-specify a hypothetical online learning procedure and then utilizes the similarity between the adaptive and pre-speciﬁed modiﬁcations to improve testing power.
In empirical analyses, both procedures protect against overﬁtting to the test data, even for small sample sizes, and approve a higher proportion of acceptable modiﬁcations than existing approaches.
Code is publicly available at https: //github.com/jjfeng/adaptive_SRGP .
PROBLEM SETUP
Suppose the test dataset is composed of n independently and identically distributed (IID) observations ( X 1 , Y 1 ) , ( X 2 , Y 2 ) , . . .
, ( X n , Y n ) ∈
X × Y drawn from the target population.
Consider a model developer who adaptively proposes a sequence of T algorithmic modiﬁcations { ˆ f adapt1 , ..., ˆ f adapt T } , where each modiﬁcation is a model that predicts some value in Y given input X .
Given criteria for deﬁning the acceptability of a modiﬁcation, our goal is to approve as many acceptable modiﬁcations as possible while controlling the probability of approving an unacceptable modiﬁcation.
Because the decision to approve a modiﬁcation can be framed as a hypothesis test, a procedure for approving adaptively-deﬁned modiﬁcations is equivalent to testing a sequence of adaptively-deﬁned hypotheses
H adapt1 , ..., H adapt T .
Moreover, control of the online familywise error rate (FWER) in the strong sense, i.e. for any conﬁguration of the null hypotheses, implies control over of the rate of approving at least one unacceptable modiﬁcation.
There are various ways to deﬁne acceptability and their corresponding hypothesis test.
For example, we may deﬁne a modiﬁcation ˆ f to be acceptable as long as its expected loss is smaller than that of the original model ˆ f 0 .
So given a real-valued loss function (cid:96) , we would test the null hypothesis
[Equation]
adapt j
[Equation]
at each iteration j = 1 , ..., T .
If we require monotonic improvement in the model performance, we can test if the j -th modiﬁcation is superior to the most recently approved modiﬁcation ˆ f approved j ( X ) by testing the null hypothesis
[Equation]
Finally, one may also consider multidimensional characterizations of model performance (e.g. model performance within subgroups) and deﬁne acceptability as a combination of superiority and non-inferiority tests (Feng et al., 2020).
The testing procedures described below only depend on the p-values, so we leave the speciﬁc deﬁnition of acceptability unspeciﬁed until the experimental section.
To limit the adaptivity of the model developer, we consider procedures that sequentially release a single bit of information for each test: One means that the modiﬁcation is approved and zero means it is not.
Consequently, modiﬁcations proposed by any adaptive strategy can be described as a bifurcating tree, where ˆ f a t is the nonadaptive model tested at time t for the history of approvals a t ∈ { 0 , 1 } t − 1 ,
H a t is the associated nonadaptive hypothesis test, and p a t is its marginal p-value.
While one can regard this set of (2 T − 1) hypotheses tests as prespeciﬁed, we are only able to observe a speciﬁc path along this tree.
The unobserved hypotheses are counterfactuals.
As such, we need a multiple testing procedure (MTP) that controls the FWER for any adaptively chosen path along the tree without knowing the exact nature of the counterfactual hypothesis tests .
A simple approach is to perform a uniform Bonferroni correction for the size of the entire tree.
However, the standard
Bonferroni procedure has low power because it ignores correlations between models and allocates substantial test mass to hypotheses that are unlikely to be considered.
Next we describe procedures that can achieve much higher power.
2.1 SEQUENTIALLY REJECTIVE GRAPHICAL PROCEDURES (SRGPS)
We can design more powerful test data reuse procedures by building on sequentially rejective graphical procedures (SRGPs), which use directed graphs to deﬁne a wide variety of iterative MTPs such as gatekeeping procedures, ﬂexible ﬁxed sequence tests, and fallback procedures (Bretz et al., 2009).
SRGPs traditionally assume the set of hypotheses { H j : j ∈ I } is prespeciﬁed and known.
The graph initially contains one node for each elementary hypothesis H j , where each node is associated with a non-negative weight w j ( I ) .
The initial node weights, which are constrained to sum to one, control how the total alpha is divided across the elementary hypotheses and correspond to a set of adjusted signiﬁcance thresholds
c j ( I ) .
We reject elementary hypothesis H j in the current graph if its marginal p-value p j is smaller than c j ( I ) .
For instance, a standard Bonferroni correction is represented by the initial weights of w j ( I ) = 1 / | I | for all j ∈ I and signiﬁcance thresholds c j ( I ) =
w j ( I ) α .
In addition, the graph contains directed edges where the edge H j to H k is associated with weight g j,k ( I ) for j, k ∈ I and edge weights starting from the same node must sum to one.
When an elementary hypothesis H j is rejected, its node is removed from the graph and its weight is propagated to its children nodes.
This redistribution of test mass, also known as alpha-recycling, increases the power for testing the remaining hypotheses and strictly improves upon simpler procedures that do not use recycling.
More speciﬁcally, the weight of the edge from H j to H k , denoted g j,k ( I ) , represents how much of H j ’s node weight will be redistributed to H k if H j is rejected.
So when H j is removed, the new weight for hypothesis H k for k ∈ I (cid:48) =
I \ { j } is
[Equation]
H H H0. H
[Figure]
H
[Figure]
0.88 H
Outgoing edges for all remaining nodes are also renormalized to sum back to one.
The SRGP continues until no more hypotheses can be rejected.
See Figure 1 for an example.
Assuming all test reports are binary, we can describe adaptive test data reuse as the following prespeciﬁed SRGP.
Let
I t be the set of hypotheses remaining at time t , where I 0 is the initial set.
We only consider SRGPs with nonzero edge weights from hypothesis H a t to the sequence of hypotheses that would be tested upon rejection of H a t but prior to the next rejection (though one may also consider more complex recycling procedures).
For such SRGPs, the graph of hypotheses has the tree structure seen in Figure 2a, where the only edges in the tree are between hypotheses H a t and H a t (cid:48) for t < t (cid:48) and [Equation]
Note that the SRGP tree is not the same as the bifurcating tree for generating hypotheses, as the former describes how alpha is recycled.
The model developer must prespecify all initial node weights w a t ( I 0 ) .
A simple approach is to perform a uniform Bonferroni correction across all nodes in the graph.
We can achieve more power by assigning larger weights to nodes that are more likely to be tested.
For example, if the model developer knows that all their modiﬁcations will be approved, they should set the initial node weight for H () and all edge weights along the top path in Figure 2a to one.
[-Algorithm 
-]The model developer will only need to incrementally reveal the edge weights. Let τ t denote the time of the latest approval prior to time t . At time t , the developer must specify the edge weights g a τt ,a t such that the outgoing weights from a τ t sum to no more than one. (Note that the edge weight can be treated as a constant because the only relevant edge weight at time t is g a τt ,a t ( I t ) and its value is equal to g a τt ,a t ( I t (cid:48) ) for all t (cid:48) < t .)
As such, this procedure for specifying node and edge weights corresponds to a fully prespeciﬁed SRGP where a subset of the edge weights are revealed sequentially. To make sure that this SRGP can be executed in the adaptive setting, we must be able to calculate the adjusted signiﬁcance thresholds for the adaptive hypotheses given the current set of node weights without observing the counterfactual hypotheses. [Table] (c)
Require:
Initialize I 0 as the set of all nodes in the prespeciﬁed tree; initialize a 1 = () and τ 1 = 0 ; choose node weights w a t (cid:48)
( I 0 ) for all t (cid:48)
= 1 , 2 , . .
, T and a t (cid:48) ∈ { 0 , 1 } t (cid:48) − 1 ; and set w a 0 ( I 0 )
= 0 .
Ensure: (cid:80) t (cid:48) ,a t (cid:48)
w a t (cid:48) ( I 0 )
= 1 .
for t = 1 , 2 , ..., T do
Specify edge weight g a τt ,a t that satisﬁes outgoing edge weight constraints.
# Weight propagation
[Equation]
for all t (cid:48) , a t (cid:48) such that H a t (cid:48) ∈
I t and a t (cid:48) (cid:54) =
a t do # Other weights remain unchanged
w a t (cid:48) ( I t ) =
w a t (cid:48) ( I t − 1 ) end for
Let p a t be the marginal p-value from testing H a t .
Compute signiﬁcance threshold c a t ( I t ) using compute_sig_threshold ( a t , { w a t (cid:48)
( I t ) :
t (cid:48) , a t (cid:48) } ) if p a t ≤ c a t ( I t ) then
Report that ˆ f adapt t has been approved.
# Remove node
I t +1 = I t \ { a t } τ t +1 = t a t +1 =
( a t , 1) else
Report that ˆ f adapt t has not been approved.
τ t +1 =
τ t a t +1 =
( a t , 0) end if end for
The entire SRGP algorithm for testing adaptive algorithmic modiﬁcations is outlined in Algorithm 1.
It accepts some function compute_sig_threshold that outputs the signiﬁcance threshold for the adaptively chosen hypothesis given node weights in the current tree.
To prove that an
SRGP with function compute_sig_threshold controls the FWER, we must show that it is a closed test procedure that satisﬁes the consonance property.
The consonance property is deﬁned as follows.
Recall that a closed test procedure uses the following recipe to control the FWER at level α : it rejects an elementary hypothesis
H j if the intersection hypothesis H K = ∩ k ∈ K H k for every subset K ⊆
I containing the elementary hypothesis H j is rejected at level α (Lehmann and Romano, 2005).
A closed test satisﬁes the consonance property if the following is true for all J ⊆
I : if intersection hypothesis H J is rejected locally (i.e. its p-value is no more than α ), there exists some j ∈ J such that H K can be rejected locally for all K ⊆ J with j ∈ K (Gabriel, 1969).
In particular, it follows that the corresponding elementary hypothesis H j can be rejected by the closed test procedure.
When the hypothesis tests are fully prespeciﬁed, the consonance procedure lets us perform a closed test using a sequentially rejective (or “shortcut”) procedure that iteratively rejects the elementary hypotheses without needing to test every intersection hypothesis (Hommel et al., 2007).
So the main beneﬁt of consonance in the non-adaptive setting is that it makes closed testing computationally tractable.
In the adaptive setting, the consonance property is even more important because we can perform closed testing without observing counterfactual or future hypotheses .
That is, the consonance property of an SRGP in the adaptive setting is not simply for computational efﬁciency, but is necessary for being able to run the MTP at all.
Below, we will describe three SRGPs for testing an adaptive sequence of algorithmic modiﬁcations, presented in order of increasing complexity.
Each differ in how compute_sig_threshold is deﬁned.
To prove that the procedures satisfy the consonance property, it is sufﬁcient to show that the following monotonicity condition holds (Bretz et al., 2009):
For every pair of subsets K, J ⊆
I where K ⊆ J and j ∈ K , we have
[Equation]
All proofs are provided in the Appendix.
The computation time and memory for the proposed SRGPs are all
O ( T 2 ) .
To see this, note that the procedures update no more than T node weights at each iteration.
2.2 BONFERRONI-BASED SRGPS
We begin with the simplest SRGP that performs closed testing with a weighted Bonferroni-Holm correction based on node weights, which was originally proposed in Bretz et al.
(2009) to test a set of fully pre-speciﬁed hypotheses.
Nevertheless, this procedure can also be applied in the adaptive setting because the signiﬁcance thresholds do not depend on observing the counterfactual hypotheses.
In particular, this procedure tests the t -th adaptive hypothesis given history a t by comparing its marginal p-value to the corrected signiﬁcance threshold c a t ( I t ) =
w a t ( I t ) α .
Because this closed test satisﬁes the monotonicity condition, Algorithmwith this signiﬁcance threshold controls the FWER for the adaptive hypotheses at level α .
As a simple example, consider an SRGP that initially assigns Bonferroni-corrected weights to every node and selects nonzero edge weights.
This is more powerful than performing a standard Bonferroni correction without any alpharecycling because the signiﬁcance thresholds are monotonically non-decreasing at each iteration.
2.3 SRGPS WITH FLEXIBLE FIXED SEQUENCE TESTS FOR CORRELATED MODIFICATIONS
When the algorithmic modiﬁcations are highly correlated, a
Bonferroni-based SRGP will be conservative.
We can design more powerful SRGPs by taking into account correlation between the p-values.
Bretz et al.
(2011a) proposed a procedure that calculates an inﬂation factor c ( I ) for intersection hypothesis I such that the probability there exists an elementary hypothesis H N with marginal p-value p N less than c ( I ) w N ( I ) α , under the null I , is no more than α .
Millen and Dmitrienko (2011) proposed a similar procedure but for test statistics and critical values.
Unfortunately, both procedures require knowing the exact correlation structure between all the hypotheses and checking that the monotonicity property holds.
This is not feasible in the adaptive setting.
To resolve these issues, we propose a new SRGP that (1) partitions the hypothesis tree into sequences of observed hypotheses and (2) uses a ﬂexible ﬁxed-sequence test within each subgroup (Huque and Alosh, 2008).
We group together hypotheses that would be tested along a streak of failures immediately following a successful approval (Figure 2b).
For each history a t ∈ { 0 , 1 } t − 1 with a t,t − 1 = 1 , we deﬁne a subgroup G a t as the set of hypotheses with histories a t (cid:48) =
( a t ,(cid:126) 0) for any length zero vector,
[Equation] e. [Equation]
To test intersection hypothesis I , we test each subgroup
G [Equation] ∩ I at level (cid:16)(cid:80)
[Equation]
We reject H [Equation] at level α if any of the subgroup-speciﬁc tests are rejected.
We can show that this controls the Type I error at level α using a union bound.
To test a subgroup, we test its hypotheses in the order they are revealed and spend up to the allocated alpha weight.
To satisfy the monotonicity property, the signiﬁcance threshold c a j ( I ) for a j ∈ G a t ∩ I is deﬁned as the maximum threshold that spends no more than the allocated alpha up to time j for all subsets of hypotheses,
i.e.
[Equation]
This expression is complicated because it handles arbitrary correlation structures between the p-values.
It greatly simpliﬁes in certain cases.
For example, if we are performing one-sided Z-tests and the pairwise correlations of the model losses are non-negative, (4) is equivalent to deﬁning c a j ( I ) as the solution to
[Equation]
Using the ﬂexible ﬁxed sequence tests from above, we sequentially calculate the signiﬁcance thresholds and test the adaptive hypotheses.
When a hypothesis is rejected, we remove its node and propagate its local weight to its children nodes per (1).
We can prove the monotonicity condition holds to establish the following result:
Theorem 1.
Algorithm 1 with signiﬁcance thresholds chosen using (4) controls the FWER for adaptively deﬁned hypotheses at level α .
2.4 SRGPS WITH PRESPECIFIED HYPOTHETICAL MODEL UPDATES
The SRGPs in the above sections protect against the worst case scenario where the model developer is adversarial.
In practice, the model developer may have a plan for how they will update their model over time (i.e. continually reﬁt the model on accumulating data) and will only make small adjustments based on test results.
As such, we do not expect the adaptively chosen model at iteration t to stray far from the initial plan.
In the most extreme case, we may ﬁnd that the model developer is not adaptive at all and follows the prespeciﬁed procedure perfectly; instead of correcting for (2 T − 1) hypotheses, we would expect that the correction factor to be O ( T ) instead.
To leverage this similarity assumption, we propose a novel
SRGP that requires the model developer to prespecify a procedure for generating hypothetical model updates.
This prespeciﬁed procedure describes the exact steps for how modiﬁcations would be generated, e.g. the data stream used, the number of training observations, and hyperparameter selection.
These hypothetical model updates are included as additional nodes in the hypothesis graph and assigned positive node weights.
Their sole purpose is to improve power for approving the adaptively-deﬁned model updates.
These model updates are never formally tested nor approved for deployment.
We also do not release any information about their test performance, because doing so would increase the amount of information leaked to the model developer and the branching factor of the adaptive tree.
At each iteration, this SRGP constructs a conﬁdence region for the performance of the t -th prespeciﬁed model update ˆ f pres t by spending its allocated alpha, accounting for its correlation with all prespeciﬁed models up to iteration t − 1 .
It then tests the t -th adaptive model by accounting for its correlation with the prespeciﬁed models up to iteration t .
As such, the power for testing the adaptive modiﬁcations increase as their correlation with the prespeciﬁed updates increases.
More formally, the critical value and signiﬁcance threshold at time t are calculated as follows.
Let P 0 denote the target population and P n denote the empirical distribution of the test dataset.
Here we consider a univariate performance measure ψ , where ψ (cid:16) ˆ f, P (cid:17) is the performance of model ˆ f with respect to distribution P .
It is straightforward to extend this procedure to multivariate performance measures (see the Appendix for an example).
Denote the deviation between the estimated and true performance as
[Equation]
For intersection hypothesis I , deﬁne ˜
I as union of I and all prespeciﬁed nodes.
Deﬁne critical value z pres t ( I ) as the largest ˜ z such that
[Equation]
The signiﬁcance threshold c a t ( I ) for testing H a t is deﬁned as the largest ˜ c such that
[Equation]
Crucially, these calculations do not depend on observing counterfactual or future hypotheses.
Using a union bound, we can show that the Type I error for falsely rejecting the intersection hypothesis I using the critical values deﬁned above is bounded by the sum of the right hand sides of (5) and (6) for all ( t, a t ) in I .
Because the total weight in the graph is always one, we achieve Type I error control at level α .
Using this idea, we can show that this SRGP indeed controls the FWER:
Theorem 2.
Algorithm 1 using signiﬁcance thresholds deﬁned using equations (5) and (6) control FWER at level α for adaptively selected hypotheses.
SIMULATION STUDIES
We now present two simulation studies of model developers who adaptively propose modiﬁcations to their initial ML algorithm.
The developers aim to improve the model’s area under the receiver operating characteristic curve (AUC) and quantify the performance increase as accurately as possible.
Because our adaptive test data reuse procedures only release a single bit of information at each iteration, we must carefully design the hypothesis tests to obtain a numeric bound on the performance improvement.
In particular, we deﬁne the j -th adaptive hypothesis test as
[Equation]
where ψ ( f, P ) denotes AUC of model f for distribution P and δ adaptj ≥ 0 is the improvement difference that we are trying to detect.
To ensure the model performance tends to improve with each approval, we set δ adaptj +
= δ adaptj
+ 0 .whenever the j -th null hypothesis is rejected.
Note that one could consider more complicated hypotheses, each with their pros and cons.
For example, one can check that the modiﬁcations are strictly improving and test for an improvement difference; however, this can be overly stringent.
The purpose of the ﬁrst simulation study is to investigate
FWER control.
We do this by simulating a model developer who tries to overﬁt to the test data based on the information released at each iteration.
The purpose of the second simulation study is to investigate power.
Here the model developer generally proposes good algorithmic modiﬁcations by continually reﬁtting the model given an IID data stream.
In both simulations, we generate X ∈ R 100 using a multivariate Gaussian distribution.
Y is generated using a logistic regression model where the coefﬁcients of the ﬁrst six variables are 0.75 and all other model parameter are zero.
The for approving algorithmic modiﬁcations generated by an adversarial model developer.
At each iteration, the model developer tests a model that is overﬁt to the reusable test dataset.
We plot the AUCs of the most recently approved model on the reusable test dataset and a completely held out test dataset left) the rate of incorrectly approving at least unacceptable detected increase
’Value’ is deﬁned modiﬁcations are also logistic regression models.
We evaluate the two SRGPs proposed in this paper—SRGP with ﬂexible ﬁxed sequence tests ( fsSRGP ) and SRGP with hypothetical prespeciﬁed model updates ( presSRGP )—against relevant baseline comparators, including the standard Bonferroni procedure ( Bonferroni ), a weighted Bonferroni procedure ( wBonferroni ), and the Bonferroni-based SRGP ( bonfSRGP ).
For wBonferroni , we assigned weights to favor hypotheses with more successful approval histories.
In particular, we split the adaptive hypotheses into two groups: those with an approval history with no more than three failed approvals and those with more than three.
We then assigned weights such that the hypotheses were equally weighted within each group and the total weight for each group was 0.5.
The weights in the SRGPs were deﬁned such that the ﬁrst outgoing edge (a successful rejection of the hypothesis) is 0.8 and for each subsequent edge, it was assigned 0.8 of the remaining weight.
Unless speciﬁed otherwise, all the MTPs control the FWER at level α = 0 .
Details for deriving test statistics and signiﬁcance thresholds are provided in the Appendix.
3.1 VERIFYING FWER CONTROL
Here we show how MTPs that fail to control the FWER can drastically elevate one’s risk of overﬁtting to the test data, as compared to appropriately-designed adaptive test data reuse procedures.
In particular, we consider the naïve procedure that tests every adaptive hypothesis at level α ( BinaryThres ).
The reusable test dataset has 100 observations and the model developer tests T = 50 modiﬁcations.
For the purpose of illustration, the initial model is set to the oracle, so all proposed modiﬁcations are unacceptable.
The simulated model developer tries to ﬁnd models that overﬁt to the test data by searching within the neighborhood of the currently approved model.
In particular, the developer iteratively perturbs the coefﬁcient of each irrelevant variable by 0.6 in the positive and negative directions.
When any such modiﬁcation is approved, the model developer will continue perturbing that coefﬁcient in the same direction until it fails to reject the null hypothesis.
For presSRGP , the prespeciﬁed model update at iteration t is the model with coefﬁcients exactly the same as the initial model except that the coefﬁcient for the (7 + (cid:98) t/ 2 (cid:99) ) -th
variable is set to 0.6 if t is even and -0.6 if t is odd.
Figure 3 shows the result from 100 replicates.
Notably,
BinaryThres approves at least one inferior modiﬁcation with probability 75% and concludes that the modiﬁcations by the last iteration improves the AUC by at least 0.02, even though the AUC actually drops by 0.025 on average .
All the other MTPs appropriately control the FWER at the desired rate of 10% and, thus, protect against over-ﬁtting.
3.2 ASSESSING POWER
Here the simulated model developer has access to an IID data stream and iteratively reﬁts a logistic regression model on this data.
Because training on more data from the target population tends to improve model performance, the modiﬁcations are usually beneﬁcial.
However, there is a risk that the modiﬁcation does not improve performance or that the improvement is negligible, especially because there is a potential for overﬁtting to the reusable test data set.
By testing hypotheses (7), we can restrict approval to only those model updates with meaningful improvements in the AUC.
The test dataset has 800 observations and we allow T =adaptive tests.
At each time point, the model developer receives a new observation and reﬁts the model.
To spend alpha more judiciously, the model developer will only submit the reﬁtted model if the power calculations suggest that the probability for rejecting the null hypothesis exceeds 50%.
Speciﬁcally, they perform power calculations by setting the true performance improvement to the CI lower bound, which is estimated using split-sample validation.
(For simplicity, the power calculations do not perform any multiple testing correction.)
To run presSRGP , the prespeciﬁed model updating procedure also selects updates based on a hypothesis test similar to (7) but replacing the adaptive difference sequence δ adapt j with the prespeciﬁed difference sequence δ pres j = 0 .
0025( j − 1) as well as replacing the adaptive modiﬁcations with the prespeciﬁed ones.
The procedures differed signiﬁcantly in power (Figure 4).
presSRGP performed the best, followed by fsSRGP . In particular, the total number of approved modiﬁcations and the average AUC of the ﬁnal approved model were higher when using presSRGP compared to fsSRGP (p=0.05 and 0.03, respectively).
Both presSRGP and fsSRGP signiﬁcantly outperformed the other methods ( p ≤ 0 . 005 for all comparisons).
DATA ANALYSIS: PREDICTING ACUTE HYPOTENSION EPISODES
We now apply our procedure for approving modiﬁcations to a risk prediction model for acute hypotension episodes (AHEs), one of the most frequent critical events in the intensive care unit (ICU)
(Walsh et al., 2013).
The ICU is a clinical environment that continuously generates high throughput data.
Thus, a model developer can readily collect new data in this setting to retrain an existing model.
To mimic this, we use data from the eICU Collaborative Research Database (Pollard et al., 2018).
We randomly select 40 admissions to train an initial model, reserve another 500 admissions to construct the reusable test dataset, and use the remainingadmissions to simulate a data stream.
Data from one admission is observed at each time point, and the model developer is allowed to test a maximum of T = 15 modiﬁcations.
The task is to predict AHE 30 minutes in advance, where we deﬁne AHE as any 5-minute time period where the average mean arterial pressure (MAP) falls below 65 mmHg.
The input features to the model are baseline variables age, sex, height, and weight; vital signs MAP, heart rate, and respiration rate at the current time point; and the same set of vital signs ﬁve minutes prior.
The prediction model is a gradient boosted tree (GBT) and is continually reﬁt on the incoming data.
Here we consider a more complex hypothesis test that checks for calibration-in-the-large (Steyerberg, 2009) and improvement in AUC.
The j -th adaptive null hypothesis is
H adapt j : ψ (cid:16) ˆ f adapt j ; P 0 (cid:17) ≤ ψ (cid:16) ˆ f 0 ; P 0 (cid:17) + δ adapt j or E (cid:104) ˆ f adapt j ( X ) − Y (cid:105) (cid:54)∈ [ − (cid:15), (cid:15) ] , (8) where δ adapt j is deﬁned using the same procedure as that in
Section 3.2, ˆ f adapt j is the modiﬁcation determined to have sufﬁcient power for rejecting the null, and margin of error (cid:15) is 0.05.
We will refer to E (cid:104) ˆ f adapt j ( X ) − Y (cid:105) as calibration- error-in-the-large.
Details on calculating the test statistic and signiﬁcance thresholds are provided in the Appendix.
Results from 40 replicates are shown in Figure 5.
We observe the same ranking of MTPs as that in Section 3.2. The total number of approved models is highest using presSRGP compared to the other MTPs ( p ≤ 0 .for all comparisons). Also, the AUC of the ﬁnal model approved was also higher using prespecSRGP compared to Bonferroni , wBonferroni , bonfSRGP , and fsSRGP ( p = 0 . 002 , 0 . 010 , 0 . 020 , 0 . 015 , respectively).
Compared to the previous section, the relative improvement between the methods is smaller because the GBTs improved rapidly at early time points and slowed down thereafter.
To better understand how performance can vary across different datasets, we have included a second data analysis in the Appendix based on a medical imaging task.
There, we ﬁnd that fsSRGP and presSRGP outperform the rest of the methods, and their approval rates are nearly indistinguishable.
RELATED WORK
Our paper relates to a large body of work on methods for providing valid statistical inference and preventing false discoveries.
Much of this literature has focused on testing prespeciﬁed hypotheses on the same dataset while controlling the FWER (Hochberg and Tamhane, 1987; Westfall et al., 2010), false discovery rate (FDR) (Benjamini and
Hochberg, 1995), or some variant thereof (van der Laan et al., 2004).
More recent work consider testing a sequence of adaptive hypotheses on prospectively-collected data from a data stream and controlling online error rates (Foster and
Stine, 2008; Ramdas et al., 2018).
This work considers the setting where we adaptively test hypotheses on the same dataset.
To control for the bias of reusing the same dataset, testing procedures must limit the amount of information released about the test data (Russo and Zou, 2016).
Techniques based on differential privacy, which is a mathematically rigorous formalization of data privacy (Dwork and Roth, 2014), do this by adding random noise (e.g. Laplace or Gaussian noise) to the test statistic or, more generally, the queried result (Dwork et al., 2015a).
While theoretical guarantees are available for differential privacy based methods for test data reuse (e.g., (Dwork et al., 2015b; Russo and Zou, 2016; Rogers et al., 2016; Cummings et al., 2016;
Dwork et al., 2017; Feldman and Steinke, 2017, 2018; Shenfeld and Ligett, 2019; Gossmann et al., 2021) and others), the required size of the test dataset is prohibitively large for many application domains or require injecting very large amounts of noise (Rogers et al., 2019; Gossmann et al., 2021).
An alternative approach is to directly limit the number of bits of information released to the model developer by discretizing the queried result along some grid (Blum and Hardt, 2015).
Existing methods essentially perform a Bonferroni correction for the number of distinct hypotheses, which also require unreasonably large test datasets for many applications.
To improve testing power, a number of works have assumed that the adaptivity of the model developer is limited (e.g. the models are highly correlated, or the model developer is not entirely “adversarial”) to justify the use of a less conservative correction factor (Mania et al., 2019; Zrnic and Hardt, 2019).
In contrast, the SRGPs proposed in this work achieve higher power via alpha-recycling and account for the correlation structure without needing to make assumptions about the model developer.
CONCLUSION
We show how to leverage SRGPs to design valid and powerful approaches for testing a sequence of adaptively-deﬁned algorithmic modiﬁcations on the same dataset.
The overall steps of this framework are (i) limit the amount of information leakage by reporting only binary test results (approve versus deny modiﬁcations), (ii) spend and recycle alpha using an SRGP, and (iii) design consonant, closed-testing procedures whose signiﬁcance thresholds can be computed without needing to observe the counterfactual hypotheses.
To account for correlation between the algorithmic modiﬁcations, we presented two new SRGPs.
fsSRGP achieves higher power by leveraging the correlation structure between the observed algorithmic modiﬁcations.
presSRGP asks the model developer to generate a sequence of algorithmic modiﬁcations using a prespeciﬁed learning procedure and leverages the correlation between the adaptive and prespeciﬁed algorithmic modiﬁcations.
In empirical studies, these procedures approved more algorithmic modiﬁcations than existing methods, with presSRGP achieving the highest power.
A limitation of this work is that the model developers were simulated, because there are no publicly available datasets that document modiﬁcation strategies taken by real-world ML practitioners. In particular, we designed a modiﬁcation strategy based on one of the most commonly used strategies in the literature: retraining the model on all previously accumulated data (Breck et al., 2017; Amershi et al., 2019).
To provide a more complete picture, we have included sensitivity analyses in the Appendix to understand how variations in the model developer’s strategy can impact the performance of the proposed methods.
Generally, we ﬁnd that the additional gain in power using presSRGP and fsSRGP over the other methods depends on the modiﬁcation strategy.
As such, one direction of future work is to design modiﬁcation strategies that can be paired with these MTPs to maximize power.
Other areas for future research include tuning the node and edge weights in the SRGPs to optimize statistical power and extending the SRGPs to control other error rates (e.g. k -FWER and False Discovery Rate)
(Robertson et al., 2020).
In addition, model developers are often interested in obtaining more detailed test results like p-values and conﬁdence intervals.
So an important next step is to design SRGPs that release more information per iteration, perhaps by leveraging differential privacy techniques.
Acknowledgements
We thank Noah Simon, Charles McCulloch, and Zhenghao Chen for helpful discussions and suggestions.
We are grateful to Nicholas Fong for sharing cleaned eICU data.
This work was supported by the Food and Drug Administration (FDA) of the U.S. Department of Health and Human
Services (HHS) as part of a ﬁnancial assistance award Center of Excellence in Regulatory Science and Innovation grant to University of California, San Francisco (UCSF) and Stanford University, U01FD005978 totaling $79, with 100% funded by FDA/HHS.
The contents are those of the author(s) and do not necessarily represent the ofﬁcial views of, nor an endorsement, by FDA/HHS, or the U.S.
Government.