Estimation of prediction error with known covariate shift
Abstract
In supervised learning, the estimation of prediction error on unlabeled test data is an important task.
Existing methods are usually built on the assumption that the training and test data are sampled from the same distribution, which is often violated in practice.
As a result, classical estimators like cross-validation (CV) will be biased and this may result in poor model selection.
In this paper, we assume that we have a test dataset in which the feature values are available but not the outcome labels, and focus on a particular form of distributional shift of covariate shift.
We propose an alternative method based on parametric bootstrap of the target of conditional error Err X [2].
Empirically our method outperforms CV for both simulation and real data example across different modeling tasks, and is comparable to state-of-the-art methods for image classification.
Introduction
In predictive modeling, it is essential to estimate the generalization error on future test datasets.
Given a particular model, such generalization error is implicitly dependent on the distribution from which the test data is drawn.
Existing methods such as cross-validation (CV) ([3]) usually rely on stationary assumptions between training and test data, which are often violated in practice due to time shift, location change, sampling bias, batch effects etc.
We consider estimation of generalization error when the covariate shift between training and test data is observed and seek to improve upon existing methods by leveraging the additional covariate information of test data.
Our method is based on a slightly modified version of the target of inference Err X [2] which is the average prediction error of models fit on other unseen training datasets and is shown to be an approximate estimand for CV without covariate shift ([19], [6], [17], [11], [15], [2]). That is, CV can be seen as a special case of Err X estimator under no distribution shift, thus motivating the principled use of Err X as the target of inference for estimation of prediction error under covariate shift. Unlike the instance specific out of sample error, Err X can be conveniently estimated based on parametric bootstrap.
We propose two ways to estimate the target of inference Err X using either direct estimation or decomposition formula, resulting in two alternative estimators Err X .dir and Err X .dec .
We will mainly discuss linear regression and classification, but our method can be applied to any predictive model.
Setup and Notation
We consider the supervised learning setting, where we have a training data set X = ( x 1 , . . .
, x n ) , Y =
( y 1 , . . .
, y n ) of n observations drawn i.i.d from some joint distribution P .
That is, ( x i , y i ) i ∈ [ n ] i.i.d ∼ P ∈ P ( X × Y ) , where X ⊂ R p and Y ⊂ R .
Denote by δ X the empirical distribution of covariates in X , P X as the marginal distribution of covariates, and P Y | X as the conditional distribution of Y given X .
Let ˆ f ( x, θ ) be a function that predicts outcome y from covariates x ∈ R p using a parametric model with parameter θ ∈ Θ ⊂ R d .
Let ˆ θ be a function that maps values from ( X × Y ) n to parameter estimates in Θ .
Suppose that there is a new dataset consisting of i.i.d draws from Q ∈ P ( X × Y ) , where Q could be a different joint distribution but the conditional distribution of outcome given covariates remains the same.
We are interested in predicting the test error, without access to the ground truth outcomes, as measured by a loss function,
[Equation]
such that ℓ ( y, y )
=
for all y .
(For example, ℓ could be square error loss, misclassification error, or deviance.)
The most intuitive target of inference for test error is the out-of-sample error ,
[Equation]
which is the expected loss when applying a model trained with X , Y on a new data point ( x 0 , y 0 ) ∼ Q .
Suppose that we are given a particular unlabeled test set with n test samples and test covariates X test .
Similarly, let δ X test be the empirical distribution of covariates in X test , and Q Y | X be the conditional distribution of outcome given covariates in the test population.
Since we make the assumption that the conditional distribution remains unchanged and only deals with observed covariate shift, we have that
Q Y | X = P Y | X .
Notice that for an abuse of notation, we write y ∼ P Y | X
= X to mean the sampling of a random vector, ( y 1 , . . .
y n ) ∼ P Y | X = x 1 × . . .
× P Y | X = x n .
ErrX method 3.1 Target
We propose to estimate out-of-sample error in equation (1) by studying a similar averaged target first introduced in [2] as follows:
[Equation]
Notice that since Q is usually unknown, it is difficult to estimate Err Q X directly.
But given test data covariates and the assumption of no conditional distribution shift, we can study a slightly modified version of the estimand, by replacing sampling from an unknown joint distribution ( x 0 , y 0 ) ∼ Q with its empirical counterpart.
[Equation]
Since our new target Err Q X is a function of features in the training set, it has connections with insample error , which is the target of estimation for traditional covariance-penalty based methods ([10], [1]), [12], [13]).
The decomposition formula and its correponding estimation method is in Appendix
A.1.
3.2 General method of estimation
Our methods of estimation are based on the idea of parametric bootstrap.
Let P θY | X be a parametric model and ˆ θ be a parameter estimate.
Then drawing parametric bootstrap samples y ∼ P ˆ θ Y | X means generating new outcomes for given covariate information based on the model parameterized by ˆ θ .
For direct estimation of the target in (3), we illustrate the steps in Algorithm 1.
After obtaining the initial parameter estimate from training data, we draw parametric bootstrap samples of pseudo outcomes for both training and test covariates.
For each bootstrap sample, we refit and compute empirical loss with the bootstrap labels.
The final estimate Err X .dir can be obtained as an average of bootstrap errors.
[Equation]
[Equation] [Equation] [Equation]
[Equation] [Equation] [Equation]
Similarly, for estimation of ErrX target via decomposition, we illustrate the steps in Algorithm 2 in
Appendix.
For the remaining of the paper, we will denote estimators for the target in (3) as Err X .dir (introduced later in Algorithm 1) and that for the decomposition in (5) as Err X .dec (introduced later in Algorithm 2). 3.3 Debias estimation
The performance of Err X estimator as described in Algorithm 1 relies on the accuracy of ˆ θ from which we generate bootstrap samples.
While it can be shown that both Err X .dir and Err X .dec are unbiased when fitting ordinary least squares under the correct linear model, it is usually not the case when the initial fitted parameter is biased and when the model is not well specified. In such cases, debiasing modification is needed in order to achieve better prediction accuracy. In general, one can fit a saturated model such as deep neural network as a basis for the parametric bootstrap step for debiasing. However, as can be seen in the experimental results section below, application of Algorithm 1 with no debiasing or simple debiasing already demonstrates improvements over existing methods.
For example, for linear regression or logistic regression with Lasso regularization as model fitting, we use either multiplicative bias correction or relaxed Lasso correction for debiasing.
Further descriptions can be found in A.2.
We illustrate the application of the estimators Err X .dir and Err X .dec to specific settings, including linear models (OLS and Lasso) and logistic regression for classification.
The summary of result comparisons under covariate shift can be found in Table 1.
Since the metric used is the average signed difference between error estimates and true test error, we compare estimators based on bias.
We also include an illustration example in Figure 1 that plots visualize the comparison under different magnitudes of covariate shifts.
Under covariate shift, our two proposed estimators of Err X .dir and Err X .dec perform much better than CV across all simulation settings. In the illustration example presented in Figure 1, we compare our method of Err X with CV in a simulated example. Consider a linear model y i
= x Ti θ + ϵ i , where ϵ i are i.i.d N (0 , σ 2 ) and x i ∈ R p for p = 50 features. Suppose that we have a training data set of 100 observations and an unlabeled test data set of 1000 samples.
We choose the training feature matrix X to be comprised of independent and identically distributed (i.i.d.)
standard normal variables, while the test feature matrix X test has entries of i.i.d N (0 , λ 2 ) random variables, where λ represents the amount of covariate shift from training data.
For each simulation setting, we choose λ such that the signal-to-noise ratio (SNR) is approximately 3 . We fit the training data with Lasso using the glmnet" package [5] and evaluates its performance on the test set. In Figure 1, we compare the performance of error estimates in terms of the signed proportional difference from true test error, which can be seen as a measure of bias. Proportional difference is the ratio of the difference between error estimate and true test error divided by true test error.
For example, the signed proportional difference for an estimator ˆ e and true test error e is (ˆ e − e ) /e . We can see that while CV predicts true test error well when the test data matrix is drawn from the same distribution as that of training data, its performance deteriorates significantly when there is covariate shift. On the other hand, the two estimators using our proposed method estimate true test error better across the spectrum of covariate shifts.
OLS
[Table] [Table]
[Table] [Table]
S i gn
[Table] d p r opo r t i [Table] a l d i ff e r e n
[Table]
f r o m t r
[Table] t
[Table]
t [Table]
[Figure] 4.2 Real data example We apply ErrX method on the K-class image classification task CIFAR10 [9] and compare with two other existing methods: Average confidence (ConfScore) ([8]), and Projection Norm (ProjNorm) ([18]). For test data involving distribution shift, we consider both the orginal version and some adapted version of the common corruptions dataset ([7]).
Since labels for images remain the same after corruption, they may not satisfy the exact covariate shift assumption.
We provided an adapted common corruptions dataset via relabeling to ensure only covariate shift in test data.
From Figures 2 and 3, it can be seen that the method of ErrX has similar performance in terms of correlation with actual test errors as compared to the other two methods that specialize in classification tasks.
While the method of ErrX is computationally slower when calculating estimates using parametric bootstrap, it saves computation time by avoiding the task of calibration.
Among the three predictions of test error, only ErrX is a direct estimate while the other two need calibration to match
ProjNorm/ConfScore to final error predictions, where calibration parameters may differ depending on training data, neural network architecture, etc.
A Appendix A.1 ErrX method via decomposition
Since our new target Err Q X is a function of features in the training set it has connections with in-sample error , which is the target of estimation for traditional covariance-penalty based methods.
Recall that in-sample error Err in ( X ) is the error for a fresh sample with the same covariates as training data.
[Equation]
Notice that we can combine equations (3) and (4) to obtain the following decomposition similarly as in [2].
[Equation]
The value of this decomposition in (5) is that it offers an alternative way to estimate our target of
Err Q X , X test .
For estimation of in-sample error Err in , we can use standard Mallows C p for linear models and bootstrap estimation for covariance penalty otherwise.
The details of general estimation method for Err Q X , X test can be found in Algorithm 2. [Equation]
[Equation] = test l
[Equation]
b
[Equation]
X
[Equation] [Equation]
It remains to discuss possible ways to obtain suitable estimates of in-sample error (cid:100) Err in ( X ) in step 2 of Algorithm 2.
Notice that for ordinary least squares (OLS) with linear model, the well-known
Mallows C p [10] is an unbiased estimate of in-sample error Err in ( X ) by
[Equation]
For Lasso penalty in a linear model, we can replace the dimension of covariates p with the number of nonzero coefficient estimates for estimating in-sample error via a degree of freedom argument ([14]).
When dropping the linear model assumption, [16] and [4] give a more general form of covariance penalty identity for in-sample error,
[Equation]
This identity allows us to estimate in-sample error by parametric bootstrap as follows in Algorithm[-
-]
Input training data ( X Y ) , number bootstrap samples B , fitting algorithm ˆ θ ( · , parametric
Y ( b ) ∼ P ˆ θ ( X , Y ) Y | X = X . b θ b ) = θ do 6:
Compute sample averages ¯ Y i = 1 B (cid:80)
Bb =1 Y ( b ) i and ¯ f i = 1 B (cid:80)
Bb =1 ˆ f ( x i , ˆ θ ( b ) ) .
Compute (cid:100)
Cov i =
1 B
B (cid:88) b
= (cid:16) Y ( b )
i − ¯ Y
i (cid:17) (cid:16) ˆ f ( x i , ˆ θ ( b ) ) − ¯ f i (cid:17) 8: Compute (cid:100)
Err in ( X ) = 1 n n (cid:88) i = (cid:16) Y i − ˆ f ( X i , ˆ θ ( X , Y )) (cid:17) 2 + 2 n n (cid:88) i =
(cid:100)
Cov i
Output : Err X )
The covariance penalty based method of estimating in-sample can be generalized to a wider class of loss functions beyond squared error [4].
In the case of logistic regression with counting error, error function q ( u ) = min ( u, 1 − u ) .
We have the following identity,
[Equation]
[Equation] E [Equation] where l is counting error function, λ i = − ∂q∂u ( ˆ f ( X i , ˆ θ ( X , Y )))
That is, λ i = − 1 /
if ˆ f ( X i , ˆ θ ( X , Y ))
=
and λ i = 1 / 2 otherwise.
We shift all λ i by 1 / 2 to get ˆ
λ i
=
λ i + 1 /
=
ˆ f
( X i , ˆ θ ( X , Y )) without changing the covariance penalty term.
Therefore, to estimate in-sample error for logistic regression with being counting error loss, we only need to replace the last step in algorithm 3 with
[Equation]
A.2 Bias correction
Since our method of ErrX relies on having an unbiased initial model, we need bias correction steps in order to achieve better prediction accuracy especially when regularization is applied.
Since the Lasso estimator is biased, the parametric bootstrap step will carry on the bias, requiring corrections to the output estimators Err X .dir and Err X .dec .
Here we propose two existing methods of bias correction for predicting test error of Lasso as an example, i.e. (1) Multiplicative bootstrap correction and (2)
Relaxed Lasso.
The details are as follows.
Multiplicative bootstrap bias correction:
Multiply estimators Err X .dir and Err X .dec by a constant shrinking factor c .
Let ˆ θ ( X , Y ) and ˆ θ ( X , Y ( b ) ) denote the fitted parameters from initial model and bootstrap samples.
Then we propose to choose,
[Equation]
The intuition is that we adjust for the scaling factor between the true parameter θ and learned parameter ˆ θ by that between ˆ θ and refitted parameter after bootstrap.
Relaxed Lasso correction:
Use relaxed Lasso fit on the initial training data to form parametric bootstrap samples.
The idea is that we want to reduce the bias between true parameter θ and that used in generating parametric bootstrap samples.
A.3 Additional simulation results Linear model
First consider the setting of linear model with homoscedastic Gaussian errors, y i =
x Ti θ + ϵ
i , where ϵ i i.i.d ∼ N (0 , σ 2 ) .
Given a particular choice of loss function such as square error loss, the general method of ErrX can be applied.
[Figure]
Err X estimates ( Err X .dir and
Err X .dec ), and cross validation. For each setting, a pairwise comparison is included in the second row corresponding to the proportion of deviation from true test error for each of the three estimates.
[Figure]
For simulation, we consider the setting with n = 100 observations of p = 10 features for training, n test = 1000 observations of unlabeled test data, and coefficient vector of 4 nonzero entries with equal strength of 2 .
The training feature matrix consists of i.i.d entries drawn from N (0 , 1) .
We
Correct model Misspecification errtest errx.dir errx.dec errcv errtest errx.dir errx.dec errcv
E s t i m a t e s
Correct model
Misspecification errx.dir errx.dec errcv errx.dir errx.dec errcv −0.
−0.
P a i r w
i s e
C o m pa r i s on
[Figure]
Correct model
Misspecification errtest errx.dir errx.dec errcv errtest errx.dir errx.dec errcv
E s t i m a t e s
Correct model
Misspecification errx.dir errx.dec errcv errx.dir errx.dec errcv −
P a i r w
i s e
C o m pa r i s on
[Figure]
consider two situations, one without covariate shift and one with covariate shift.
The detailed setup is as follows.
No
covariate shift:
The feature matrix of test data are comprised of i.i.d entries drawn from N (0 , 1) . We choose σ = 3 so that signal-to-noise ratio (snr) is approximately 2 . For sensitivity to model misspecification, we include a quadratic transformation to
1 / 3 of the feature coordinates. Covariate shift: The feature matrix of test data are comprised of i.i.d entries drawn from N (2 , 2) . We choose σ = 5 so that signal-to-noise ratio (snr) is approximately 2 . For sensitivity to model misspecification, we include a quadratic transformation to 1 / 5 of the feature coordinates.
Notice that we choose different transformations such that the change in test error with or without taking the transformation into account is approximately 50 − 100% .
From Figure 4 and 5, it can be seen that while both cross validation and Err X estimation predicts true test error well when there is no covariate shift (left plot), our method of Err X estimation performs better than CV in the presence of covariate shift (right plot).
Under model misspecification, while CV demonstrates slightly more robustness without covariate shift, the bias in CV error estimation in the presence of covariate shift outweighs the robustness advantage.
An additional high dimensional example ( p = 50 ) with linear regression fit by Lasso regularization is provided in Figure[-
-]
Generalized Linear Model (GLM)
Our method of estimating Err X can similarly be applied to other nonlinear generalized linear models (GLM).
For Bernoulli observations as an example, we can replace square error loss with a suitable loss for binary classification such as counting error or binomial deviance, and use logistic regression as the fitting algorithm.
It is worth noting that as part of the procedure to produce the estimator Err X .dec , we need to estimate in-sample error, which can be obtained with general covariance penalties [4].
For simulation of Err X estimation in nonlinear GLM, we consider a sparse logistic model
[Equation]
with n = 200 observations and two cases for the number of features: a low dimensional setting p = 10 , and a higher dimensional setting p = 50 .
The training feature matrix consists of i.i.d entries drawn from N (0 , 1) .
We are interested in the comparison of different error estimates using counting error, both with and without covariate shift.
For covariate shift, we draw i.i.d test data from N (3 , 1) and subsample training data so that the training labels are imbalanced with ratio of 3 .
We chose the sparsity and signal strength so that signal-to-noise ratio is approximately 3 .
Similarly as in the case of linear regression, we need to apply bias correction to the Lasso parameter estimates.
Here we use relaxed Lasso correction for bootstrap in both Err X estimation and in estimating in-sample error
[Figure]
pairwise comparison is included in the second row corresponding to the proportion of deviation from true test error for each of the three estimates.
[Figure]
The simulation results for p = 10 and p = 50 are given in Figure 7 and Figure 8, respectively.
Similarly as in OLS and linear regression with Lasso penalty, the two proposed estimates Err X .dir and Err X .dec
recovers true test error better than CV in the presence of covariate shift.
It has to be acknowledged that when there is model misspecification along covariate shift, none of the error estimates resembles true test error.
However, the model misspecification that we introduced is an artificial and drastic one, which makes about 1 / 5 of the covariates in the linear model quadratic instead.
Good performance under model misspecification is not a reasonable expectation, and the setting is included only as a caution for application under model misspecification.
A.4 Additional real data example
We analyze a public data set on yearly state crime rates from 1977 to 2014.
The data set containsdemographic variables as predictors for the outcome of violent crime rate, with a total number of 1887 entries.
We first split the data set into two parts for training and testing.
We fit a linear model with Lasso penalty on the training set and apply different methods for error estimations, including our proposed estimators Err X .dir and Err X .dec , as well as cross-validation.
We then compare different error estimates with true test error evaluated on the test set outcomes.
We consider three different scenarios for splitting as follows and summarize the mean squared difference between error estimates and true test error in Table 2.
Random half splits : Data is randomly assigned to training and testing, regardless of the state and year.
Random half splits by states : States are randomly assigned to training and testing.
Data entries belonging to the same state are kept in the same fold in cross-validation.
Two-means clustering by states : Two-means clustering is applied on centroid of all states to split states into training and testing.
Data entries belonging to the same state are kept in the same fold in cross-validation.
[Table]
In the first case of random half splits, we expect no distribution shift, where all error estimates considered are close to the actual test error.
In the second case of random half splits by states, we expect some covariate shift as well as possible distribution shift due to different relations between demographic outcome perform slightly better than cross-validation.
The recovery of true test error is not perfect due to potential violation of the assumption of no conditional distribution shift.
In the third case of two-means clustering by states, we try to maximize covariate shift between training and testing.
Estimators
Err X .dir and Err X .dec perform much better than cross-validation despite potential conditional distribution shift.
We also analyze the crime rate data by fixing a training set consisting of data from a few states in the west including California, Washington, Nevada, New Mexico, Arizona, and Texas.
We then compare different error estimates and actual test error by traversing over the remaining test states in Figure 9.
The estimates of Err X .dir and Err X .dec perform strictly better than cross-validation in every test case, especially in Nebraska, Iowa, Oklahoma, New York, District of Columbia, Pennsylvania,
Missouri, Florida, and Utah, possibly due to smaller shift in conditional distribution of crime rate given predictor variables.
[Figure] [Figure]