O N S TATISTICAL B IAS I N A CTIVE L EARNING : H OW AND W HEN TO F IX I T
Anonymous authors
Paper under double-blind review
A BSTRACT
Active learning is a powerful tool when labelling data is expensive, but it introduces a bias because the training data no longer follows the population distribution.
We formalize this bias and investigate the situations in which it can be harmful and sometimes even helpful.
We further introduce novel corrective weights to remove bias when doing so is beneﬁcial.
Through this, our work not only provides a useful mechanism that can improve the active learning approach, but also an explanation of the empirical successes of various existing approaches which ignore this bias.
In particular, we show that this bias can be actively helpful when training overparameterized models—like neural networks—with relatively little data.
I NTRODUCTION
In modern machine learning, unlabelled data can be plentiful while labelling requires scarce resources and expert attention, for example in medical imaging or scientiﬁc experimentation.
A promising solution to this is active learning—picking the most informative datapoints to label that will hopefully let the model be trained in the most sample-efﬁcient way possible (Atlas et al., 1990; Settles, 2010).
Active learning has a complication.
By picking the most informative labels, the acquired dataset is not drawn from the population distribution.
This sampling bias, noted by e.g., MacKay (1992) and
Dasgupta & Hsu (2008), is worrying: key results in machine learning depend on the training data being identically and independently distributed (i.i.d.)
samples from the population distribution.
For example, we train neural networks by minimizing a Monte Carlo estimator of the population risk.
If training data are actively sampled, that estimator is biased and we optimize the wrong objective.
The possibility of bias in active learning has been considered by e.g., Beygelzimer et al.
Chu et al.
; Ganti & Gray (2012), but the full problem is not well understood.
Methods that remove active learning bias have been restricted to special cases, so it has been impossible to even establish whether removing active learning bias is helpful or harmful in typical situations.
To this end, we show how to remove the bias introduced by active learning with minimal changes to existing active learning methods.
As a stepping stone, we build a Plain Unbiased Risk Estimator, ˜
R PURE , which applies a corrective weighting to actively sampled datapoints in pool-based active learning.
Our Smoothed Unbiased Risk Estimator, ˜ R SURE , builds on this and has lower variance and additional desirable ﬁnite-sample properties.
We prove that both estimators are unbiased and consistent for arbitrary functions, and characterize the variance of the estimators.
Having shown how to remove the bias introduced by active learning, we show that this can sometimes lead to signiﬁcant improvements.
The beneﬁts are most pronounced in underparameterized models where each datapoint affects the learned function globally.
For example, in linear regression adopting our weighting allows better estimates of the parameters with less data.
But in cases where the model is overparameterized and datapoints mostly affect the learned function locally—like neural networks—we ﬁnd that correcting active learning bias can be ineffective or even harmful.
This explains the existing success of active learning with neural networks, which almost entirely ignores the bias from active sampling (e.g., (Gal et al., 2017b;
Shen et al., 2018) and others in Appendix D).
Risk estimator bias is especially important for model evaluation.
Perhaps because of this, active learning has historically been used only for training.
However, test labels cost as much as training labels, despite active learning works often relying on large test and validation sets (Haut et al., 2018;
Sinha et al., 2019).
Our estimators allow active sampling for model selection also and thus provide a means of avoiding this problem by labeling test data in an efﬁcient manner without introducing bias.
Our main contributions are:
We offer a formalization of the problem of statistical bias in active learning.
We introduce active learning risk estimators, ˜ R PURE and ˜ R PURE , and prove both are unbiased, consistent, and with variance that can be less than the naive (biased) estimator. 3.
Using these, we show that active learning bias can hurt in underparameterized cases like linear regression but help in overparameterized cases like neural networks and explain why.
B IAS IN A CTIVE L EARNING
We begin by characterizing the bias introduced by active learning.
In supervised learning, generally, we aim to ﬁnd a decision rule f θ corresponding to inputs, x , and outputs, y, drawn from a population data distribution p data ( x , y ) which, given a loss function L ( y , f θ ( x )) , minimizes the population risk:
[Equation]
The population risk cannot be found exactly, so instead we consider the empirical distribution for some dataset of N points drawn from the population.
This gives the empirical risk: an unbiased and consistent estimator of r when the data are drawn i.i.d from p data and are independent of θ ,
[Equation]
In pool-based active learning (Lewis & Gale, 1994; Settles, 2010), we begin with a large unlabelled dataset, known as the pool dataset D pool ≡ { x n | 1 ≤ n ≤ N } , and sequentially pick the most useful points for which to acquire labels.
The lack of most labels means we cannot evaluate ˆ R directly, so we use the sub-sample empirical risk evaluated using the M actively sampled labelled points:
[Equation]
Though almost all active learning research uses this estimator (see Appendix D), it is not an unbiased estimator of either ˆ R or r when the M points are actively sampled.
Under active—i.e. non–uniform— sampling the M datapoints are not drawn from the population distribution, resulting in a bias which we formally characterize in §5.
See Appendix A for a more general overview of active learning.
Note an important distinction between what we will call “statistical bias” and “overﬁtting bias.”
The bias from active learning above is a statistical bias in the sense that using ˜ R biases our estimation of r , regardless of θ .
As such, optimizing θ with respect to ˜ R induces bias into our optimization of θ .
In turn, this breaks any consistency guarantees for our learning process: if we keep M/N ﬁxed, take
M → ∞ , and optimize for θ , we no longer get the optimal θ that minimizes r .
Almost all work on active learning for neural networks currently ignores the issue of statistical bias.
However, even without this statistical bias, indeed even if we use ˆ R directly, the training process itself also creates an overﬁtting bias: evaluating the risk using training data induces a dependency between the data and θ .
This is why we usually evaluate the risk on held-out test data when doing model selection.
Dealing with overﬁtting bias is beyond the scope of our work as this would equate to solving the problem of generalization.
The small amount of prior work which does consider statistical bias in active learning entirely ignores this overﬁtting bias without commenting on it.
In §3-7, we focus on statistical bias in active learning, so that we can produce estimators that are valid and consistent and let us optimize the intended objective, not so they can miraculously close the train–test gap.
From a more formal perspective, our results all assume that θ is chosen independently of the training data; an assumption that is almost always (implicitly) made in the literature.
This ensures our estimators form valid objectives, but also has important implications that are typically overlooked.
We return to this in §8, examining the interaction between statistical and overﬁtting bias.
U NBIASED A CTIVE L EARNING : ˜ R PURE AND ˜ R SURE
We now show how to unbiasedly estimate the risk in the form of a weighted expectation over actively sampled data points.
We denote the set of actively sampled points D train ≡ { ( x m , y m ) | 1 ≤ m ≤ M } , where ∀ m : x m ∈ D pool .
We begin by building a “plain” unbiased risk estimator, ˜ R PURE , as a stepping stone—its construction is quite natural in that each term is an individually unbiased estimator of the risk.
We then use it to construct a “smoothed” unbiased risk estimator, ˜ R SURE , which is an unbiased and consistent estimator of the population risk just like ˜ R PURE , but which reweights individual terms to produce lower variance and resolve some pathologies of the ﬁrst approach.
Both estimators are easy to implement and have trivial compute/memory requirements.
3. ˜ R PURE : P LAIN U NBIASED R ISK E STIMATOR
For our estimators, we introduce an active sampling proposal distribution over indices rather than the more typical distribution over datapoints.
This simpliﬁes our proofs, but the two are algorithmically equivalent for pool-based active learning because of the one–to–one relationship between datapoints and indices.
We deﬁne the probability mass for each index being the next to be sampled, once D train contains M − 1 points, as q ( i M ; i 1: M − 1 , D pool ) .
Because we are learning actively, the proposal distribution depends on the indices sampled so far ( i 1: M − 1 ) and the available data ( D pool ).
The only requirement on this proposal distribution for the proofs in this section is that it must place non-zero probability on all of the training data: anything else necessarily introduces bias.
Considerations for the acquisition proposal distribution are discussed further in §4 We ﬁrst present the estimator before proving the main results:
[Equation]
where a m ≡ w m L
i m + 1 N (cid:88)
m − 1 t =1 L i t , where the loss at a point L i m ≡ L ( y i m , f θ ( x i m )) , the weights w m ≡ 1 /Nq
( i m ; i 1: m − 1 , D pool ) and i m ∼ q ( i m ; i 1: m − 1 , D pool ) .
For practical implementation, ˜ R PURE can further be written in the following more computationally friendly form that avoids a double summation:
[Equation]
but we focus on the ﬁrst form for our analysis because a m has some beneﬁcial properties not shared by the above that we can exploit.
In particular, in Appendix B.1 we prove that:
Lemma 1.
The individual terms a m of ˜ R PURE are unbiased estimators of the risk: E [ a m ]
= r .
The motivation for the construction of a m directly originates from constructing an estimator where
Lemma 1 holds while only using the i 1 , . .
, i m datapoints, taking care with the fact that each new proposal distribution does not have support over points that have already been acquired.
Except for trivial solutions, it is unique in this regard; naive importance sampling (i.e. 1 M (cid:80) Mm =
w m L i m ) does not lead to an unbiased, or even consistent, estimator.
However, the overall estimator ˜ R PURE is not the only unbiased estimator of the risk, as we discuss in §3.2.
We can now characterize the behaviour of ˜ R PURE as follows (see Appendix B.2 for proof)
Theorem 1.
˜ R PURE as deﬁned above has the properties:
[Equation]
where
[Equation]
Remark 1.
The ﬁrst term of (2) is the variance of the loss on the whole pool, while the second term accounts for the interaction between sampled points and D pool .
As such, the second term, which is
O ( N ) times larger, will always dominate in practice.
Armed with Theorem 1, we can prove the consistency of ˜ R PURE under standard assumptions:
˜ R PURE converges in expectation (i.e. its mean squared error tends to zero) as M → ∞ under the assumptions that N > M , L ( y , f θ ( x )) is integrable, and q ( i m ; i 1: m − 1 , D pool ) is a valid proposal in the sense that it puts non-zero mass on each unlabelled datapoint.
Formally, as proved in Appendix B.3,
Theorem 2.
Let α = N/M and assume that α > 1 .
If E (cid:2) L ( y , f θ ( x )) 2 (cid:3) < ∞ and ∃
[Equation]
>
[Equation] [Equation]
then ˜ R PURE converges in its L 2 norm to r as M → ∞ , i.e., lim M →∞ E (cid:20)(cid:16)
˜
R PURE − r (cid:17) 2 (cid:21) = 0 .
3. ˜ R SURE : S MOOTHED U NBIASED R ISK E STIMATOR
˜ R PURE is natural in that each term is an unbiased estimator of r .
However, this creates surprising behaviour given the sequential structure of active learning.
For example, with a uniform proposal distribution—equivalent to not actively learning—points sampled earlier are more highly weighted than later ones and ˜ R PURE (cid:54) =
˜ R .
Speciﬁcally, a uniform proposal, q ( i m ; i 1: m − 1 , D pool )
=
1 N − m +1 , gives a weight on each sampled point of 1 + M − 2 m +1 N (cid:54) = 1 .
Similarly, as M → N (such that we use the full pool) the weights also fail to become uniform: setting M = N gives a weight for each point of 1 + M − 2 m +1 N (cid:54)
= 1 .
˜ R SURE ﬁxes this.
We ﬁrst quote the estimator before proving key results:
[Equation]
This estimator ensures that the expected value of the weight, v m , does not depend on the position it was sampled in, but only on the probability with which it was sampled.
That is, E [ v m ] = 1 for all m , M , N , and q ( i m ; i 1: m − 1 ; D pool ) .
As a consequence the variance is generally lower.
Moreover, we resolve the ﬁnite-sample behaviour shown by ˜ R PURE .
The weights become more even as M increases for a given N , and when M = N , each v m = 1 such that ˜ R SURE =
˜ R
=
ˆ R .
Additionally, if the proposal is uniform, all weights are always exactly 1 such that ˜ R SURE =
˜ R .
To derive
˜ R SURE note that each a m estimates r without bias so for any normalized linear combination:
[Equation]
provided that the c m are constant with respect to the data and sampled indices (they can depend on
M , N , and m ).
In Appendix B.4 we show that the choice of :
[Equation]
produces the v m from (3) and in turn that these v m have the desired property E [ v m ] = 1 , ∀ m ∈ { 1 , . .
, M } .
We note also that (cid:80)
Mm =1 c m =
M , such that ˜ R SURE = 1 M (cid:80)
Mm =
c m a m .
We further characterise the variance and unbiasedness of ˜ R SURE as follows (see Appendix B.5 for proof)
Theorem 3.
˜ R SURE as deﬁned above has the following properties:
[Equation]
where
[Equation]
The variance of ˜ R SURE is typically lower than that of ˜ R PURE .
Note that (2) and (4) differ only in the scaling term c 2 m .
By deﬁnition, c m increases with m and 1 M (cid:80) Mm =
c m = 1 .
Meanwhile, the variance of w m L i m will typically increase with m because the average value of w m is ( N − m + 1) /N .
As such, the c m generally upweight the lower variance latter terms and downweight the higher variance earlier terms, such that the variance of ˜ R SURE is typically less than ˜ R PURE .
This can also be seen by noting that the c m were introduced to ensure that the expected values of the overall weighting factors, v m (as per (3)), are identical.
As such, this weighting reduces the variance by removing the tendency of ˜ R PURE to overemphasize the earlier samples.
We ﬁnish by conﬁrming that ˜ R SURE is a consistent estimator as M → ∞ (proof in Appendix B.6):
Theorem 4.
Under the same assumptions as Theorem 2: lim M →∞ E (cid:104)(cid:0)
˜ R
SURE − r (cid:1) 2 (cid:105) = 0 .
F ROM A CTIVE L EARNING S CHEMES TO P ROPOSALS
So far, we have introduced two separable elements of the active learning scheme.
§3 introduced risk estimators ˜ R PURE and ˜ R SURE .
The acquisition proposal distribution— q ( i m | i 1: m − 1 , D pool ) —has remained quite general.
So long as the acquisition proposal puts non-zero mass on all the training data, ˜ R PURE and ˜ R SURE are unbiased and consistent as proven above.
This is in contrast to the naive risk estimator ˜ R , for which the choice of proposal distribution affects the bias of the estimator.
Using our scheme lets us disentangle the choice of risk estimator from the choice of acquisition proposal.
In §5, we will discuss how the properties of the acquisition proposal distribution can however affect the variance of the unbiased estimators, potentially resulting in lower variance than the naive biased estimator.
Then in §8 we will turn to a third element of active learning schemes: the way in which optimization introduces a bias separately from the choice of risk estimator and proposal distribution.
At that point, we will consider a ﬁnal interaction: where the acquisition proposal distribution depends on a model which has been optimized using one of the risk estimators under consideration.
It is easy to statisfy the requirement for non-zero mass everywhere, even for prior work which selects points deterministically (e.g., Bayesian Active Learning by Disagreement (BALD) (Houlsby et al., 2011) or a geometric heuristic like coreset construction (Sener & Savarese, 2018)).
Any scheme, like
BALD, that selects the points to acquire by performing an argmax operation can instead return a distribution that supports the whole space by substituting a softmax operation.
Alternatively, a distribution can be constructed analogous to epsilon-greedy exploration.
With probability 1 − (cid:15) we select the deterministically proposed datapoint from any arbitrary acquisition strategy, while with probability (cid:15) we uniformly select any datapoint.
In this way, any existing deterministic active learning scheme can be adapted in a way that allows unbiased risk estimation.
It is possible to use ˜ R SURE and ˜ R PURE with data collected using a proposal distribution that does not fully support the training data, though they will not fully correct the bias here.
Namely, if we have a set of points, I , that are ignored by the proposal (i.e. that are assigned zero mass), we can still use ˜
R SURE and ˜ R PURE in the same way
but they both introduce the same following bias:
[Equation]
Sometimes this bias will be small and may be acceptable if it enables a desired acquisition scheme, but in general one of the stochastic adaptations described above is likely to be preferable.
One can naturally also extend this result to cases where I varies at each iteration of the active learning (including deterministic acquisition strategies), for which we again have a non–zero bias.
U NDERSTANDING THE E FFECT OF ˜ R SURE AND ˜ R PURE ON V ARIANCE
In order to show that the variance of our unbiased estimators can be lower than that of the biased ˜ R , with a well-chosen acquisition function, we ﬁrst introduce an analogous result to Theorems 1 andfor ˜ R , the proof for which is given in Appendix B.7:
Theorem 5.
Let µ m :=
E [ L i m ] and µ m |
i, D :=
E [ L i m | i 1: m − 1 , D pool ] .
For ˜ R (deﬁned in (1) ):
[Equation]
M
[Equation]
r in general )
[Equation]
Examining this expression shows that the variance of ˜ R PURE can be either larger or smaller than ˜ R , depending on the the proposal distribution and dataset.
Consider the terms of (5): 1 and 2 are analagous to terms of the variance of ˜ R PURE in (2).
Recall that 1 is smaller than 2 by O ( N ) in any case, so set it aside.
2 is the expected conditional variance of L i m , while the ˜ R PURE analogue has at the weighted loss w i L i m .
An effective proposal distribution should select surprising/informative points—with above average losses—with high probability—low w m —resulting in anticorrelation between L i m and w m .
This means the expected conditional variance of L i m should be greater than that of w i L i m for a well-designed acquisition strategy.
Now consider the terms only in (5) with no analogue to the previous equations.
3 is always positive, so it contributes to ˜ R having higher variance, and comes from the variability in the bias itself in index sampling.
4 expresses the correlation between the bias and the value of the pool estimate itself.
It can be positive or negative, but is typically small and decreases as N grows because the variability of ˆ R scales as O (1 /N ) .
5 reﬂects correlations between the losses at different iterations, an effect that our estimators have carefully eliminated.
It is somewhat harder to quantify: it can be positive or negative depending on the acquisition function and dataset.
For some acquistition strategies, it is conceivable that this term may not converge to zero even in the limit of large N .
Nonetheless, we note that when this term is large, such that it might affect which estimator has higher variance, the correlation between losses in different acquired points would lead to high bias in ˜ R .
In contrast, we prove in Appendix B.8 that under an optimal proposal distribution both ˜ R PURE and ˜
R SURE become exact estimators of the empirical risk for any number of samples M —such that they will inevitably have lower variance than ˜ R in this case.
A similar result holds when we are estimating gradients of the loss, though note that the optimal proposal is different in the two cases.
Theorem 6.
Given a non-negative loss, the optimal proposal distribution
[Equation]
yields estimators exactly equal to the pool risk, that is ˜ R PURE =
˜ R SURE =
ˆ R almost surely ∀ M .
In practice, it is impossible to sample using the optimal proposal distribution.
However, we make this point in order to prove that adopting our unbiased estimator is certainly capable of reducing variance relative to standard practice if appropriate acquisition strategies are used.
It also provides interesting insights into what makes a good acquisition strategy from the perspective of the risk estimation itself.
R ELATED W ORK
Pool-based active learning (Lewis & Gale, 1994) is useful in cases where input data are prevalent but labeling them is expensive (Atlas et al., 1990; Settles, 2010).
The bias from selective sampling was noted by MacKay (1992), but dismissed from a Bayesian perspective based on the likelihood principle.
Others have noted that the likelihood principle remains controversial (Rainforth, 2017), and in this case would assume a well-speciﬁed model.
Moreover, from a discriminative learning perspective this bias is uncontentiously problematic.
Lowell et al.
(2019) observe that active learning algorithms and datasets become coupled by active sampling and that datasets often outlive algorithms.
Despite the potential for bias, in deep learning it is generally ignored.
As an informal survey, we examined the 15 most-cited peer-reviewed papers citing Gal et al.
(2017b), which introduced active learning to image data using neural networks.
Of these, only two mentioned estimator bias but did not address it while the rest either ignored or were unaware of this problem (see Appendix D).
Nguyen et al.
(2018) propose active testing but ignore bias introduced by active sampling.
There have been some attempts to address active learning bias, but these have generally required fundamental changes to the active learning approach and only apply to particular setups.
Beygelzimer et al.
(2009), Chu et al.
(2011), and (Cortes et al., 2019) apply importance-sampling corrections (Sugiyama, 2006; Bach, 2006) to online active learning.
Unlike pool-based active learning, this involves deciding whether or not to sample a new point as it arrives from an inﬁnite distribution.
This makes importance-sampling estimators much easier to develop, but as Settles (2010) notes, “ the pool-based scenario appears to be much more common among application papers ”.
Ganti &
Gray (2012) address unbiased active learning in a pool-based setting by sampling from the pool with replacement .
This effectively converts pool-based learning into a stationary online learning setting, although it overweights data that happens to be sampled early.
Sampling with replacement is unwanted in active learning because it requires retraining the model on duplicate data which is either impossible or wasteful depending on details of the setting.
Moreover, they only prove the consistency of their estimator under very strong assumptions (well-speciﬁed linear models with noiseless labels and a mean-squared-error loss).
Imberg et al.
(2020) consider optimal proposal distributions in an importance-sampling setting.
Outside the context of active learning, Byrd & Lipton (2019) question the value of importance-weighting for deep learning, which aligns with our ﬁndings below.
A PPLYING ˜ R SURE AND ˜ R PURE
[Figure]
would get
We ﬁrst verify that ˜ R SURE and ˜ R PURE remove the bias introduced by active learning and examine the variance of the estimators.
We do this by taking a ﬁxed function whose parameters are independent of D train and estimating the risk using actively sampled points.
We note that this is equiv alent to the problem of estimating the of an already trained model in a sample-efﬁcient way given unlabelled test data.
We consider two settings: an inﬂexible model (linear regression) on toy but non-linear data and an overparameterized Bayesian neural ) on modiﬁed MNIST with unbalanced classes and noisy labels.
Linear regression.
For linear functions, removing active learning bias (ALB), i.e., the statistical bias introduced by active learning, is critical.
We illustrate this in Figure 1.
Actively sampled points overrepresent unusual parts of the distribution, so a model learned using the unweighed D train differs from the ideal function ﬁt to the whole of D pool .
Using our corrective weights more closely approximates the ideal line.
The full details of the population distribution and geometric acquisition proposal distribution are in Appendix C.1, where we also show results using an alternative epsilon-greedy proposal.
We inspect the ALB in Figure 2a by comparing the estimated risk (with squared error loss and a ﬁxed function) to the corresponding true population risk ˆ R .
While M < N , the unweighted ˜ R is biased (in practice we never have M = N as then actively learning is unnecessary).
˜ R PURE and ˜
R SURE are unbiased throughout.
However, they have high variance because the proposal is rather poor.
Shading represents the std. dev.
of the bias over 1000 different acquisition trajectories.
Bayesian Neural Network.
We actively classify MNIST and FashionMNIST images using a convolutional Bayesian neural network (BNN) with roughly 80,000 parameters.
In Figure 2b and 2c we show that ˜ R PURE and ˜ R SURE remove the ALB.
Here the variance of ˜ R PURE and ˜ R SURE is lower or similar to the biased estimator.
This is because the acquisition proposal distribution, a stochastic relaxation of the Bayesian Active Learning by Disagreement (BALD) objective (Houlsby et al., 2011), is effective (c.f. §5).
A full description of the dataset and procedure is provided in Appendix C.2.
Our modiﬁed MNIST dataset is unbalanced and has noisy labels, which makes the bias more distorting.
Overall, Figure 2 shows that our estimators remove the bias introduced by active learning, as expected, and can do so with reduced variance given an acquisition proposal distribution that puts a high probability mass on more informative/surprising high-expected-loss points.
[Figure] [Figure]
Next, we examine the overall effect of using the unbiased estimators to learn a model on downstream performance .
Intuitively, removing bias in training while also reducing the variance ought to improve the downstream task objective: test loss and accuracy.
To investigate this, we train models using ˜ R , ˜
R SURE , and ˜ R PURE with actively sampled data and measure the population risk of each model.
For linear regression (Figure 3a), the new estimators improve the test loss—even with small numbers of acquired points we have nearly optimal test loss (estimated with many samples).
However, for the BNN, there is a small but signiﬁcant negative impact on the full test dataset loss of training with ˜
R SURE or ˜ R PURE (Figure 3b) and a slightly larger negative impact on test accuracy (Figure 3c).
That is, we get a better model by training using a biased estimator with higher variance!
To validate this further, we consider models trained instead on FashionMNIST (Fig.
3d), on MNIST but with Monte Carlo dropout (MCDO) (Gal & Ghahramani, 2015) (Fig. 3e), and on a balanced version of the MNIST data (Fig. 3f).
In all cases we ﬁnd similar patterns, suggesting the effects are not overly sensitive to the setting.
Further experiments and ablations can be found in Appendix C.2.
A CTIVE L EARNING B IAS IN THE C ONTEXT OF O VERALL B IAS
In order to explain the ﬁnding that ˜ R SURE hurts training for the BNN, we return to the bias introduced by overﬁtting, allowing us to examine the effect of removing statistical bias in the context of overall bias.
Namely, we need to consider the fact that training would induce am overﬁtting bias (OFB) even if we had not used active learning.
If we optimize parameters θ according to ˆ R , then E [ ˆ R ( θ ∗ )] (cid:54) =
r , because the optimized parameters θ ∗ tend to explain training data better than unseen data.
Using ˜
R SURE , which removes statistical bias , we can isolate OFB in an active learning setting.
More formally, supposing we are optimizing any of the discussed risk estimators (which we will write using ˜ R ( · ) as a placeholder to stand for any of them)
we deﬁne the OFB as:
[Equation]
where [Equation] B OFB ( ˜ R ( · ) ) depends on the details of the optimization algorithm and the dataset.
Understanding it fully means understanding generalization in machine learning and is outside our scope.
We can still gain insight into the interaction of active learning bias (ALB) and OFB.
Consider the possible relationships between the magnitudes of ALB and OFB:
[ ALB >> OFB ] Removing ALB reduces overall bias and is most likely to occur when f θ is not very expressive such that there is little chance of overﬁtting.
[ ALB << OFB] Removing ALB is irrelevant as model has massively overﬁt regardless.
[ ALB ≈ OFB]
Here sign is critical.
If ALB and OFB have opposite signs and similar scale, they will [Figure] [Figure] F
[Figure]
̃ [Figure] ̃ [Figure]
̃
[Figure]
̃
[Figure]
70 M 0.
B O F B = r − ̃ R S U R E
( θ * )
[Figure]
: ̃
[Figure]
[Figure]
MNIST.
[Figure]
[Figure]
[Figure] tend to cancel each other out. Indeed, they usually have opposite signs. B OFB is usually positive: ﬁts the training data better than unseen data.
ALB is generally negative: we actively choose unusual/surprising/informative points which are harder to ﬁt than typical points.
Therefore, when signiﬁcant overﬁtting is possible, unless ALB is also large, removing ALB will have little effect and can even be harmful .
This hypothesis would explain the observations in §7 if we were to show that B OFB was small for linear regression but had a similar magnitude and opposite sign to
ALB for the BNN.
This is exactly what we show in Figure 4.
Speciﬁcally, we see that for linear regression, the B OFB for models trained with ˜ R , ˜ R PURE , and ˜ R SURE are all small (Figure 4a) when contrasted to the ALB shown in Figure 2a.
Here ALB >> OFB; removing ALB matters.
For BNNs we instead see that the OFB has opposite sign to the ALB but is either similar in scale for MNIST (Figures 2b and 4b), or the OFB is much larger than ALB for
Fashion MNIST (Figures 4c and 2c).
The two sources of bias thus (partially) cancel out.
Essentially, using active learning can be treated (quite instrumentally) as an ad hoc form of regularization.
This explains why removing ALB can hurt active learning with neural networks.
C ONCLUSIONS
Active learning is a powerful tool but raises potential problems with statistical bias.
We offer a corrective weighting which removes that bias with negligible compute/memory costs and few requirements—it suits standard pool–based active learning without replacement.
It requires a non– zero proposal distribution over all unlabelled points but existing acquisition functions can be easily transformed into sampling distributions.
Indeed, estimates of scores like mutual information are so noisy that many applications already have an implicit proposal distribution.
We show that removing active learning bias (ALB) can be helpful in some settings, like linear regression, where the model is not sufﬁciently complex to perfectly match the data, such that the exact loss function and input data distribution are essential in discriminating between different possible (imperfect) model ﬁts.
We also ﬁnd that removing ALB can be counter-productive for overparameterized models like neural networks, even if its removal also reduces the variance of the estimators, because here the ALB can help cancel out the bias originating from overﬁtting.
This leads to the interesting conclusion that active learning can be helpful not only as a mechanism to reduce variance as it was originally designed, but also because it introduces a bias that can be actively helpful by regularizing the model.
This helps explain why active learning with neural networks has shown success despite using a biased risk estimator.
We propose the following rules of thumb for deciding when to embrace or correct the bias, noting that we should always prefer ˜ R SURE to ˜ R PURE .
First, the more closely the acquisition proposal distribution approaches the optimal distribution (as per Theorem 6), the relatively better ˜ R SURE will be to ˜ R .
Second, the less overﬁtting we expect, the more likely it is that ˜ R SURE will be useful as it reduces the chance that the ALB will actually help.
Third, ˜ R SURE will tend to have more of an effect for highly imbalanced datasets, as the biased estimator will over-represent actively selected but unlikely datapoints.
Fourth, if the training data does not accurately represent the test data, using ˜ R SURE will likely be less important as the ALB will tend to be dwarfed by bias from the distribution shift.
Fifth, at test–time, where optimization and overﬁtting bias are no-longer an issue, there is little cost to using ˜
R SURE to evaluate a model and it will usually be beneﬁcial.
This ﬁnal application, of active learning for model evaluation, is a new research area that is opened up by our estimators.
A O VERVIEW OF A CTIVE L EARNING
Active learning selectively picks datapoints for which to acquire labels with the aim of more sampleefﬁcient learning.
For an excellent overview of the general active learning problem setting we refer the reader to Settles (2010).
Since that review was written, a number of signiﬁcant advances have further developed active learning.
Houlsby et al.
(2011) develop an efﬁcient way to estimate the mutual information between model parameters and the output distribution, which can be used for the Bayesian Active Learning by
Disagreement (BALD) score.
Active learning has been applied to deep learning, especially for vision e.g., Gal et al.
(2017b)
; Wang et al.
In neural networks speciﬁcally, empirical work has suggested that simple geometric core-set style approaches can outperform uncertainty-based acquisition functions (Sener & Savarese, 2018).
Rather than trying to improve the quality of active learning training itself, recent work has aimed to speed up acquisition from a computational perspective (Coleman et al., 2020) or to allow batch acquisition in order to parallelize labelling (Kirsch et al., 2019; Ash et al., 2020).
In some cases, work has also focused on applying active learning to speciﬁc settings with particular constraints (Krishnamurthy et al., 2017; Yan et al., 2018;
Sundin et al., 2019; Behpour et al., 2019; Shi & Yu, 2019; Hu et al., 2019).
B P ROOFS B. P ROOF OF L EMMA
Lemma 1.
The individual terms a m of ˜ R PURE are unbiased estimators of the risk: E [ a m ]
= r .
Proof.
We begin by applying the tower property of expectations:
[Equation]
By further noting that E i m [ w m L i m | D pool , i 1: m − 1 ] can be written out analytically as a sum over all the possible values of i m , while 1 N (cid:80) m − 1 t =1 L i t is deterministic given D pool and i 1: m − 1 we have:
[Equation]
But L n is now independent of the indices which have been sampled:
[Equation]
B. P ROOF OF THE U NBIASEDNESS OF ˜ R PURE : T HEOREM
Theorem 1.
˜ R PURE as deﬁned above has the properties:
[Equation]
where
[Equation]
Proof.
Having established Lemma
[Equation]
unbiasedness follows quickly from the linearity of expectations:
[Equation]
For the variance we instead have (starting with the deﬁnition of ˜ R PURE ):
[Equation]
explicitly writing out the sum of the squares and moving expectations past the summation:
[Equation]
− r )
[Equation]
a k −
[Equation]
and noting the form of the ﬁrst term:
[Equation]
We attack this line in two parts, beginning with the summands of the second term— E [( a m − r )
( a k − r )] —showing that they are necessarily zero.
Later, we will address the variance of a m .
Substitute in the deﬁnition of a m from equation (3.1):
[Equation]
We introduce the notation r remm =
r − 1 N (cid:80) m − 1 t =1 L i t to describe the remainder of the risk ascribable to the datapoint with index i m .
Then, by multiplying out the terms we have:
[Equation]
Now by the tower property:
[Equation]
k |
[Equation]
, i
[Equation] [Equation]
and noting that because k < m , w k L i k is deterministic given D pool and i 1: m − 1 :
[Equation]
which thus cancels with c .
The b and d cancel similarly:
[Equation]
Putting this together, we have that:
[Equation]
It follows that:
[Equation]
We now prove the theorem by examining this variance.
Note, similarly to (6), Var [ a m ] =
E (cid:104) ( a m − r ) 2 (cid:105) because a m are unbiased.
Then, applying the tower property of expectations:
[Equation]
and using the deﬁnition of the conditional variance of a m − r :
[Equation]
noting that r is not a random variable and that E i m [ a m |
i
[Equation] [Equation]
further noting that 1 N (cid:80) m − 1 t =1 L i t is deteriministic conditioned on i 1: m − 1 , and using linearity of expectations:
[Equation]
and noting again that r is not a random variable:
[Equation]
where x , y ∼ p data , as required.
B. P ROOF OF THE CONSISTENCY OF ˜ R PURE : T HEOREM
Theorem 2.
Let α = N/M and assume that α > 1 .
If E (cid:2) L ( y , f θ ( x )) 2 (cid:3) < ∞ and ∃ β >
[Equation] [Equation]
then ˜ R PURE converges in its L 2 norm to r as M → ∞ , i.e., lim M →∞ E (cid:20)(cid:16)
˜
R PURE − r (cid:17) 2 (cid:21) =
[Equation]
Proof.
Theorem 1 showed that ˜ R PURE is an unbiased estimator and so we ﬁrst note that its MSE is simply its variance, which we found in (2).
Substituting N = αM :
[Equation]
The ﬁrst term tends to zero as M → ∞ as our standard assumptions guarantee that guarantee that w m ≤ 1 /β and thus:
[Equation]
as our assumptions guarantee that 1 β 2 < ∞ , E i m (cid:2)
L 2 i m (cid:3) < ∞ and the empirical risk and losses are ﬁnite.
Given that each Var [ w m L i m | i 1: m − 1 , D pool ] is ﬁnite
[Equation]
it follows that:
[Equation]
and we thus have:
[Equation]
as desired.
B. D ERIVATION OF THE C ONSTANTS OF ˜ R SURE
We note from before that because of the unbiasedness of a m :
[Equation]
where
[Equation]
To construct our improved estimator ˜ R
SURE
[Equation]
we now need to ﬁnd the right constants c m , that in turn lead to overall weights v m (as per (3)) such that E [ v m ] = 1 .
We start by substituting in the deﬁnition of a m :
[Equation]
and then redistributing the L i t from later terms where they match m :
[Equation]
Note that though ˜ R SURE remains an unbiased estimator of the risk, each individual term v m L i m is not.
Now we require: E
[ v m ] = 1 ∀ m ∈ { 1 , ..., M } .
Remembering that w m = 1 / ( Nq ( i m ; i
[Equation]
m
[Equation]
1 , D pool )) :
[Equation]
Imposing that each E [ v m ] = 1 , we now have M equations for M unknowns c 1 , . . .
, c M , such that we can ﬁnd the required values of c m by solving the set of simultaneous equations:
[Equation]
We do this by induction.
First, consider E [ v m ] − E [ v m +1 ]
= 0 , for which can be rewritten:
[Equation]
and thus:
[Equation]
By further noting that the solution for m =
M is trivial:
[Equation]
we have by induction
[Equation]
Now we can exploit the fact that there is a canceling of most of the term in this sum.
The exceptions are − log
[Equation]
N − m + 1) , − log( N − m ) , log( N − M + 1) , and log( N − M ) .
We thus have:
[Equation]
which is our ﬁnal simple form for c m .
We can now check that this satisﬁes the required recursive relationship (noting it trivially gives the correct value for c M ) as per (8):
[Equation]
as required.
Similarly, we can straightforwardly show that this form of c m satisﬁes (7) by substitution.
We then ﬁnd the form of v m given this expression for c m .
Remember that:
[Equation]
We can rearrange (7) to:
[Equation]
from which it follows that:
[Equation]
Substituting in our expressions for c m and w m we thus have:
[Equation]
which is the form given in the original expression.
To ﬁnish our deﬁnition, we simply need to derive C :
[Equation]
where the result follows from a geometric sequence argument.
We thus see that our v m always sum to M , giving the quoted form for ˜ R SURE in the main paper.
B. P ROOF OF U NBIASEDNESS AND V ARIANCE FOR ˜ R SURE : T HEOREM
Theorem 3.
˜ R SURE as deﬁned above has the following properties:
[Equation]
where
[Equation]
Proof.
˜
R SURE is, by construction a linear combination of the weights a m .
By Lemma 1 each a m is an unbiased estimator of r .
So by the linearity of expectation, E (cid:104)
˜
R SURE (cid:105) =
r .
The variance requires a degree of care because although we previously showed that Var [ ˜ R PURE ] = 1 M (cid:80)
Mm =1 Var [ a m ] , we cannot directly apply this result to the weighted c m a m because a m are not independent.
Instead we employ an analogous proof to Theorem 1:
[Equation]
by the unbiasedness of ˜ R SURE :
[Equation]
Now considering the expectation inside term 2 we have:
[Equation]
Substituting this into 2 , taking out r and multiplying in c m
and c k we have (remembering (cid:80)
Mm =1 c m =
M from (9)):
[Equation]
where the second term does not have an immediate simple form.
Now consider 1 and again similarly to Theorem 1 we have:
[Equation]
This ﬁnal r 2 ( c m − 1) 2 terms now cancel exactly with 2 as per (11), such that substituting the above and (11) back into (10) yields the desired result.
B. P ROOF OF THE C ONSISTENCY OF ˜ R SURE : T HEOREM
Theorem 4.
Under the same assumptions as Theorem 2: lim M →∞
[Equation]
(cid:104)(cid:0)
[Equation]
Proof.
As before
[Equation]
since ˜ R SURE is unbiased the MSE is simply the variance so:
[Equation]
We already showed in the proof of Theorem 2 that E D pool ,
i
[Equation] [Equation]
is ﬁnite given our assumptions
[Equation]
while our assumptions further guarantee that Var [ L ( y , f θ ( x )] < ∞ .
As such, there must be some ﬁnite constant d such that
Var
[Equation]
x
[Equation]
and thus
[Equation]
and by substituting N = αM =
d
M
[Equation]
M (cid:88) m =
[Equation]
(cid:18) αM (
[Equation]
) ( αM
[Equation] [Equation]
) (cid:19)
[Equation]
<
d
M
[Equation]
M (cid:88) m
[Equation]
(cid:18) αM αM − M
[Equation]
(cid:19)
[Equation]
= dα 2 M (( α − 1) M + 1)which clearly tends to zero as M → ∞ (remembering that α > 1 ) and we are done.
B. P ROOF OF B IAS AND V ARIANCE OF S TANDARD A CTIVE L EARNING E STIMATOR : T HEOREM
Theorem 5.
Let µ m :=
E [ L i m ] and µ m |
i, D :=
E [ L i m | i 1: m − 1 , D pool ] .
For ˜ R (deﬁned in (1) ):
[Equation]
general )
Var
[Equation]
˜
[Equation]
(cid:123)(cid:122)
[Equation]
Proof.
The result of the bias follows immediately from deﬁnition of µ m and the linearity of expectations.
For the variance we start by applying standard rules for the variance of the sum of correlated variables to give
[Equation]
Focusing on the ﬁrst term we have through the tower property of expectations
[Equation]
now applying a bias–variance decomposition to the inner conditional expectation
[Equation]
and noting that E (cid:2) µ
[Equation]
i
[Equation]
We then further have
[Equation]
and the proof is completed but substituting the above into (13) and then (13) into (12)
[Equation]
B. P ROOF OF O PTIMAL P ROPOSAL D ISTRIBUTION : T HEOREM
Theorem 6.
Given a non-negative loss, the optimal proposal distribution
[Equation]
yields estimators exactly equal to the pool risk, that is ˜ R PURE =
˜ R SURE =
ˆ R almost surely ∀ M .
Proof.
We start by proving the result for the simpler case of ˜ R PURE before considering ˜ R SURE .
To make the notation simpler, we will introduce hypothetical indices i t for t > M , noting that their exact values will not change the proof provided that they are all distinct to each other and the real indices (i.e. that they are a possible realization of the active sampling process in the setting M = N ).
For ˜ R PURE , the proof follows straightforwardly from substituting the deﬁnition of the optimal proposal into the a m form of the estimator:
[Equation]
and because all possible indices are uniquely visited:
[Equation]
The proof proceeds identically for the case of ∇L i m because the gradient passes through the summations.
For ˜ R SURE , we similarly substitute the optimal proposal into the deﬁnition of the estimator:
[Equation]
pulling out the loss:
[Equation]
Remark 2.
The optimal proposal for estimating the gradient of the pool risk, ∇ φ ˆ R , with respect to some scalar φ is instead
[Equation]
Note that when taking gradients with respect to multiple variables, the optimal proposal for each will be different for each.
[Figure]
C E XPERIMENTAL D ETAILS C. L INEAR R EGRESSION
Our training dataset contains a small cluster of points near x = − 1 and two larger clusters at 0 ≤ x ≤ 0 .
5 and 1 ≤ x ≤ 1 . 5 , sampled proportionately to the ‘true’ data distribution.
The data distribution from which we select data in a Rao-Blackwellised manner has a probability density function over x equal to:
[Equation]
while the distribution over y is then induced by:
[Equation]
We set N = 101 , where there are 5 points in the small cluster and 96 points in each of the other two clusters, and consider 10 ≤ M ≤ 100 .
We actively sample points without replacement using a geometric heuristic that scores the quadratic distance to previously sampled points and then selects points based on a Boltzman distribution with β = 1 using the normalized scores.
Here, we also show in Figure 6 results that are collected using an epsilon-greedy acquisition proposal.
The results are aligned with those from the other acquisition distribution we consider in the main body of the paper.
This proposal selects the point that is has the highest total distance to all previously selected points with probability 0.9 and uniformly at random with probability (cid:15)
= 0 .
That is, the acquistion proposal is given by:
[Equation]
[Table]
[Equation]
otherwise where of course D train are the i 1: m − 1 elements of D pool .
For all graphs we use 1000 trajectories with different random seeds to calculate error bars.
Although, of course, each regression and scoring is deterministic, the acquistion distribution is stochastic.
Although the variance of the estimators can be inferred from Figure 2a, we also provide Figure 5a which displays the variance of the estimator directly.
C. B AYESIAN N EURAL N ETWORK
We train a Bayesian neural network using variational inference (Jordan et al., 1999).
In particular, we use the radial Bayesian neural network approximating distribution (Farquhar et al., 2020).
The details of the hyperparameters used for training are provided in Table 1.
[Figure]
(a) Bias (like Fig.
2a)
[Figure] [Figure]
[Table]
[Figure]
The unbalanced dataset is constructed by ﬁrst noising 10% of the training labels, which are assigned random labels, and then selecting a subset of the training dataset such that the numbers of examples of each class is proportional to the ratio (1., 0.5, 0.5, 0.2, 0.2, 0.2, 0.1, 0.1, 0.01, 0.01)—that is, there are 100 times as many zeros as nines in the unbalanced dataset.
(Figure 3f shows a version of this experiment which uses a balanced dataset instead, in order to make sure that any effects are not entirely caused by this design choice.)
In fact, we took only a quarter of this dataset in order to speed up acquisition (since each model must be evaluated many times on each of the candidate datapoints to estimate the mutual information).
1000 validation points were then removed from this pool to allow early stopping.
The remaining points were placed in D pool .
We then uniformly selectedpoints from D pool to place in D train .
Adding noise to the labels and using an unbalanced dataset is designed to mimic the difﬁcult situations that active learning systems are deployed on in practice, despite the relatively simple dataset.
However, we used a simple dataset for a number of reasons.
Active learning is very costly because it requires constant retraining, and accurately measuring the properties of estimators generally requires taking large numbers of samples.
The combination makes using more complicated datasets expensive.
In addition, because our work establishes a lower bound on architecture complexity for which correcting the active learning bias is no longer valuable, establishing that lower bound with MNIST is in fact a stronger result than showing a similar result with a more complex model.
The active learning loop then proceeds by:
training the neural network on D train using ˜ R ; 2.
scoring D pool ; 3.
sampling a point to be added to D train ; 4.
Every 3 points, we separately trained models on D train using ˜ R , ˜ R PURE , and ˜ R SURE and evaluate them.
This ensures that all of the estimators are on data collected under the same sampling distribution for fair comparison.
As a sense-check, in Figures 7a and 7b we show an alternate version in which the ﬁrst step trains with ˜ R SURE instead of ˜ R , and ﬁnd that this does not have a signiﬁcant effect on the results.
When we compute the bias of a ﬁxed neural network in Figure 2b, we train a single neural network on 1000 points.
We then sample evaluation points using the acquisition proposal distribution from the test dataset and evaluate the bias using those points.
In Figures 8a and 8b we review the graphs shown in Figures 3b and 3c, this time showing standard errors in order to make clear that the biased ˜ R estimator has better performance, while the earlier ﬁgures show that the performance is quite variable.
We considered a range of alternative proposal distributions.
In addition to the Boltzman distribution which we used, we considered a temperature range between 1,000 and 20,000 ﬁnding it had relatively little effect.
Higher temperatures correspond to more certainly picking the highest mutual information
[Figure] [Figure]
point, which approaches a deterministic proposal.
We found that because the mutual information had to be estimated, and was itself a random variable, different trajectories still picked very different sets of points.
However, for very high temperatures the estimators became higher variance, and for lower temperatures, the acquisition distribution became nearly uniform.
In Figure 9 we show the results of networks trained with a variety of temperatures other than the 10,000 ultimately used.
We also considered a proposal which was simply proportional to the scores, but found this was also too close to sampling uniformly for any of the costs or beneﬁts of active learning to be visible.
We considered Monte Carlo dropout as an alternative approximating distribution (Gal & Ghahramani, 2015)
(see Figures 3e and 10b).
We found that the mutual information estimates were compressed in a fairly narrow range, consistent with the observation by Osband et al.
(2018) that Monte Carlo dropout uncertainties do not necessarily converge unless the dropout probabilities are also optimized (Gal et al., 2017a).
While this might be good enough when only the relative score is needed in order to calculate the argmax, for our proposal distribution we would ideally prefer to have good absolute scores as well.
For this reason, we chose the richer approximate posterior distribution instead.
Last, we considered a different architecture, using a full-connected neural network with a single hidden layer with 50 units, also trained as a Radial BNN.
This showed higher variance in downstream performance, but was broadly similar to the convolutional architecture (see Figures 10d and 10e).
[Figure]
Reference
Application
Corrects Bias
Acknowledges Bias
Notes (Sener & Savarese, 2018)
(Shen et al., 2018) (cid:33) (Beluch et al., 2018)
(Haut et al., 2018) (cid:33) (Sinha et al., 2019) (Siddhant & Lipton, 2018)
[Table]
[Table]
(cid:33)
Discusses bias in D pool .
[Table] & Lee,
[Table]
) (cid:33)
Discusses bias in D pool .
[Table]
[Table]
) (cid:33)
D D EEP A CTIVE L EARNING I N P RACTICE
In Table 2, we show an informal survey of highly cited papers citing Gal et al.
(2017b), which introduced active learning to computer vision using deep convolutional neural networks.
Across a range of papers including theory papers as well as applications ranging from agriculture to molecular science only two papers acknowledged the bias introduced by actively sampling and none of the papers took steps to address it.
It is worth noting, though, that at least two papers motivated their use of active learning by observing that they expected their training data to already be unrepresentative of the population data and saw active learning as a way to address that bias.
This does not quite work, unless you explicitly assume that the actively chosen distribution is more like the population distribution, but is an interesting phenomenon to observe in practical applications of active learning.