Addressing Bias in Active Learning with Depth Uncertainty Networks... or Not
Abstract
Farquhar et al.
[2021] show that correcting for active learning bias with underparameterised models leads to improved downstream performance.
For overparameterised models such as NNs, however, correction leads either to decreased or unchanged performance.
They suggest that this is due to an “overﬁtting bias” which offsets the active learning bias.
We show that depth uncertainty networks operate in a low overﬁtting regime, much like underparameterised models.
They should therefore see an increase in performance with bias correction.
Surprisingly, they do not.
We propose that this negative result, as well as the results Farquhar et al.
[2021], can be explained via the lens of the bias-variance decomposition of generalisation error.
Introduction
Active learning improves data efﬁciency by identifying which examples maximise the expected gain in model performance [Cohn et al., 1995, Settles, 2010].
An acknowledged but relatively poorly understood problem with this approach, however, is that actively selecting informative training points introduces bias in inference, as the training data no longer follow the population distribution [MacKay, 1992].
Given that many statistical results rely on data points being identically and independently distributed (i.i.d.)
samples from the population distribution, applying standard estimators to actively sampled datasets results in optimising for the wrong objective [Farquhar et al., 2021].
Recently, Farquhar et al.
[2021] introduced an estimator, ˜ R LURE , that corrects for the bias induced by active sampling by re-weighting the contribution of individual data points to the estimated population risk.
Farquhar et al.
ﬁnd that, while underparameterised models such as linear regression beneﬁt from the elimination of active learning bias, the performance of overparameterised models such as neural networks is either unaffected or negatively impacted by the unbiased risk estimator.
This result is explained as active learning bias acting as a regulariser and helping to prevent ﬂexible models from overﬁtting on small datasets.
Depth uncertainty networks (DUNs) are a class of Bayesian neural network (BNN) in which a prior is placed over the depth of the network, allowing us to favour simpler (i.e., shallower) functions.
This enables DUNs to adapt their complexity to the varying size of the training dataset as additional labels are acquired during active learning.
Because of this, we posit that DUNs will suffer less from overﬁtting in the early stages of training than other BNNs.
This would imply that using the unbiased risk estimator of Farquhar et al.
[2021] to eliminate active sampling bias should improve the performance of DUNs.
Our main contributions are:
• Empirically validating the hypothesis that DUNs present less overﬁtting bias than other BNNs, speciﬁcally Monte Carlo dropout (MCDO), during active learning; I (Still) Can’t Believe It’s Not Better Workshop at NeurIPS 2021.
• Investigating the application of the ˜ R LURE estimator to eliminate active learning bias from DUNs.
Unexpectedly, despite DUNs having larger active learning bias than overﬁtting bias, we ﬁnd that ˜ R LURE does not result in improved downstream performance; • Providing hypotheses for why we do not observe an impact on DUNs’ downstream performance when applying the unbiased risk estimator.
Background 2.1 Bias in active learning
We use notation similar to Farquhar et al.
Given a loss function L ( y , f θ ( x )) , we aim to ﬁnd θ that minimises the population risk over p data ( y , x ) : E x , y ∼ p data [ L ( y , f θ ( x ))] .
In practice, we only have access to N samples from p data .
These yield the empirical risk, r = 1 N (cid:80) Nn =1 L ( y n , f θ ( x n )) .
r is an unbiased estimator of the population risk when the data are drawn i.i.d.
from p data .
If these samples have been used to train θ , our estimators become biased.
We refer to estimators biased by computing with the training data with capital letters (i.e., R ).
In the active learning setting, our model is optimised using a subset of M actively sampled data points:
[Equation]
The proposal distribution α ( i m ; i 1: m − 1 , D pool ) over the pool of unlabelled data D pool represents the probability of each index being sampled next, given that we have already acquired D train = { x i } 1: m − 1 .
As a result of active sampling, ˜ R is not an unbiased estimator of R .
Farquhar et al.
[2021] propose ˜ R LURE (“levelled unbiased risk estimator”), which removes active learning bias:
[Equation]
where L i m ≡ L ( y i m , f θ ( x i m )) .
Intuitively, the estimator works by re-weighting each example’s contribution to the total loss by its inverse acquisition probability, such that if unusual examples have particularly high acquisition probabilities, they contribute proportionally less to the total risk.
A detailed derivation of this estimator is given by Farquhar et al.
Surprisingly, Farquhar et al.
ﬁnd that using ˜ R LURE during training can negatively impact the performance of ﬂexible models.
They explain this by noting that the overﬁtting bias ( r − R ) typical in such models has the opposite sign to the bias induced by active learning.
That is, active learning encourages the selection of “difﬁcult” to predict data points, increasing the overall risk, whereas overﬁtting results in the total risk being underestimated.
Where overﬁtting is substantial, it is (partially) offset by active learning bias, such that correcting for active learning bias does not improve model performance.
2.2 Depth uncertainty networks for Active Learning
[Figure]
DUNs, depicted in ﬁgure 1, are a BNN variant in which the depth of the network d ∈ [1 , 2 ,...,D ] is treated as a random variable.
Model weights θ are kept deterministic, simplifying optimisation.
We place a categorical prior over depth p ( d ) =
Cat( d |{ β i } Di =0 ) , resulting in a categorical posterior which can be marginalised in closed form.
This procedure can be seen as Bayesian model averaging over an ensemble of subnetworks of increasing depth.
Thus, for
DUNs, the predictive distribution and Bayesian active learning by disagreement (BALD) acquisition objective [Houlsby et al., 2011] are tractable and cheap to compute.
See appendix A or [Antorán et al., 2020] for a detailed description.
In DUNs, predictions from different depths of the network correspond to different kinds of functions—shallow networks induce simple functions, while deeper networks induce more complex functions.
We choose a prior that favours shallow models, limiting the ﬂexibility of DUNs when few datapoints are observed.
As the actively acquired set increases, the model likelihood will start to dominate the prior, leading the posterior over depth to favour deeper models and thus more ﬂexible functions.
We show this in
Figure C.6.
We hypothesize that the capability to automatically adapt model ﬂexibility as more data are observed will reduce overﬁtting in the small data regimes typical of active learning.
Therefore
DUNs, unlike standard BNNs, may in fact beneﬁt from the use active learning bias corrective weights.
It is worth noting that automatically adapting model complexity to the observed data is also a promised feature of traditional BNNs.
However, unlike in DUNs where inferences are exact, weight space models require crude approximations which fall short in this regard [Ghosh et al., 2019].
Results
First, using the analysis of Farquhar et al.
[2021], we show that the weighted estimator removes active learning bias for DUNs.
We then show that DUNs overﬁt less than MCDO.
Finally, we examine how training with ˜ R LURE impacts downstream performance for DUNs.
Experiments are performed on nine UCI regression datasets [Hernández-Lobato and Adams, 2015]; results for the Boston, Concrete and Energy datasets are presented in the following sections, with results for the rest provided in appendix C. The experimental setup is described in appendix B. All experiments are repeatedtimes, with the mean and standard deviations over the repetitions reported in our ﬁgures.
3.1 Quantifying active learning bias
We estimate the extent of active learning bias when using ˜ r and verify the unbiasedness of ˜ r LURE .
A DUN is trained on 1 , 000 randomly selected data points from D pool using the standard negative log-likelihood (NLL) loss.
We then estimate the model’s risk using both ˜ r and ˜ r LURE , drawing M samples from an unobserved test set D test .
The active learning bias is estimated as
[Equation]
(3) where • is either ˜ r or ˜ r LURE and r is computed on the whole D test .
Figure 2 shows that ˜ r LURE is unbiased, whereas the bias in ˜ r approaches zero only as M → N
[Figure]
3.2 Overﬁtting bias
To test the hypothesis that DUNs’ prior over depth reduces overﬁtting in low data regimes, we follow the procedure of Farquhar et al.
[2021] to isolate overﬁtting bias.
We train a model with M actively sampled points using ˜ R LURE as the training objective.
After acquiring each new point, we evaluate
[Equation]
The difference between the risk evaluated on the test set ( r ) and on the train set once the active learning bias is removed ( ˜ R LURE ) is taken as a measure of overﬁtting ( B OFB ).
Figure 3 shows that the magnitude of B OFB for DUNs is small relative to BNNs trained with MCDO [Gal and Ghahramani, 2016], one of the methods used in Farquhar et al.
This suggests that
DUNs are, indeed, robust to overﬁtting on small datasets.
Overﬁtting bias in DUNs is also larger in magnitude than active learning bias for the datasets shown (comparing ﬁgure 3 to ﬁgure 2).
This is not the case for MCDO (comparing ﬁgure 3 to ﬁgure C.2).
[Figure]
3.3 Training with unbiased risk estimators
According to Farquhar et al.
[Figure]
removing the active learning bias in cases where | B ALB | (cid:29) | B OFB | should positively affect performance.
Indeed, they observe this for linear models.
Thus, having conﬁrmed that the LURE weights eliminate active learning bias in DUNs
[Figure]
and that
DUNs are in the | B ALB | (cid:29) | B OFB | regime
[Figure]
at least for certain datasets
[Figure]
we evaluate whether using the unbiased estimator during training improves downstream performance.
We compare the NLL obtained by evaluating our models on D test when using ˜ R and ˜ R LURE as the training objectives.
Surprisingly, ﬁgure 4 shows that using ˜ R LURE does not affect NLL performance at all.
[Figure]
Discussion
In ﬁgure [Figure] we measure our models’ generalisation error, given by bias 2 + variance
[Bishop, 2007].
Considering that neural networks are very ﬂexible, we expect model bias to be negligible, and active learning bias to dominate.
When we eliminate the latter with importance weights, as in equation (2), we pay the price of additional variance
[Owen, 2013].
We therefore hypothesise that our results can be explained by gains due to bias reduction being lost due to increased variance.
This increase in variance will depend on the choice of sampling distribution α .
“Overﬁtting bias”, as presented in equation (4) and [Farquhar et al., 2021] is computed using the train set and thus is not directly comparable with active learning bias in equation (3).
Furthermore, the effect of “overﬁtting bias” is captured by model variance.
The error induced by model variance is always additive with any statistical biases that may exist, precluding any cancellation effects.
This hypothesis also explains why Farquhar et al.
ﬁnd ˜ R LURE to be successful for linear models: since inﬂexible models are well-speciﬁed by small amounts of data, applying ˜
R LURE barely increases variance.
We also highlight that in the experimental design used by Farquhar et al.
[2021] to obtain their linear model results, the temperature of the proposal distribution ( T = 10 , 000 ) is much larger than that used here ( T = 10 ).
Higher temperatures approach a deterministic proposal, interfering with the assumption that α ( · ) has full support.
A Depth uncertainty networks
BNNs translate uncertainty in weight-space into predictive uncertainty by marginalising out the posterior over weights.
The intractability of this posterior necessitates the use of approximate inference methods.
DUNs, in contrast, leverage uncertainty about the appropriate depth of the network in order to obtain predictive uncertainty estimates.
There are two primary advantages of this approach: 1) the posterior over depth is tractable, mitigating the need for limiting approximations; and 2) due to the sequential structure of feed-forward neural networks, both inference and prediction can be performed with a single forward pass, making DUNs suitable for deployment in low-resource environments [Antorán et al., 2020].
DUNs are composed of subnetworks of increasing depth, with each subnetwork contributing one prediction to the ﬁnal model.
The computational model for DUNs is shown in ﬁgure A.5.
Inputs are passed through an input block, f 0 ( · ) , and then sequentially through each of D intermediate blocks { f i ( · ) } Di =1 , with the activations of the previous layer acting as the inputs of the current layer.
The activations of each of the D + 1 layers are passed through the output block f D +1 ( · ) to generate per-depth predictions.
These are combined via marginalisation over the depth posterior, equivalent to forming an ensemble with networks of increasing depth.
Variation between the predictions from each depth can be used to obtain predictive uncertainty estimates, in much the same way that variance in the predictions of different sampled weight conﬁgurations yield predictive uncertainties in BNNs.
[Figure]
A.1 Uncertainty over depth
Figure A.6 compares the graphical model representations of a BNN and a DUN.
In BNNs, the weights θ are treated as random variables drawn from a distribution p γ ( θ ) with hyperparameters γ .
In DUNs, the depth of the network d is assumed to be stochastic, while the weights are learned as hyperparameters.
A categorical prior distribution is assumed for d , with hyperparameters β : p β ( d ) =
Cat( d | { β i } Di =0 ) .
The model’s marginal log likelihood (MLL) is given by
[Equation]
where the likelihood for each depth is parameterised by the corresponding subnetwork’s output: p ( y | x , d = i ; θ )
= p ( y | f D +
( a i ; θ )) , where a i = f
i ( a i − 1 ) .
[Equation]
[Figure]
[Equation]
[Figure]
The posterior,
[Equation]
is also a categorical distribution, whose probabilities reﬂect how well each depth subnetwork explains the data.
DUNs leverage two properties of modern neural networks: ﬁrst, that successive layers have been shown to extract features at increasing levels of abstraction [Zeiler and Fergus, 2014]; and second, that networks are typically underspeciﬁed, meaning that several different models may explain the data well [D’Amour et al., 2020].
The ﬁrst property implies that initial layers in a network specialise on low-level feature extraction, yielding poor predictions from the shallower subnetworks in a DUN.
This is handled by the depth posterior assigning low probabilities to earlier layers, in preference for later layers that specialise on prediction.
The second property suggests that subnetworks at several depths are able to explain the data simultaneously and thus have non-zero posterior probabilities, yielding the diversity in predictions required to obtain useful estimates of model uncertainty.
A.2 Inference in DUNs
Antorán et al.
ﬁnd that directly optimising the MLL equation (5) with respect to θ results in convergence to a local optimum in which all but one layer is attributed a posterior probability of zero.
That is, direct optimisation returns a deterministic network of arbitrary depth.
The authors instead propose a stochastic gradient variational inference (VI) approach with the aim of separating optimisation of the weights θ from the posterior.
A surrogate categorical distribution q φ ( d ) is introduced as the variational posterior.
The following evidence lower bound (ELBO) can be derived:
[Equation]
allowing θ and the variational parameters φ to be optimised simultaneously.
Antorán et al.
[2020] show that under the VI approach, the variational parameters converge more slowly than the weights, enabling the weights to reach a setting at which a positive posterior probability is learnt for several depths.
It is important to note that equation (7) can be computed exactly, and that optimising L ( φ, θ ) is equivalent to exact inference of the true posterior p ( d | D train ; θ ) in the limit: since both q and p are categorical, equation (7) is convex and tight at the optimum (i.e., q φ ( d )
= p ( d | D train ; θ ) ).
Since the expectation term can be computed exactly using the activations at each layer (recall p ( y | x , d = i ; θ )
= p ( y | f D +
( a i ; θ )) ), the ELBO can be evaluated exactly without Monte Carlo sampling.
B Experimental setup B.1 Data
We implement batch-based active learning, with batches of 20 data points acquired in each query (with the exception of Yacht, which has a query size of 10 due to its smaller size).
An initial training set representing a small proportion of the full data is selected uniformly at random in the ﬁrst query.
For all datasets, 80% of the data are used for training, 10% for validation and 10% for testing.
Details about dataset sizes, input dimensionality, and active learning speciﬁcations for each dataset are provided in table B. [Table]
[Table]
B.2 Model speciﬁcations
We implement a fully-connected network with residual connections, with 100 hidden nodes per layer.
The networks contain 10 hidden layers for DUNs, or three hidden layers for MCDO.
We use ReLU activations, and for DUNs batch normalisation is applied after every layer [Ioffe and Szegedy, 2015].
Optimisation is performed over 1 , 000 iterations with full-batch gradient descent, with momentum of 0 . 9 and a learning rate of 10 − 4 .
A weight decay value of 10 − 5 is also used.
We do not implement early stopping, but the best model based on evaluation of the evidence lower bound on the validation set is used for reporting ﬁnal results.
For MCDO models a ﬁxed dropout probability of 0 . 1 is used and prediction is based on 10 MC samples.
DUNs use a uniform categorical prior over depth.
The hyperparameter settings largely follow those used in Antorán et al.
[2020] for their experiments with toy regression datasets.
All experiments are repeated 40 times with different weight initialisations and train-test splits (with the exception of the Protein dataset, for which experiments are repeated only 30 times due to the cost of evaluation on the larger test set).
Unless otherwise speciﬁed, we report the mean and standard deviation of the relevant metric over the repeated experiment runs.
Following Farquhar et al.
[2021], the proposal distribution α ( i m ; i 1: m − 1 , D pool ) used is a stochastic relaxation of the Bayesian active learning by disagreement (BALD) acquisition function proposed by [Houlsby et al., 2011].
BALD scores are multiplied by a constant temperature T before being passed through the softmax function; candidate points are then sampled with probability proportional to the softmax probabilities.
We use
T = 10 as this is found to yield the best performance for most of the datasets considered (results for experiments to determine the optimal value of T are given in appendix C.4).
C Additional experimental results C.1 Quantifying active learning bias
Figure C.1 presents the results of the active learning bias experiments, as in ﬁgure 2, for the remaining datasets.
Figure C.2 shows equivalent results for BNNs trained with MCDO.
[Figure]
[Figure]
C.2 Overﬁtting bias
Figure C.3 compares the magnitude of overﬁtting bias for DUNs and BNNs trained with MCDO, as in ﬁgure 3.
Figure C.3 shows additionally the overﬁtting bias when training with ˜ R .
[Figure]
C.3 Training with unbiased risk estimators
Figure C.4 presents the results of training DUNs with ˜ R and ˜ R LURE , as in ﬁgure 4, for the remaining datasets.
[Figure]
Figure C.5 shows the test NLL for DUNs using different temperatures T for the proposal distribution.
[Figure]
C.5 Depth posteriors for DUNs at different stages of active learning
Figure C.6 shows the posterior distributions over depth for DUNs for the smallest and largest dataset sizes used during active learning
[Figure]