B EST P RACTICES IN P OOL - BASED A CTIVE L EARNING FOR I MAGE C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
The recent popularity of active learning (AL) methods for image classiﬁcation using deep-learning has led to a large number of publications that contributed to signiﬁcant progress in the ﬁeld.
Benchmarking the latest works in an exhaustive and uniﬁed way and evaluating the improvements made by the novel methods is of key importance to advance the research in AL.
Reproducing state-of-the-art AL methods is often cumbersome, since the results and the ranking order of different strategies are highly dependent on several factors, such as training settings, used data type, network architectures, loss function and more.
With our work we highlight the main factors that should be considered when proposing new AL strategies.
In addition, we provide solid benchmarks to compare new with existing methods.
We therefore conduct a comprehensive study on the inﬂuence of these key aspects, providing best practices in pool-based AL for image classiﬁcation.
We emphasize aspects such as the importance of using data augmentation, the need of separating the contribution of a classiﬁcation network and the acquisition strategy to the overall performance, the advantages that a proper initialization of the network can bring to AL.
Moreover, we make a new codebase available, that enables state-of-the-art performance for the investigated methods, which we hope will serve the AL community as a new starting point when proposing new AL strategies.
I NTRODUCTION
Deep learning methods require large amounts of labeled data samples to train.
Unfortunately, annotating new datasets consisting of thousands or millions of images is very costly.
A research topic that focuses on maximizing the performance of deep learning models with a given annotation budget is active learning (AL).
AL is a machine learning’s sub-ﬁeld in which the algorithm is allowed to query an information source for the label of new data samples (Settles, 2009).
Recently, an abundance of pool-based AL methods for image classiﬁcation have been proposed (Caramalau et al., 2021a;b;
Kim et al., 2021; Liu et al., 2021; Choi et al., 2021).
In pool-based AL (Lewis & Gale, 1994) there exists a large unlabeled U and a small labeled sample set L .
The labeled set is used to train the classiﬁer and the large unlabeled set is used to query new samples that then are annotated and added to the labeled set in order to include them for training.
The goal of pool-based AL is to iteratively sample and annotate unlabeled training samples from the pool and add them to the training set in order to achieve the best performance given a certain annotation budget.
Pool based AL is often particularly useful because large amounts of unlabeled data is often available but annotating it is cumbersome and costly.
In AL there exist multiple baselines that either form lower or upper bounds for AL methods.
Typically, using all the unlabeled samples, annotate them and include them for training forms an upper bound that AL methods aim to reach with as few samples as possible.
In contrast, selecting new samples that should be labeled and used for training randomly from the set of unlabeled samples establishes a lower bound that any AL method should exceed in every AL cycle.
AL methods can be split into uncertainty or diversity based samples.
The former select samples for which the classiﬁer is uncertain (Gal et al., 2017; Beluch et al., 2018; Yoo & Kweon, 2019; Mayer & Timofte, 2020), while the latter selects samples in a diverse way (Sener & Savarese, 2018; Sinha et al., 2019; Gissin & Shalev-Shwartz, 2019; Caramalau et al., 2021a), ensuring to cover the whole data distribution of the unlabeled set.
Recently proposed strategies, are often still based on these concepts, combine them (Ash et al., 2019; Zhang et al., 2020; Kim et al., 2021) or propose new ideas such as using unlabeled data to train addition modules (Shui et al., 2020; Caramalau et al., 2021b) used for sample selection or combine AL with semi-supervised learning (Sim’eoni et al., 2021;
Huang et al., 2021; Gao et al., 2020; Mittal et al., 2019; Liao et al., 2021).
The recent popularity of AL methods complicates comparing these works in a complete and standardized manner.
However, assessing the improvements made by new methods is crucial to boost the progress in AL.
Hence, this work provides best practices to keep in mind, when proposing new pool-based AL methods for image classiﬁcation using deep learning and comparing them to existing methods form the literature.
Contributions:
In summary, our contributions are as follows: (i)
We perform an extensive analysis of popular AL methods and study the effect of different settings typically used in the literature.
(ii)
In addition, we discuss the merit of recent trends in AL such as using unlabeled data, pretraining or initial set construction.
(iii)
We incorporate our ﬁndings into a collection of best practices to keep in mind when evaluating pool based AL methods for image classiﬁcation.
(iv) We make a new Pytorch code base available 1 for AL that contains many popular AL methods and datasets and allows to evaluate and compare different strategies in a fair way.
Code will be released upon publication.
R ELATED W ORK
AL evaluations: Recently the attention on benchmarking and unifying results of new AL strategies increased.
Beck et al.
(2021) conducted a rich set of experiments to evaluate the performance of
AL methods, including (Wei et al., 2015; Sener & Savarese, 2018; Ash et al., 2019; Killamsetty et al., 2021).
They assessed the robustness of the methods by adding redundancy to the datasets, concluding that diversity based strategies are more stable in this scenario.
In addition, they studied the effect of using data augmentation, using different optimizers, updating the models instead of reinitializing them in every iteration, and using different batch sizes.
In contrast, we perform similar experiments but include more AL methods on multiple datasets, evaluated additional settings, and study the effect of using unlabeled samples for AL.
Munjal et al.
(2020) focused on benchmarking the strategies proposed in (Gal et al., 2017; Beluch et al., 2018; Sener & Savarese, 2018; Sinha et al., 2019) using a uniform experimental setting.
They vary the AL batch size, the validation set size, and the amount of class imbalance.
They ﬁnd that under changing experimental condition no strategy offers gain over Random sampling.
Conversely, our experiments show that AL methods clearly outperform Random sampling when training the classiﬁer carefully.
Chong et al.
(2021) evaluates traditional AL methods and Coreset on an imbalanced medical dataset (Rahman et al., 2021) and on CIFAR10.
They highlight the importance of respecting class imbalance in AL and conclude that Coreset performs better than Uncertainty sampling.
In contrast to these last two works, we omit analyzing the robustness of AL strategies on imbalanced datasets but rather focus on balanced datasets such as CIFAR10.
Investigated AL strategies: The investigated AL strategies are a relevant selection of recent deep pool based AL methods used in deep learning.
In particular, we study Coreset, BADGE, LL4AL,
JLS and WAAL, and Uncertainty sampling.
We brieﬂy summarize the most recent ones.
Coreset (Sener & Savarese, 2018) selects new unlabeled samples by solving the k-Center problem (Wolf, 2011) on the feature space of the last fully-connected layer.
Ash et al.
(2019) propose an acquisition strategy (BADGE) based on k-MEANS++ (Arthur & Vassilvitskii, 2007), which is applied on so called hallucinated gradients with respect to the parameters of the ﬁnal layer of the deep network.
Hallucinated gradients are used because the true labels are not available for unlabeled samples.
Therefore, the most likely labels according to the classiﬁer are used to compute the gradient.
Yoo &
Kweon (2019) propose LL4AL, based on the assumption that data points for which the classiﬁcation network is likely to produce a wrong prediction are most informative and should be used for training.
To this end, they jointly train a target module and a loss prediction module that aims at predicting the loss of a given unlabeled sample.
The samples achieving the highest losses are then added to the training set.
JLS (Caramalau et al., 2021b) employs a discriminator network besides the classiﬁer.
The task of the discriminator is differentiating whether a given sample belongs to the set of labeled or unlabeled samples.
The samples that the discriminator predicts to be unlabeled with a high conﬁdence are selected and annotated.
JLS shares intermediate backbone layers for classiﬁcation and discrimination such that training the discriminator with unlabeled data turns out to be beneﬁcial for the classiﬁer as well.
Shui et al.
(2020) propose an acquisition strategy (WAAL) that combines the uncertainty score of the task model with a diversity term based on the Wasserstein metric.
To compute the diversity metric, an adversarial network is used that is optimized to discriminate between labeled and unlabeled data.
Both networks share the same feature extractor and are jointly trained through a min-max optimization problem that uses labeled and unlabeled data.
E XPERIMENTAL S ETUP
Since the accuracy range of AL methods extensively changes across the literature and their provided codebases, we provide our own that aims at maximizing AL accuracy for all the implemented methods.
Our codebase builds upon a repository 2 which only implements the LL4AL strategy for
CIFAR10 and achieves the result presented in (Yoo & Kweon, 2019).
We re-implement multiple other methods such as Random sampling, Uncertainty sampling, Coreset (greedy K-center variant),
BADGE, JLS, VAAL and include them into the codebase.
Furthermore, we adapt the codebase in order to support additional datasets such as SVHN and FashionMNIST.
The different methods can be tested on any dataset and our codebase allows to examine the effect of various different settings such as warm or cold start, different classiﬁers, unsupervised pretraining, changing the acquisition strategy to sample the initial dataset, different training settings, such as jointly training a discriminator or a loss prediction module for all methods.
We use the settings in (Yoo & Kweon, 2019) because they achieve results with high accuracy on
CIFAR10 for all the reported methods.
Note that the default settings serves mainly as comparison benchmark, and does not lead to the highest absolute accuracy for each method across all datasets.
Unless stated otherwise, our default training setting is as follows: We use ResNet-18 (He et al., 2016) as backbone and train with a mini-batch size of 128, for 200 epochs, with SGD optimizer, an initial learning rate of 0.1, momentum of 0.9 and weight decay of 0.0005.
After training forepochs, we decrease the learning rate to 0.01.
Instead of initializing the network weights in every
AL cycle we use the weights of the network obtained in the previous AL cycle (warm start).
We use data augmentation (random horizontal ﬂips and random crop) and normalize the input images.
The initial training set consists of 1000 labeled samples.
We add 1000 samples in each AL step, except for the experiments focusing on the use of unlabeled data where we use the JLS and WAAL settings.
Our codebase allows to reproduce state-of-the-art results for the aforementioned methods for all three datasets, for different training setting and annotation budgets.
All results reported in the paper are averaged over three runs with different random seeds.
For the experiments we use three common datasets used in the AL literature: CIFAR10 (Krizhevsky, 2009), consisting of 60k 32 × 32 RGB images, divided into 50k training and 10k testing images split into 10 classes.
The datasets are balanced, i.e. each class has the same number of images.
SVHN (Netzer et al., 2011) contains images showing house numbers obtained from Google Street
View.
It consists of 73,257 training, 26,032 testing, and 53,1131 extra training images.
The RGB images have a resolution of 32 × 32 , are centered around a single digit and are split into 10 classes.
The extra training images are not used in this work.
FashionMNIST (Xiao et al., 2017) contains 28 × 28 gray-scale images showing articles from an online shop.
The dataset consists of 60k training and 10k testing images.
Figure 1 shows the results when using our codebase and the default settings on three datasets.
In the paper we often report only the accuracy difference plots but omit the ﬁgures showing the absolute accuracy, or we only show the result on a single dataset, or we omit other AL strategies such as
VAAL or JLS to keep the ﬁgures clean, or we only report the mean but omit the standard deviation.
For additional plots containing all these results we refer the reader to the Appendix.
D IFFERENT TRAINING SETTINGS AND THEIR EFFECT ON AL E VALUATION
Newly proposed AL strategies typically come with their own hyper parameters, training strategies and model architectures in order to outperform existing methods on the same dataset.
However, changing these aspects complicates comparing different AL methods with each other.
For example, when using another classiﬁcation network (backbone) than existing methods a fair comparison
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
requires to re-implement these methods or change the provided source code.
However, the default hyper parameters and training settings may not be optimal for the new backbone.
In Table 1 we summarize settings for recent AL methods.
We observe multiple choices for all the reported settings.
In order to enable a fair comparison among AL methods a uniﬁed evaluation strategy is required.
Hence, we examine in this section the effect on AL when using the different settings by changing the default values in our proposed code base.
In particular, we study whether changing the settings affect only the overall accuracy, leads to a change in the ranking of AL methods or even results into a vanishing beneﬁt of AL.
We compare four existing AL methods (LL4AL, BADGE, Uncertainty sampling and Coreset) with each other and the baseline of Random sampling.
Finally, we compare our code base with the code base used for BADGE that not only provides the implementation of
BADGE but also of a few existing methods.
Backbone Architecture: First, we investigate the impact on AL performance when changing the backbone.
In particular, we compare ﬁve methods on CIFAR10 using either VGG-16, ResNet-18 or
DenseNet-201.
Figure 2 shows the achieved accuracy for the different backbones.
We conclude, that different backbones achieve different classiﬁcation accuracy, with ResNet-18 achieving the highest.
Hence, it is crucial to ensure that only methods using the same backbone are compared with each other.
In addition, we observe that the ranking order of AL methods changes for Densenetcompared to ResNet-18 and VGG-16.
This probably due to overall low accuracy leading to poor sample selection.
Therefore, we propose to validate new methods on ResNet-18 since it achieves the highest accuracy and signiﬁcant improvements over Random sampling for all investigated methods.
For methods that beneﬁt from using other backbones we recommend to validate for this case on both the new backbone and ResNet-18 to enable a fair and insightful comparison.
Initializing Backbone Weights:
In AL the classiﬁer is re-trained in every active learning cycle to leverage the newly sampled and annotated data.
According to Table 1 two different strategies are common in the literature: cold and warm start.
Cold start describes the approach to re-initialize the network weights in every cycle from scratch.
In contrast, warm start uses the network weights obtained in the former AL cycle by training with the corresponding annotated samples.
Warm start resembles ﬁne-tuning but we use the same learning rate and number of training epochs for cold and warm start.
Figure 3 shows the accuracy achieved with warm start minus the accuracy of cold start on three datasets.
We observe that depending on the dataset either cold or warm start performs better in the ﬁrst few AL cycles but the difference diminishes on all datasets for almost all methods.
Except LL4AL that performs better with warm start across all datasets.
Figure 4c compares the performance of LL4AL with other AL methods when all use cold start and we observe that LL4AL no longer outperforms other AL methods (see ﬁgure 1a) but achieves comparable accuracy. [Figure] [Figure]
Hence [Figure] we conclude that cold or warm start inﬂuences AL performance in particular at the beginning of AL but the difference tends to diminish later on.
To ensure that the improvement of an AL methods originate from a superior sample selection strategy instead of from improved settings method, we advise to use the approach which works best for the investigated method.
For LL4AL we advise to always use warm start, while for the other methods it depends on the annotation budget and on the dataset.
Optimizer and Learning Rates: Different optimizers such as SGD, Adam and RMSProp with varying learning rates from 0.001 to 0.1 are commonly used in the literature (see Table 1).
First, we investigate the effect of varying learning rates when using the most popular optimizer SGD.
We perform experiments when using cold and warm start to initialize the network weights, motivated by the assumption that different initialization approaches may require different learning rates.
Figures 4a and 4b show the accuracy achieved with a learning rate of 0.1 minus the accuracy with 0.
for CIFAR10.
We observe that using a larger learning rate is beneﬁcial for both, cold and warm start.
In addition, a smaller learning rate leads to higher classiﬁcation accuracy when training only on the initial dataset.
Furthermore, we perform experiments with two different optimizers: SGD and Adam.
We tested different learning rates used in the literature (see Table 1) and use the learning rate 0.1 that leads to the best performance for either optimizer (see A.13 for more details).
Figure 5 shows the accuracy achieved with SGD minus the accuracy achieve when using Adam .
We observe SGD leads in particular on CIFAR10 to superior performance by achieving roughly 6% points higher accuracy than training with Adam.
On SVHN and FashionMNIST SGD outperforms Adam as well but the difference is smaller with 0.5 - 1.0% or 0.5 - 1.5% points.
Hence, we conclude that using the same optimizer and learning rate to train the backbone when comparing different AL methods is crucial.
Otherwise it is unclear whether the performance difference stems from the AL method or the training setup.
With and Without Data Augmentation: Whether to train the backbone with or without data augmentation is another setting that often differs between different AL works (see Table 1).
Disabling data-augmentation typically decreases the overall accuracy of the classiﬁer (see ﬁgure 7) and can affect the ranking order of AL methods.
Figure 6a shows the result on our code base with and without [Figure] [Figure]
[Figure] data augmentation on CIFAR10. We observe, that the accuracy is at least 10% points lower when not using data augmentation and can differ up to 20% points and more.
In addition, all AL methods outperform random sampling when using data augmentation.
Conversely, the baseline random sampling outperforms all AL methods at least once on different operating points when data augmentation is not used.
Hence, disabling data augmentation not only decrease the overall performance but also leads to failure of multiple AL methods.
Supported by the above observations we therefore recommend data augmentation when comparing
AL for image classiﬁcation.
Comparing Codebases: Having examined all the aforementioned settings for AL it becomes clear that these settings are crucial when comparing AL methods.
While it is important to enable a fair environment when comparing different methods, we argue that it is useful to use a codebase that aims at maximizing the overall accuracy.
In this part, we want to highlight the importance of such a codebase by comparing it to the codebase used for BADGE.
In contrast to our codebase they
[Figure] [Figure]
use Adam instead of SGD, disable data augmentation and train for fewer epochs in each AL cycle and employ early stopping.
Figure 6b shows the results of four AL methods using our codebase in solid lines and the results of BADGE’s codebase in dashed lines.
We observe that our codebase leads to much higher accuracy.
In particular we achieve between 20% to almost 50% points higher classiﬁcation accuracy on CIFAR10 using the same annotation budget.
Furthermore, we observe that every AL method outperforms Random sampling when using our codebase.
In contrast, the baseline often outperforms AL methods when using the codebase of BADGE and therefore results in AL failure.
However, this is not a problem of the AL methods but is caused by the codebase.
We conclude that poorly trained backbones perform poorly for sample selection and therefore result in low performance of the AL strategy.
Furthermore, it is clear that a codebase achieving too low classiﬁcation accuracy is unsuitable for a fair comparison of different AL methods.
U SING UNLABELED D ATA
In this section we focus on the use of unlabeled data.
Recent AL methods such as VAAL, LL4AL or
JLS employ an additional module that is trained in each active learning cycle and used to query new samples.
While most AL methods do not use unlabeled data to train the classiﬁcation network (e.g.,
VAAL)
, or only use labeled data (e.g., LL4AL), there exist methods (e.g., JLS, WAAL) that use unlabeled data to train the backbone.
Hence, comparing methods that use unlabeled samples with methods that only use labeled samples is unfair.
In order to compare such methods with existing ones, we propose to train the backbone of all methods with unlabeled samples.
We perform this experiments using the training strategies of JLS and WAAL.
In particular, we use exactly the same network for each AL strategy including the discriminators but we use the original sample selection strategy when querying new samples in each AL cycle.
Figure 8a shows a comparison between the performance of AL methods when training the classiﬁer and discriminator with unlabeled data (dashed lines) compared to their performance when only the backbone is trained (solid lines).
We observe that traditional methods such as Coreset and Uncertainty sampling outperform JLS when using the same networks and train with unlabeled samples.
Figure 8b shows similar results for
WAAL using their codebase instead of ours.
E FFECT OF ADDING M ODULES FOR COMPLEMENTARY TASKS
In this section, we demonstrate, that it is important to separate the achieved improvement caused by the acquisition strategy and a better trained classiﬁer when learning additional tasks beside classiﬁcation.
LL4AL trains the backbone combined with the loss prediction module , optimizing thus a different loss function.
To fairly asses the beneﬁt of their acquisition strategy, we train each AL method using exactly the same architecture as in LL4AL, but use the original sampling strategy when querying new images.
In ﬁgure 8c all methods train jointly also the loss prediction module .
Comparing the results to the default case, depicted in ﬁgure 1a, we observe that LL4AL is not achieving the top performance anymore.
The other methods achieve comparable accuracy.
This experiment shows that it is not the novel acquisition strategy proposed in LL4AL that leads to superior performance, but it is in fact the classiﬁer that beneﬁts of learning an additional task and results in better generalization.
B EFORE A CTIVE LEARNING STARTS
In this section we discuss two approaches that can inﬂuence the performance of AL methods, by either building the initial labeled dataset following a certain strategy instead of random sampling of the data or by pretraining the backbone in an unsupervised setting.
Construction of the Initial Training Set: Instead of randomly selecting the samples for the initial training set, it is possible to use a more sophisticate approach to construct the dataset.
In this part, we investigate whether it is worth focusing on selecting such a set and if a potential beneﬁt is preserved over multiple AL cycles or diminishes quickly.
In particular, we use a pretrained VGGon ImageNet (Deng et al., 2009) and compute the features of its last layer for every image in the datasets.
We then use K-means++ and greedy K-center on the extracted features to select the set of initial training samples.
We evaluate different AL methods and report in ﬁgure 9 the accuracy of randomly created initial sets minus the accuracy achieved when using K-means++/K-center to create the initial training set.
We observe in ﬁgure 9a that in the ﬁrst iteration the accuracy improves for all methods beside
LL4AL using K-means++ based initialization on CIFAR10 (more than 2% points comparing with the absolute numbers) and that the beneﬁt rapidly vanishes within the next few AL iterations.
Imagenet and CIFAR10 both contain natural images in contrast to SVHN that contains images less similar to Imagenet.
Therefore, it is of interest to verify if there is still a beneﬁt of another method to create the initial set than random sampling on SVHN.
The results depicted in ﬁgure 9b show that the samples selected with Kmeans++ result in lower performance in the ﬁrst iteration.
In ﬁgure 9c we observe that using K-center for constructing the initial labeled set, deteriorates the accuracy by more than 10% points in the ﬁrst AL cycle.
We assume that a diversity based method probably mainly selects outliers present in the dataset and results therefore in such a poor performance.
In contrast,
Zhang et al.
(2020) experienced a 2% points accuracy improvement on CIFAR10 with 5000 training samples.
However, they used the deep latent features of a trained VAE, which encourages dataset speciﬁc features that are spread over the latent space according to a deﬁned distribution.
One key observation is that the beneﬁt or the drawback of using another method than Random sampling to construct the initial set disappears rapidly within the next few AL cycles.
[Figure] [Figure] [Figure]
[Figure]
Unsupervised pretraining of the network: Typically, the parameters of the classiﬁer are randomly initialized before AL starts.
Conversely, in a warm start scenario, the weights obtained in the previous AL cycle are used in the current instead of re-initializing the weights from scratch.
In such a setting, AL performance could be increased by using unsupervised pretraining of the classiﬁer and use these dataset speciﬁc features as a starting point for AL.
Gidaris et al.
(2018) showed a massive improvement in accuracy for the task of classiﬁcation when using their proposed unsupervised pretraining method.
The idea is to train the network on predicting which 2d rotation has been applied to the input image.
This initialization method has already been used in AL context by Gudovskiy et al.
Preliminary experiments show that re-initializing the model at every AL cycle with the unsupervised weights compared to using them only for the ﬁrst iteration is beneﬁcial, thus we followed the former approach.
Figure 10a shows the absolute result when using unsupervised pretraining for different AL methods.
In contrast, Figure 10b shows the accuracy of the classiﬁer using randomly initialized weights minus achieve accuracy when using the unsupervised weights.
The results show that with the unsupervised pretrained network the overall accuracy improves by 25% in the ﬁrst AL cycle, and that the accuracy difference diminishes as the number of labeled samples increases.
Note, that LL4AL does not outperform the other benchmark in this case, probably because learning the additional task of the loss module does not help as much when the network is already properly trained (compare with the ﬁndings in section 6).
Furthermore, JLS that in addition to the unsupervised pretrained weights trains with unlabelled data at each training cycle achieves even lower performance than other AL methods.
The beneﬁt of jointly training the backbone and the discriminator by leveraging unlabeled data is only visible in the ﬁrst iteration, but standard acquisition strategies, such as Uncertainty sampling outperform JLS afterwards.
C ONCLUSION
In this paper we provide a comprehensive evaluation of recent deep AL strategies, setting a solid benchmark for different datasets.
We analyze different training settings and observe that they can cause overall accuracy differences and ranking shifts between AL methods.
Data augmentation and SGD combined with a learning rate of 0.1 provide higher accuracy for all AL methods when using a ResNet-18.
We study the effect of different initialization methods in the AL context.
In particular, advanced initial set construction methods rapidly reach the performance of a randomly selected one.
Furthermore, we observe that using unsupervised pretraining of the network greatly beneﬁts the classiﬁcation performance especially when the number of labeled samples is small.
Future AL methods should try to clearly disentangle the performance improvement obtained by changes in the classiﬁcation network and by the proposed acquisition strategy.
When evaluating a new method, it should ensured that the existing state-of-the-art AL strategies used as benchmarks achieve comparable results with the original accuracy reported on a speciﬁc dataset and AL interval.
Deviating from the standard settings for a novel method complicates a fair comparison.
Nonetheless, in such cases a new method should only compare to the baselines that use the novel settings instead of the standard ones if they achieve better results.
A A PPENDIX A. D ATASETS
The AL strategy must be robust to changes of the dataset.
Figure 11 shows the accuracy that the investigated AL methods have on CIFAR10, CIFAR100, SVHN, FashionMNIST and TinyImageNet.
Overall the AL strategies show an improvement over random sampling across the datasets, beside for TinyImageNet. As we have seen, often AL fails when dealing with poorly trained networks. The overall small accuracy on TinyImageNet and the big variance across the different random seeds indicate that the network is poorly trained and there is still room for further improvements for new
AL strategies and training settings.
Table 2 shows the performance of the default conﬁguration when trained on the full dataset.
[Table]
A. B ACKBONE
The inﬂuence of the backbone used for the classiﬁcation task is another key aspect that must be taken into account.
Since for LL4AL the loss module is trained jointly with the backbone by sharing the features of intermediate layers and no guideline was provided for which to use on different architecture, we chose to use for VGG networks (Simonyan & Zisserman, 2015) the features of the last four max pooling layers and for DenseNet architectures (Huang et al., 2017) the last four transition layers.
The inﬂuence of different backbones is depicted in ﬁgure 12.
Badge, Uncertainty and Coreset performer similarly to Random sampling with Densenet201.
ResNet-18 achieves the overall best performance.
A. C ODEBASE
The results for different codebases with their standard deviation are depicted in ﬁgure
A. D ATA AUGMENTATION VS NO AUGMENTATION
For all datasets LL4AL has a better relative ranking compared to the other methods when no data augmentation is used.
This probably due to the fact that without augmentation the networks are trained worse and the additional task of learning the losses increases more the ability of the classiﬁer to generalize.
Overall data augmentation improves signiﬁcantly the accuracy of AL strategies as can be seen from ﬁgure 15.
[Figure]
Badge performs similarly to better than LL4AL with the Adam optimizer.
The overall accuracy is worse.
Coreset performs poorer on the FashionMNIST dataset with Adam.
Figure 17 shows the accuracy difference between the default settings using SGD optimizer and the results using Adam optimizer for different datasets.
A. L EARNING RATES
The results of using different learning rates ( LR ) are shown in ﬁgure 18.
Using a smaller LR seems beneﬁcial in the ﬁrst iterations but it worsens the performance as the iterations progress.
Since a smaller LR combined with a Warm-start approach may lead to a model that doesn’t learn enough from the new added samples, we investigated the inﬂuence of different learning rates also with a cold-start approach as can be seen in ﬁgure 19.
Again using a smaller learning rate seems beneﬁcial in the ﬁrst iterations but it worsens the performance as the iterations progress.
A. W ARM START VS C OLD START
We corroborate the ﬁndings of Beck et al.
(2021) that especially in the ﬁrst iterations retraining the model at every cycle improves the overall performance for most AL strategies and that the difference between cold and warm start diminishes in the later AL rounds.
Surprisingly LLL4AL beneﬁts from updating the backbone and shows worse performance when compared to the other AL strategies.
Note that however this ﬁndings do not generalize over different datasets.
In fact in ﬁgure 21 we see that a warm-start approach is to prefer when using the SVHN dataset for all the methods under investigation and that no signiﬁcant difference between the two approaches can be observed on
FashionMNIST, beside LL4AL that performs slightly worse compared to the other methods with the cold-start approach.
Figure 22 shows the performance of the strategies when selecting for LL4AL a Warm-start approach while for the others Cold-start.
It emphasises the importance of choosing the best possible conﬁguration for a speciﬁc AL strategy, since here neither LL4AL outperforms the other benchmark strategies nor the contrary is true.
A. B ATCH SIZE ( NUMBER OF QUERIED SAMPLES IN EACH AL CYCLE )
In this section we want to analyze the effect of the AL batch size (the number of queried samples in each AL cycle).
For this matter, we run the same experiment with three different batch sizes: 100, 1000 and 2000.
In all of them, we started with 1000 labeled data points and stopped when more than 10000 were labeled.
The results are reported in ﬁgure 23.
Using a warm-start approach combined with a small batch size improves the overall accuracy of all
AL strategies, especially in the ﬁrst iterations, as can be seen from ﬁgure 24.
A. U SE OF LABELED AND UNLABELED DATA
Figure 25 shows a comparison between the performance of the AL strategies when all methods train jointly also the discriminator compared to their performance when only the backbone is trained.
Similarly ﬁgure 26 shows the same for the WAAL method.
A. N ETWORK A RCHITECTURE / L OSS FUNCTION
Figure 27 shows a comparison between the performance of the AL strategies when all methods train jointly also the loss module compared to their performance when only the backbone is trained
A. I NITIALIZATION A.11. I NITIAL TRAINING SET
From ﬁgure 28 we see that in the ﬁrst iteration we have a 2% points accuracy improvement for all the methods beside LL4AL, and that the beneﬁt rapidly vanishes with the following AL iterations.
The results depicted in ﬁgure 29 show us that on SVHN the samples selected with Kmeans++ with the Imagenet features worsen the performance in the ﬁrst iteration.
Figure 30 shows an accuracy deterioration of more than 10% points in the ﬁrst AL cycle using the initial labeled pool selected with K-center.
From ﬁgure 31 wee see that K-center initialization performs worse than random also with SVHN.
In this case the accuracy drop is smaller compared to the one observed with CIFAR10.
A.11. U NSUPERVISED PRETRAINING OF THE NETWORK
Results when using unsupervised pretraining of the network on CIFAR10 are shown in ﬁgure 32 and on CIFAR100 in ﬁgure 33.
A.11. S UPERVISED PRETRAINING OF THE NETWORK
Preliminary results, showed that sing a downloaded pre-trained (on ImageNet) ResNet 18 slightly worsen the overall performance of the AL methods. This probably because of the higher resolution of ImageNet.
Since pre-training ResNet18 on a low resolution ImageNet is very time consuming we opted to pre-train ResNet-18 on the TinyImageNet dataset and used it as our initial model at every
AL cycle.
As ﬁgure 34 shows, better performance is achieved using the supervised pre-trained network, compared to randomly initialized network.
A. L ABELING EFFICIENCY
Usually, when new AL methods are developed, the results are shown by comparing the test accuracy with respect to the number of labeled points for the new methods and for existing benchmarks.
This allows to highlight the beneﬁt of a newly proposed strategy compared to the baselines, but it does not give explicitly the reduction of annotated samples that a new sampling strategy needs to achieve the same test accuracy as random sampling, which is the main goal of every AL strategy.
Hence, we show in ﬁgure 35 the labelling efﬁciency of different AL methods using the default setting.
The labeling efﬁciency for a given test accuracy is deﬁned as the ratio of the minimum number of labeled images needed by random sampling divided by the minimum number of images needed by the investigated AL strategy, to reach a given accuracy.
We conclude that all AL methods are label efﬁcient across most datasets.
A. C HOICE OF THE LEARNING RATE FOR THE OPTIMIZER COMPARISON
To support our choice of comparing the optimizers using LR of 0.1, we present in ﬁgure 36 the results of the different AL strategies when using Adam or SGD as optimizers with a learning rate of 0.1 or 0.01.
Comparing the results, we can say that overall the methods reach a lower accuracy when using Adam optimizer combined with a LR of 0.01 compared to the methods when they use
SGD and a LR of 0.01.
In addition using Adam combined with a LR of 0.01 reaches slightly lower accuracy for all the methods compared to Adam and LR 0.1, making the latter the more appropriate choice when comparing different optimizers.
Concerning ranking shifts, we can see that LL4AL is only the top performer when using SGD combined with a LR of 0.1, while the ranking between the other methods stays the same.
[Figure]
There is no consensus across the literature on how many labeled data-points should be used for the ﬁrst AL cycle when reporting the results.
Figure 37 shows how methods that rely on a properly trained backbone, such as Coreset which uses its resulting feature representations of the images for the acquisition, or the Uncertainty method which relies on its output probabilities, perform similarly to Random sampling when the amount of labeled data is extremely small, while methods such as
LL4AL which learns jointly the additional task of the loss module performs signiﬁcantly better than
Random sampling also for the backbone trained on 200 images.
On the contrary when increasing by a factor of 5 the starting amount of labeled data no signiﬁcant difference in the ranking of the methods is seen compared to the default conﬁguration.
A. R EDUNDANCY
Redundancy is achieved by modifying the training set, such that there is a chosen amount of unique data-points and the remaining ones rest are repeated a given number of time.
In our experiment we chose to have 6000 unique data-points and duplicate the remaining images 2, respectively 20, times.
Figure 38 shows the respective results.
We corroborate the results of Beck et al.
(2021) that the diversity based approaches (BADGE and Coreset) are more robust to redundant dataset compared uncertainty based approaches (Uncertainty and LL4AL).
Since in our framework the Uncertainty method is performed o a random subset of the unlabeled set we still outperform Random sampling also in the case of a 20 × redundant dataset.
For LL4AL and Coreset no experiment were presented and therefore a comparison between our and their results is not possible.
A. S UBSET SELECTION
Uncertainty and LL4AL are acquisition strategies that are applied on a random subset of the unlabeled data as it is proposed in (Beluch et al., 2018) and (Yoo & Kweon, 2019).
This subset has been chosen to be 10 × the number of queried data. In this subsection we investigate the inﬂuence of selecting a subset not randomly, but using diversity based query strategies, such as BADGE and Coreset, to see if the uncertainty based methods beneﬁt from a diverse subset. The results are shown in ﬁgure 39.
The proposed methods perform almost en-par with their benchmarks.
The most relevant results are the accuracy difference between the modiﬁed Uncertainty, respectively LL4AL, and the default version (green and blue lines in the accuracy difference plots).
The beneﬁt of this subset selection can be seen best when the diversity of the subset is critical, for example in the case of a redundant dataset.
Since LL4AL and Uncertainty performed worse than Coreset and BADGE on the 20 × redundant dataset, we rerun the same experiment with the improved subset selections.
The results are shown in ﬁgure 40, where we see how LL4AL for both subset selection methods outperform all benchmarks, in particular BADGE and Coreset.
There is a signiﬁcant improvement also for Uncertainty, which now performs en-par with BADGE and Coreset.
A.16. R ANDOM SUBSET IS BENEFICIAL ?
A natural question that arises is whether the random subset selection is really beneﬁcial for the Uncertainty method.
Figure 42 shows a comparison of Uncertainty applied on the complete unlabeled dataset and only on a random subset of it.
Figure 43 shows what happens when we change the starting point and the AL batch size for Uncertainty on the full unlabeled set.
For the case where we start with 5000 labeled data and add 2500 at each iteration, we see that Uncertainty without subset selection outperforms by a large margin all benchmark methods and sets a strong benchmark that new proposed AL strategies should try to beat when considering this range of labeled data.
A. VAAL
The results for VAAL have not been consistent across the AL literature.
Some papers, such as (Sinha et al., 2019; Zhang et al., 2020; Kim et al., 2021), ﬁnd that VAAL outperforms Random sampling by signiﬁcant margin, while others, such as (Gudovskiy et al., 2020; Caramalau et al., 2021a)
ﬁnd that
VAAL achieves comparable performance as Random sampling.
We have reproduced the results of
VAAL using their codebase ﬁnding a comparable accuracy curve as presented in the paper.
However, the results for Random sampling using their codebase were signiﬁcantly better than the reported ones and comparable to the results of VAAL.
We also adapted their code to our framework, showing a signiﬁcant improvement in the overall accuracy ( ≥ 15% ).
However, comparing the best result for VAAL with the best result for Random sampling in our codebase we found again that the two methods perform en-par.
In detail our implementation differs in the following: We separated the training of the backbone from the training of the sampler; we trained the backbone with the same settings as for the other method; we got rid of the validation set, accounting for 10% of the whole dataset, to select the best model, since it is not common to have one in the AL literature (see table) and the trade-off between having additional data to train on compared to use them just for validation goes clearly in the direction of our adopted version.
A. JLS
What is the inﬂuence of the main difference hyperparameter settings that our default conﬁguration has compared to the ones of the origial JLS implementation?
Figure 45 shows the results thereof.
To clarify the legend we give a small explanation for every curve: JLS standard is the original paper implementaiton; Random standard is Random sampling using their default conﬁguration; Random + train sampler is Random sampling when the backbone is jointly trained with the discriminator;
LR 1 refers to a LR = 0 .
1 and LR 2 to a LR = 0 .
01 ; the order of the appearance of the learning rates is corresponding to the LR of the backbone (if it appears ﬁrst) or to the LR of the discriminator (if it appears second); JLS NoNesterov means we do not use Nesterov momentum (Sutskever et al., 2013) in the SGD optimizer; JLS Model update refers to the original implementation but with a
Warm-start approach.
Since the best performing variant is the JLS Model update we adopted that conﬁguration in our codebase.
The results of JLS in our codebase for different datasets can be seen in ﬁgure 46.
JLS on SVHN (not reported in (Caramalau et al., 2021b))
after the ﬁrst AL cycle performs worse than the other benchmark methods.
The authors reported better accuracy also for FashionMNIST, but chose for this dataset to use an extreme AL case with an initial labeled set consisting of only 100 images and a ﬁnal one of 700 images.
A. E XTREME AL
Which are the observations we can make when only a handful of samples are used?
Figure 47 shows the results for this extreme AL scenario on different datasets.
We can see that on more challenging datasets (CIFAR10 and CIFAR100) where the accuracy is relatively poor, there is no consistent gain of the AL strategies over Random sampling.
On CIFAR100 LL4AL, Uncertainty and Coreset even underperform Random sampling.
On easier datasets, such as FashionMNIST the methods perform slightly better than Random sampling.
In general from the dashed lines we see a big variance in the results especially in the ﬁrst iterations.
If the CNN is better trained (unsupervised pretraining), are we going to see better performance for the benchmark methods compared to Random sampling?
Figure 48 shows the corresponding experiment.
Even though the performance is much better with unsupervised pretraining (more than 100% accuracy improve in the ﬁrst cycles and still more than 40% in the last iteration) the benchmark methods perform comparably to Random sampling.
Note however that a real improvement over Random sampling is seen only after 75% accuracy in the case where we start with 1000 samples and add 1000 at each iteration.
This is also seen in this experiment: when the accuracy is ≥ 75% the methods tend to outperform Random sampling.
Still the variance stays high leading to unexpected behaviours like the huge accuracy drops for all methods after subsequent AL cycles.
B T ABLES
In this section we provide two tables: Table 3 summarize the recent proposed AL strategies, deﬁnes the investigated AL setup (AL budget, number of queried samples, etc) and gives the reported accuracy range of the proposed method and of Random sampling.
Table 4 provides the implementation details of the methods.
Unless otherwise stated, if for different datasets the setup differed too, then the corresponding setups where ordered in the same way as the datasets and separated by a slash.
A ’-’ means that the information was not mentioned.
The following abbreviation are used for the AL methods: Coreset for (Sener & Savarese, 2018), VAAL for (Sinha et al., 2019), BADGE for (Ash et al., 2019), LL4AL for (Yoo & Kweon, 2019), JLS for (Caramalau et al., 2021b), WAAL for (Shui et al., 2020), SRAAL for (Zhang et al., 2020), TA-VAAL for (Kim et al., 2021), ISAL for (Liu et al., 2021), FKSSAL for (Gudovskiy et al., 2020), CoreGCN and UncertainCGN for (Caramalau et al., 2021a), VaB-AL for (Choi et al., 2021). [Figure] [Figure]