Original: IALE: I MITATING A CTIVE L EARNER E NSEMBLES
Revised: IALE: I MITATING A CTIVE L EARNER E NSEMBLES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Active learning (AL) prioritizes the labeling of the most informative data samples.
Revised: Active learning (AL) prioritizes the labeling of the most informative data samples.

Original: However, the performance of AL heuristics depends on the structure of the underlying classiﬁer model and the data.
Revised: However, the performance of AL heuristics depends on the structure of the underlying classiﬁer model and the data.

Original: We propose an imitation learning scheme that imitates the selection of the best expert heuristic at each stage of the AL cycle in a batch-mode pool-based setting.
Revised: We propose an imitation learning scheme that imitates the selection of the best expert heuristic at each stage of the AL cycle in a batch-mode pool-based setting.

Original: We use DA GGER to train the policy on a dataset and later apply it to datasets from similar domains.
Revised: We use DA GGER to train the policy on a dataset and later apply it to datasets from similar domains.

Original: With multiple AL heuristics as experts, the policy is able to reﬂect the choices of the best AL heuristics given the current state of the AL process.
Revised: With multiple AL heuristics as experts, the policy is able to reﬂect the choices of the best AL heuristics given the current state of the AL process.

Original: Our experiment on well-known datasets show that we both outperform state of the art imitation learners and heuristics.
Revised: Our experiment on well-known datasets show that we both outperform state of the art imitation learners and heuristics.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The high performance of deep learning on various tasks from computer vision (Voulodimos et al., 2018) to natural language processing (NLP) (Barrault et al., 2019) also comes with disadvantages.
Revised: The high performance of deep learning on various tasks from computer vision (Voulodimos et al., 2018) to natural language processing (NLP) (Barrault et al., 2019) also comes with disadvantages.

Original: One of their main drawbacks is the large amount of labeled training data they require.
Revised: One of their main drawbacks is the large amount of labeled training data they require.

Original: Obtaining such data is expensive and time-consuming and often requires domain expertise.
Revised: Obtaining such data is expensive and time-consuming and often requires domain expertise.

Original: Active Learning (AL) is an iterative process where during every iteration an oracle (e.g. a human) is asked to label the most informative unlabeled data sample(s).
Revised: Active Learning (AL) is an iterative process where during every iteration an oracle (e.g. a human) is asked to label the most informative unlabeled data sample(s).

Original: In pool-based AL all data samples are available (while most of them are unlabeled).
Revised: In pool-based AL all data samples are available (while most of them are unlabeled).

Original: In batch-mode pool-based AL, we select unlabeled data samples from the pool in acquisition batches greater than 1 .
Revised: In batch-mode pool-based AL, we select unlabeled data samples from the pool in acquisition batches greater than 1 .

Original: Batch-mode AL decreases the number of AL iterations required and makes it easier for an oracle to label the data samples (Settles, 2009).
Revised: Batch-mode AL decreases the number of AL iterations required and makes it easier for an oracle to label the data samples (Settles, 2009).

Original: As a selection criteria we usually need to quantify how informative a label for a particular sample is.
Revised: As a selection criteria we usually need to quantify how informative a label for a particular sample is.

Original: Well-known criteria include heuristics such as model uncertainty (Gal et al., 2017; Roth & Small, 2006; Wang & Shang, 2014; Ash et al., 2020), data diversity (Sener & Savarese, 2018), query-by-committee (Beluch et al., 2018), and expected model change (Settles et al., 2008).
Revised: Well-known criteria include heuristics such as model uncertainty (Gal et al., 2017; Roth & Small, 2006; Wang & Shang, 2014; Ash et al., 2020), data diversity (Sener & Savarese, 2018), query-by-committee (Beluch et al., 2018), and expected model change (Settles et al., 2008).

Original: As ideally we label the most informative data samples at each iteration, the performance of a machine learning model trained on a labeled subset of the available data selected by an AL strategy is better than that of a model that is trained on a randomly sampled subset of the data.
Revised: As ideally we label the most informative data samples at each iteration, the performance of a machine learning model trained on a labeled subset of the available data selected by an AL strategy is better than that of a model that is trained on a randomly sampled subset of the data.

Original: Besides the above mentioned, in the recent past several other data-driven AL approaches emerged.
Revised: Besides the above mentioned, in the recent past several other data-driven AL approaches emerged.

Original: Some are modelling the data distributions (Mahapatra et al., 2018; Sinha et al., 2019; Tonnaer, 2017;
Revised: Some are modelling the data distributions (Mahapatra et al., 2018; Sinha et al., 2019; Tonnaer, 2017;

Original: Hossain et al., 2018) as a pre-processing step, or similarly use metric-based meta-learning (Ravi &
Revised: Hossain et al., 2018) as a pre-processing step, or similarly use metric-based meta-learning (Ravi &

Original: Larochelle, 2018; Contardo et al., 2017) as a clustering algorithm.
Revised: Larochelle, 2018; Contardo et al., 2017) as a clustering algorithm.

Original: Others focus on the heuristics and predict the best suitable one using a multi-armed bandits approach (Hsu & Lin, 2015).
Revised: Others focus on the heuristics and predict the best suitable one using a multi-armed bandits approach (Hsu & Lin, 2015).

Original: Recent approaches that use reinforcement learning (RL) directly learn strategies from data (Woodward &
Revised: Recent approaches that use reinforcement learning (RL) directly learn strategies from data (Woodward &

Original: Finn, 2018; Bachman et al., 2017; Fang et al., 2017).
Revised: Finn, 2018; Bachman et al., 2017; Fang et al., 2017).

Original: Instead of pre-processing data or dealing with the selection of a suitable heuristic they aim to learn an optimal selection sequence on a given task.
Revised: Instead of pre-processing data or dealing with the selection of a suitable heuristic they aim to learn an optimal selection sequence on a given task.

Original: However, these pure RL approaches not only require a huge amount of samples they also do not resort to existing knowledge, such as potentially available AL heuristics.
Revised: However, these pure RL approaches not only require a huge amount of samples they also do not resort to existing knowledge, such as potentially available AL heuristics.

Original: Moreover, training the RL agents is usually very time-intensive as they are trained from scratch.
Revised: Moreover, training the RL agents is usually very time-intensive as they are trained from scratch.

Original: Hence, imitation learning (IL) helps in settings where very few labeled training data and a potent algorithmic expert are available.
Revised: Hence, imitation learning (IL) helps in settings where very few labeled training data and a potent algorithmic expert are available.

Original: IL aims to train, i.e., clone , a policy to transfer the expert to the related few data problem.
Revised: IL aims to train, i.e., clone , a policy to transfer the expert to the related few data problem.

Original: While IL mitigates some of the previously mentioned issues of RL, current approaches are still limited with respect to their algorithmic expert and their acquisition size (including that of Liu et al.
Revised: While IL mitigates some of the previously mentioned issues of RL, current approaches are still limited with respect to their algorithmic expert and their acquisition size (including that of Liu et al.

Original: (2018)), i.e., some only pick one sample per iteration, and were so far only evaluated on NLP tasks.
Revised: (2018)), i.e., some only pick one sample per iteration, and were so far only evaluated on NLP tasks.

Original: We propose an batch-mode AL approach that enables larger acquisition sizes and that allows to make use of a more diverse set of experts from different heuristic families, i.e., uncertainty, diversity, expected model-change, and query-by-committee.
Revised: We propose an batch-mode AL approach that enables larger acquisition sizes and that allows to make use of a more diverse set of experts from different heuristic families, i.e., uncertainty, diversity, expected model-change, and query-by-committee.

Original: Our policy extends previous work (see Section 2) by learning at which stage of the AL cycle which of the available strategies performs best.
Revised: Our policy extends previous work (see Section 2) by learning at which stage of the AL cycle which of the available strategies performs best.

Original: We use
Revised: We use

Original: Dataset Aggregation (DA GGER ) to train a robust policy and apply it to other problems from similar domains (see Section 3).
Revised: Dataset Aggregation (DA GGER ) to train a robust policy and apply it to other problems from similar domains (see Section 3).

Original: We show that we can (1) train a policy on image datasets such as MNIST ,
Revised: We show that we can (1) train a policy on image datasets such as MNIST ,

Original: Fashion-MNIST , Kuzushiji-MNIST , and CIFAR-10 , (2) transfer the policy between them, and (3) transfer the policy between different classiﬁer architectures (see Section 4).
Revised: Fashion-MNIST , Kuzushiji-MNIST , and CIFAR-10 , (2) transfer the policy between them, and (3) transfer the policy between different classiﬁer architectures (see Section 4).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Next to the AL approaches for traditional ML models (Settles, 2009) also ones that are applicable to deep learning have been proposed (Gal et al., 2017; Sener & Savarese, 2018; Beluch et al., 2018;
Revised: Next to the AL approaches for traditional ML models (Settles, 2009) also ones that are applicable to deep learning have been proposed (Gal et al., 2017; Sener & Savarese, 2018; Beluch et al., 2018;

Original: Settles et al., 2008; Ash et al., 2020).
Revised: Settles et al., 2008; Ash et al., 2020).

Original: Below we discuss AL strategies that are trained on data.
Revised: Below we discuss AL strategies that are trained on data.

Original: Generative Models.
Revised: Generative Models.

Original: Explicitly modeled data distributions capture the informativeness that can be used to select samples based on diversity.
Revised: Explicitly modeled data distributions capture the informativeness that can be used to select samples based on diversity.

Original: Sinha et al.
Revised: Sinha et al.

Original: (2019) propose a pool-based semi-supervised
Revised: (2019) propose a pool-based semi-supervised

Original: AL where a discriminator discriminates between labeled and unlabeled samples using the latent representations of a variational autoencoder.
Revised: AL where a discriminator discriminates between labeled and unlabeled samples using the latent representations of a variational autoencoder.

Original: The representations are used to pick data points that are most diverse and representative (Tonnaer, 2017).
Revised: The representations are used to pick data points that are most diverse and representative (Tonnaer, 2017).

Original: Mirza & Osindero (2014) use a conditional generative adversarial network to generate samples with different characteristics from which the most informative are selected using the uncertainty measured by a Bayesian neural network (Kendall & Gal, 2017; Mahapatra et al., 2018).
Revised: Mirza & Osindero (2014) use a conditional generative adversarial network to generate samples with different characteristics from which the most informative are selected using the uncertainty measured by a Bayesian neural network (Kendall & Gal, 2017; Mahapatra et al., 2018).

Original: Such approaches are similar to ours (as they capture dataset properties) but instead we model the dataset implicitly and infer a selection heuristic via imitation.
Revised: Such approaches are similar to ours (as they capture dataset properties) but instead we model the dataset implicitly and infer a selection heuristic via imitation.

Original: Metric Learning.
Revised: Metric Learning.

Original: Metric learners such as Ravi & Larochelle (2018) use a set of statistics calculated from the clusters of un-/labeled samples in a Prototypical Network’s
Revised: Metric learners such as Ravi & Larochelle (2018) use a set of statistics calculated from the clusters of un-/labeled samples in a Prototypical Network’s

Original: (Snell et al., 2017) embedding space, or learn to rank (Li et al., 2020) large batches.
Revised: (Snell et al., 2017) embedding space, or learn to rank (Li et al., 2020) large batches.

Original: Such statistics use distances (e.g. Euclidean distance) or are otherwise converted into class probabilities.
Revised: Such statistics use distances (e.g. Euclidean distance) or are otherwise converted into class probabilities.

Original: Two MLPs predict either a quality or diversity query selection using backpropagation and the REINFORCE gradient (Mnih & Rezende, 2016).
Revised: Two MLPs predict either a quality or diversity query selection using backpropagation and the REINFORCE gradient (Mnih & Rezende, 2016).

Original: However, while they rely on statistics over the classiﬁer’s embedding and explicitly learn two strategies (quality and diversity) we use a richer state and are not constrained to speciﬁc strategies.
Revised: However, while they rely on statistics over the classiﬁer’s embedding and explicitly learn two strategies (quality and diversity) we use a richer state and are not constrained to speciﬁc strategies.

Original: Reinforcement Learning (RL).
Revised: Reinforcement Learning (RL).

Original: The AL cycle can be modeled as a sequential decision making problem.
Revised: The AL cycle can be modeled as a sequential decision making problem.

Original: Woodward & Finn (2018) propose a stream-based AL agent based on memory-augmented neural networks where an LSTM-based agent learns to decide whether to predict a class label or to query the oracle.
Revised: Woodward & Finn (2018) propose a stream-based AL agent based on memory-augmented neural networks where an LSTM-based agent learns to decide whether to predict a class label or to query the oracle.

Original: Matching Networks (Bachman et al., 2017) extensions allow for pool-based AL.
Revised: Matching Networks (Bachman et al., 2017) extensions allow for pool-based AL.

Original: Fang et al.
Revised: Fang et al.

Original: (2017) use Deep Q-Learning in a stream-based AL scenario for sentence segmentation.
Revised: (2017) use Deep Q-Learning in a stream-based AL scenario for sentence segmentation.

Original: In contrast to them we consider batch-mode AL with acquisition sizes ≥ 1 , and work on a poolinstead of a stream-settings.
Revised: In contrast to them we consider batch-mode AL with acquisition sizes ≥ 1 , and work on a poolinstead of a stream-settings.

Original: While Bachman et al.
Revised: While Bachman et al.

Original: (2017) propose a strategy to extend the RL-based approaches to a pool setting, they do still not work on batches.
Revised: (2017) propose a strategy to extend the RL-based approaches to a pool setting, they do still not work on batches.

Original: Instead, we allow batches of arbitrary acquisition sizes.
Revised: Instead, we allow batches of arbitrary acquisition sizes.

Original: Fan et al.
Revised: Fan et al.

Original: (2018) propose a meta-learning approach that trains a student-teacher pair via RL.
Revised: (2018) propose a meta-learning approach that trains a student-teacher pair via RL.

Original: The teacher also optimizes data teaching by selecting labeled samples from a minibatch that lets the student learn faster.
Revised: The teacher also optimizes data teaching by selecting labeled samples from a minibatch that lets the student learn faster.

Original: In contrast, our method learns to selects samples from an unlabeled pool, i.e., in a missing target scenario.
Revised: In contrast, our method learns to selects samples from an unlabeled pool, i.e., in a missing target scenario.

Original: The analogy of teacher-student is related, however, the objective, method and available (meta-)data to learn a good teacher (policy) are different.
Revised: The analogy of teacher-student is related, however, the objective, method and available (meta-)data to learn a good teacher (policy) are different.

Original: Multi-armed Bandit (MAB).
Revised: Multi-armed Bandit (MAB).

Original: Baram et al.
Revised: Baram et al.

Original: (2004) treat the online selection of AL heuristics from an ensemble as the choice in a multi-armed bandit problem.
Revised: (2004) treat the online selection of AL heuristics from an ensemble as the choice in a multi-armed bandit problem.

Original: COMB uses the known EXP4 algorithm to solve it, and ranks AL heuristics according to a semi-supervised maximum entropy criterion (Classiﬁcation Entropy Maximization) over the samples in the pool.
Revised: COMB uses the known EXP4 algorithm to solve it, and ranks AL heuristics according to a semi-supervised maximum entropy criterion (Classiﬁcation Entropy Maximization) over the samples in the pool.

Original: Building on this Hsu & Lin (2015) learn to select an AL strategy for an SVM-classiﬁer, and use importance-weighted accuracy extension to EXP4 that better estimates each AL heuristics’ performance improvement, as an unbiased estimator for the test accuracy.
Revised: Building on this Hsu & Lin (2015) learn to select an AL strategy for an SVM-classiﬁer, and use importance-weighted accuracy extension to EXP4 that better estimates each AL heuristics’ performance improvement, as an unbiased estimator for the test accuracy.

Original: Furthermore, they reformulate the MAB setting so that the heuristics are the bandits and the algorithm selects the one with the largest performance improvement, in contrast to COMB ’s formulation where unlabeled samples are the bandits.
Revised: Furthermore, they reformulate the MAB setting so that the heuristics are the bandits and the algorithm selects the one with the largest performance improvement, in contrast to COMB ’s formulation where unlabeled samples are the bandits.

Original: Chu & Lin (2016) extend Hsu & Lin (2015) to a setting where the selection of AL heuristics is done through a linear weighting, aggregating experience over multiple datasets.
Revised: Chu & Lin (2016) extend Hsu & Lin (2015) to a setting where the selection of AL heuristics is done through a linear weighting, aggregating experience over multiple datasets.

Original: They adapt the semi-supervised reward scheme from
Revised: They adapt the semi-supervised reward scheme from

Original: Hsu & Lin (2015) to work with their deterministic queries.
Revised: Hsu & Lin (2015) to work with their deterministic queries.

Original: In our own work, we instead learn a uniﬁed AL policy instead of selecting from a set of available heuristics.
Revised: In our own work, we instead learn a uniﬁed AL policy instead of selecting from a set of available heuristics.

Original: This allows our policy to learn interpolation between batches of samples proposed by single heuristics and furthermore, to exploit the classiﬁer’s internal state, so that it is especially suited for deep learning models.
Revised: This allows our policy to learn interpolation between batches of samples proposed by single heuristics and furthermore, to exploit the classiﬁer’s internal state, so that it is especially suited for deep learning models.

Original: Imitation Learning (IL).
Revised: Imitation Learning (IL).

Original: Liu et al.
Revised: Liu et al.

Original: (2018) propose a neural network that learns an AL strategy based on the classiﬁer’s loss on a validation set using Dataset Aggregation (DA GGER )
Revised: (2018) propose a neural network that learns an AL strategy based on the classiﬁer’s loss on a validation set using Dataset Aggregation (DA GGER )

Original: (Ross et al., 2011).
Revised: (Ross et al., 2011).

Original: One of their key limitations is that only a single sample is labeled during every acquisition.
Revised: One of their key limitations is that only a single sample is labeled during every acquisition.

Original: As the DL model is trained from scratch after every acquisition this results in a very slow active learning process and expensive expert-time is requested less efﬁciently (Kirsch et al., 2019; Sener & Savarese, 2018).
Revised: As the DL model is trained from scratch after every acquisition this results in a very slow active learning process and expensive expert-time is requested less efﬁciently (Kirsch et al., 2019; Sener & Savarese, 2018).

Original: Hence, we extend this work for batch-mode AL using a top-k -like loss function, and select more samples to increase the suitability to deep learning and its efﬁciency (as we do not retrain after each sample).
Revised: Hence, we extend this work for batch-mode AL using a top-k -like loss function, and select more samples to increase the suitability to deep learning and its efﬁciency (as we do not retrain after each sample).

Original: We also incorporate recent ideas (Ash et al., 2020) to extend the state and imitate multiple AL heuristics.
Revised: We also incorporate recent ideas (Ash et al., 2020) to extend the state and imitate multiple AL heuristics.

Original: This is computationally more efﬁcient and leads to better results.
Revised: This is computationally more efﬁcient and leads to better results.

Original:  IALE: I MITATING AN E NSEMBLE OF A CTIVE L EARNERS
Revised:  IALE: I MITATING AN E NSEMBLE OF A CTIVE L EARNERS

Original: IALE learns an AL sampling strategy for similar tasks from multiple experts in a pool-based setting.
Revised: IALE learns an AL sampling strategy for similar tasks from multiple experts in a pool-based setting.

Original: We train a policy with data consisting of states (i.e., that includes an encoding of the labeled data samples) and best expert actions (i.e., samples selected for labeling) collected over the AL cycles.
Revised: We train a policy with data consisting of states (i.e., that includes an encoding of the labeled data samples) and best expert actions (i.e., samples selected for labeling) collected over the AL cycles.

Original: The policy is then used on a similar (but different) task.
Revised: The policy is then used on a similar (but different) task.

Original: To see states that are unlikely to be produced by the experts, DA GGER (Ross et al., 2011) collects a large set of states and actions over AL iterations.
Revised: To see states that are unlikely to be produced by the experts, DA GGER (Ross et al., 2011) collects a large set of states and actions over AL iterations.

Original: The policy network is trained on all the previous states and actions after each iteration.
Revised: The policy network is trained on all the previous states and actions after each iteration.

Original: 3. B ACKGROUND
Revised: 3. B ACKGROUND

Original: In pool-based AL we train a model M on a dataset D by iteratively labeling data samples.
Revised: In pool-based AL we train a model M on a dataset D by iteratively labeling data samples.

Original: Initially,
Revised: Initially,

Original: M is trained on a small amount of labeled data D lab randomly sampled from the dataset.
Revised: M is trained on a small amount of labeled data D lab randomly sampled from the dataset.

Original: The rest of the data is considered as the unlabeled data pool D pool , i.e., D = D lab ∪ D pool .
Revised: The rest of the data is considered as the unlabeled data pool D pool , i.e., D = D lab ∪ D pool .

Original: From that point onwards during the AL iterations a subset of D sel is selected from D pool by using an acquisition function a ( M, D pool ) .
Revised: From that point onwards during the AL iterations a subset of D sel is selected from D pool by using an acquisition function a ( M, D pool ) .

Original: The data is labeled and then removed from D pool and added to D lab .
Revised: The data is labeled and then removed from D pool and added to D lab .

Original: The size of D sel is based on the acquisition size acq ( > 1 for batch-mode AL).
Revised: The size of D sel is based on the acquisition size acq ( > 1 for batch-mode AL).

Original: The AL cycle continues until a labeling budget of B is reached.
Revised: The AL cycle continues until a labeling budget of B is reached.

Original: M is retrained after each acquisition to evaluate the performance boost with respect to the increased labeled dataset only (and not the additional training time).
Revised: M is retrained after each acquisition to evaluate the performance boost with respect to the increased labeled dataset only (and not the additional training time).

Original: The acquisition function a is a heuristic that uses the trained model M to decide which of the data samples in D pool are most informative.
Revised: The acquisition function a is a heuristic that uses the trained model M to decide which of the data samples in D pool are most informative.

Original: For deep AL popular heuristics include uncertainty-based
Revised: For deep AL popular heuristics include uncertainty-based

Original: MC-Dropout (Gal et al., 2017), query-by-committee-based Ensembles (Beluch et al., 2018), data diversity-based CoreSet (Sener & Savarese, 2018), gradient-based BADGE (Ash et al., 2020) and soft-max-based Confidence - or Entropy -sampling (Wang & Shang, 2014).
Revised: MC-Dropout (Gal et al., 2017), query-by-committee-based Ensembles (Beluch et al., 2018), data diversity-based CoreSet (Sener & Savarese, 2018), gradient-based BADGE (Ash et al., 2020) and soft-max-based Confidence - or Entropy -sampling (Wang & Shang, 2014).

Original: MC-Dropout uses a Monte-Carlo inference scheme based on a dropout layer to approximate the model’s predictive uncertainty (Gal & Ghahramani, 2016).
Revised: MC-Dropout uses a Monte-Carlo inference scheme based on a dropout layer to approximate the model’s predictive uncertainty (Gal & Ghahramani, 2016).

Original: The heuristic (Gal et al., 2017) then uses these values to select the most uncertain samples.
Revised: The heuristic (Gal et al., 2017) then uses these values to select the most uncertain samples.

Original: Ensembles (Beluch et al., 2018) model predictive uncertainty using a committee of N classiﬁers initialized with different random seeds.
Revised: Ensembles (Beluch et al., 2018) model predictive uncertainty using a committee of N classiﬁers initialized with different random seeds.

Original: However, while at inference time we need to run only N forward-passes per sample (compared to
Revised: However, while at inference time we need to run only N forward-passes per sample (compared to

Original: MC-Dropout performing two dozen or more Monte-Carlo passes), the training of N − 1 additional deep models can become prohibitively expensive in many use-cases.
Revised: MC-Dropout performing two dozen or more Monte-Carlo passes), the training of N − 1 additional deep models can become prohibitively expensive in many use-cases.

Original: CoreSet (Sener & Savarese, 2018) aims to select diverse samples by solving the k -centers problem on the classiﬁer’s embeddings.
Revised: CoreSet (Sener & Savarese, 2018) aims to select diverse samples by solving the k -centers problem on the classiﬁer’s embeddings.

Original: This involves minimizing the distance between each of the unlabeled data samples to its nearest labeled samples.
Revised: This involves minimizing the distance between each of the unlabeled data samples to its nearest labeled samples.

Original: BADGE uses (pseudo labels of) the magnitudes of the gradients in a batch to select samples by uncertainty, and the gradient directions together with a k-means++ clustering to select samples by diversity.
Revised: BADGE uses (pseudo labels of) the magnitudes of the gradients in a batch to select samples by uncertainty, and the gradient directions together with a k-means++ clustering to select samples by diversity.

Original: Soft-max-based heuristics ( Confidence - and Entropy -sampling) use predictive uncertainty and are computationally lightweight at lower AL performance (Gal &
Revised: Soft-max-based heuristics ( Confidence - and Entropy -sampling) use predictive uncertainty and are computationally lightweight at lower AL performance (Gal &

Original: Ghahramani, 2016; Ash et al., 2020) ( Confidence selects the samples with the lowest class probability and Entropy the ones with with largest entropy of their probability distribution).
Revised: Ghahramani, 2016; Ash et al., 2020) ( Confidence selects the samples with the lowest class probability and Entropy the ones with with largest entropy of their probability distribution).

Original: 3. L EARNING M ULTIPLE E XPERTS
Revised: 3. L EARNING M ULTIPLE E XPERTS

Original: Instead of using speciﬁc heuristics we propose to learn the acquisition function using a policy network.
Revised: Instead of using speciﬁc heuristics we propose to learn the acquisition function using a policy network.

Original: Once the policy is trained on a source dataset it can be applied to different target datasets.
Revised: Once the policy is trained on a source dataset it can be applied to different target datasets.

Original: Figure 1 sketches the idea.
Revised: Figure 1 sketches the idea.

Original: The policy network π is a Multi-Layer Perceptron (MLP) trained to predict the usefulness of labeling samples from the unlabeled data pool D pool for training the model M , similar to an AL acquisition function.
Revised: The policy network π is a Multi-Layer Perceptron (MLP) trained to predict the usefulness of labeling samples from the unlabeled data pool D pool for training the model M , similar to an AL acquisition function.

Original: As input the policy network takes the current state , consisting
Revised: As input the policy network takes the current state , consisting

Original: [Figure]
Revised: [Figure]

Original: of the model M ’s embeddings of pool data next to other elements, extending similar work (Contardo et al., 

Revised: of the model M ’s embeddings of pool data next to other elements, extending similar work (Contardo et al.,

Original: ; Konyushkova et al., 2017; Liu et al., 2018), see below.
Revised: ; Konyushkova et al., 2017; Liu et al., 2018), see below.

Original: π then outputs the action to be taken at that step.
Revised: π then outputs the action to be taken at that step.

Original: Action here refers to an AL acquisition, i.e., which of the unlabeled data samples should be labeled and added to the training data.
Revised: Action here refers to an AL acquisition, i.e., which of the unlabeled data samples should be labeled and added to the training data.

Original: π learns the best actions from a set of experts E which predict the best actions for a given AL state.
Revised: π learns the best actions from a set of experts E which predict the best actions for a given AL state.

Original: A subset of the pool dataset D sub with size n is used instead of the whole pool dataset at each active learning iteration for training the policy.
Revised: A subset of the pool dataset D sub with size n is used instead of the whole pool dataset at each active learning iteration for training the policy.

Original: States.
Revised: States.

Original: As π uses the state information to make decisions, a state s should be maximally compact but still unique, i.e., different situations should have different state encodings.
Revised: As π uses the state information to make decisions, a state s should be maximally compact but still unique, i.e., different situations should have different state encodings.

Original: Our state encoding uses two types of information: (1) model-dependent parameters (that describe the state from the perspective of the model M and the already labeled samples D lab ), and (2) AL-cycle-dependent parameters (that describe the elements of the samples D sub that we can choose to label).
Revised: Our state encoding uses two types of information: (1) model-dependent parameters (that describe the state from the perspective of the model M and the already labeled samples D lab ), and (2) AL-cycle-dependent parameters (that describe the elements of the samples D sub that we can choose to label).

Original: Together, these parameters form a minimal description of a current state.
Revised: Together, these parameters form a minimal description of a current state.

Original: We use the following parameters to describe the model-dependent aspects:
Revised: We use the following parameters to describe the model-dependent aspects:

Original: • The mean of the already labeled data samples µ (cid:0) M e ( D lab ) (cid:1) : the embedding M e of a sample by M is the output of the ﬁnal layer (i.e., the layer before the soft-max layer in case of a classiﬁcation model), see Figure 1.
Revised: • The mean of the already labeled data samples µ (cid:0) M e ( D lab ) (cid:1) : the embedding M e of a sample by M is the output of the ﬁnal layer (i.e., the layer before the soft-max layer in case of a classiﬁcation model), see Figure 1.

Original: The size of this representation is independent of the (growing) size of D lab and thus will not become a computational bottle-neck.
Revised: The size of this representation is independent of the (growing) size of D lab and thus will not become a computational bottle-neck.

Original: • The ground-truth empirical distribution of class labels
Revised: • The ground-truth empirical distribution of class labels

Original: [Equation]
Revised: [Equation]

Original: which is a normalized vector of length
Revised: which is a normalized vector of length

Original: i , i.e., the number of classes, with percentage of occurrence per class using the labels of the already acquired data samples.
Revised: i , i.e., the number of classes, with percentage of occurrence per class using the labels of the already acquired data samples.

Original: • M ’s predicted empirical distribution of class labels for the labeled data (cid:126)e M ( D lab )
Revised: • M ’s predicted empirical distribution of class labels for the labeled data (cid:126)e M ( D lab )

Original: (i.e., a normalized vector as above but with predicted class labels instead of ground-truth).
Revised: (i.e., a normalized vector as above but with predicted class labels instead of ground-truth).

Original: The rationale for including both the ground truth and the predicted empirical distribution is to enable the policy to base its decisions on the model M ’s prediction errors.
Revised: The rationale for including both the ground truth and the predicted empirical distribution is to enable the policy to base its decisions on the model M ’s prediction errors.

Original: In other words, when the model makes mistakes on already labeled samples for a class (that were part of the model’s training)
Revised: In other words, when the model makes mistakes on already labeled samples for a class (that were part of the model’s training)

Original: it likely needs more samples from those classes to correct the erroneous predictions.
Revised: it likely needs more samples from those classes to correct the erroneous predictions.

Original: The AL-cycle-dependent parameters describe the n data samples in D sub that we evaluate in the current iteration.
Revised: The AL-cycle-dependent parameters describe the n data samples in D sub that we evaluate in the current iteration.

Original: For each data sample x i ∈ D
Revised: For each data sample x i ∈ D

Original: sub we calculate M ’s embedding M e ( x i ) in the same embedding space as the already labeled samples of our model-dependent parameters.
Revised: sub we calculate M ’s embedding M e ( x i ) in the same embedding space as the already labeled samples of our model-dependent parameters.

Original: We also predict each sample’s label M ( x i ) .
Revised: We also predict each sample’s label M ( x i ) .

Original: Similarly, we capture the expected model change via per-sample gradient information g ( M e ( x i )) in the embedding space for D sub , using the method proposed in (Ash et al., 2020).
Revised: Similarly, we capture the expected model change via per-sample gradient information g ( M e ( x i )) in the embedding space for D sub , using the method proposed in (Ash et al., 2020).

Original: Hence, we consider the gradients of the loss at the embedding layer, given unlabeled samples with proxy labels, both as predictive of model uncertainty and expected model change.
Revised: Hence, we consider the gradients of the loss at the embedding layer, given unlabeled samples with proxy labels, both as predictive of model uncertainty and expected model change.

Original: The proxy label ˆ y is the most likely prediction M ( x i ) (determined by an argmax operation in the softmax layer).
Revised: The proxy label ˆ y is the most likely prediction M ( x i ) (determined by an argmax operation in the softmax layer).

Original: The magnitude of the loss gradient at the embedding layer then describes both the model’s uncertainty and its expected change.
Revised: The magnitude of the loss gradient at the embedding layer then describes both the model’s uncertainty and its expected change.

Original: This information enables the policy to learn to select samples (1) where the model is uncertain (i.e., where it predicts the wrong labels), (2) where the model might gain most information (i.e., the loss is high), and (3) to learn to select more diverse samples from less well represented classes using the label statistics.
Revised: This information enables the policy to learn to select samples (1) where the model is uncertain (i.e., where it predicts the wrong labels), (2) where the model might gain most information (i.e., the loss is high), and (3) to learn to select more diverse samples from less well represented classes using the label statistics.

Original: Hence, we describe a state s as follows:
Revised: Hence, we describe a state s as follows:

Original: [Equation]
Revised: [Equation]

Original: Actions.
Revised: Actions.

Original: The action of the MLP is a desirability score
Revised: The action of the MLP is a desirability score

Original: ρ i for each unlabeled sample from D sub , i.e., ρ i :=
Revised: ρ i for each unlabeled sample from D sub , i.e., ρ i :=

Original: π ( s i ) .
Revised: π ( s i ) .

Original: We choose the samples to be labeled based on this score, i.e., we choose the top- k ranked values, where k = acq , resulting in a binary selection vector (cid:126)v = top-k
Revised: We choose the samples to be labeled based on this score, i.e., we choose the top- k ranked values, where k = acq , resulting in a binary selection vector (cid:126)v = top-k

Original: ( ρ 0 , · · · , ρ n ) with (cid:80) ni =

Revised: ( ρ 0 , · · · , ρ n ) with (cid:80) ni =

Original: (cid:126)v i =
Revised: (cid:126)v i =

Original: acq .
Revised: acq .

Original: The ground truth actions provided by the experts are binary vectors of length k , where a 1 at index i means that x i should be selected for labeling (and indices of those sample that should not be labeled are 0).
Revised: The ground truth actions provided by the experts are binary vectors of length k , where a 1 at index i means that x i should be selected for labeling (and indices of those sample that should not be labeled are 0).

Original: From here we can use a binary cross entropy loss to update π ’s weights:
Revised: From here we can use a binary cross entropy loss to update π ’s weights:

Original: [Equation]
Revised: [Equation]

Original: where (cid:126)t is the target vector provided by the best expert (similar to a greedy multi-armed bandit approach (Hsu & Lin, 2015)).
Revised: where (cid:126)t is the target vector provided by the best expert (similar to a greedy multi-armed bandit approach (Hsu & Lin, 2015)).

Original: This brings π ’s output closer to the suggestion of the best expert.
Revised: This brings π ’s output closer to the suggestion of the best expert.

Original: Our IL-based approach uses the experts to turn AL into a supervised learning problem, i.e., the action of the best expert becomes the label for the current state s .
Revised: Our IL-based approach uses the experts to turn AL into a supervised learning problem, i.e., the action of the best expert becomes the label for the current state s .

Original: Our choice of AL heuristics for the set of experts E includes particular types but is arbitrarily extendable.
Revised: Our choice of AL heuristics for the set of experts E includes particular types but is arbitrarily extendable.

Original: Using MC-Dropout ,
Revised: Using MC-Dropout ,

Original: Ensemble , CoreSet , BADGE , Confidence or Entropy allows us to only minimally modify the classiﬁer model M .
Revised: Ensemble , CoreSet , BADGE , Confidence or Entropy allows us to only minimally modify the classiﬁer model M .

Original: π aims to learn certain derived properties from the state , such as model uncertainty or similarity.
Revised: π aims to learn certain derived properties from the state , such as model uncertainty or similarity.

Original: These measures are based on M ’s predictions and embeddings.
Revised: These measures are based on M ’s predictions and embeddings.

Original: Our hypothesis is that π learns to imitate the best suitable heuristic for each phase of the AL cycle, i.e., starting with relying on one type of heuristics for selections of samples in the beginning and later switch to a ﬁne-tuning using a different one (see also Section 4.3).
Revised: Our hypothesis is that π learns to imitate the best suitable heuristic for each phase of the AL cycle, i.e., starting with relying on one type of heuristics for selections of samples in the beginning and later switch to a ﬁne-tuning using a different one (see also Section 4.3).

Original: This is in line with previous research that combines uncertainty- and density-based heuristics and that learns an adaptive combination framework that weights them over the training course (Li & Guo, 2013).
Revised: This is in line with previous research that combines uncertainty- and density-based heuristics and that learns an adaptive combination framework that weights them over the training course (Li & Guo, 2013).

Original: 3. P OLICY T RAINING
Revised: 3. P OLICY T RAINING

Original: Algorithm 1:
Revised: Algorithm 1:

Original: 1 Input: data D , labeled validation data D val , classiﬁer
Revised: 1 Input: data D , labeled validation data D val , classiﬁer

Original: M , budget B , experts E , acquisition size acq , subset size n , probability p , states S , actions
Revised: M , budget B , experts E , acquisition size acq , subset size n , probability p , states S , actions

Original: A , random policy π ( acq ≥ 1 , n = 100 ).
Revised: A , random policy π ( acq ≥ 1 , n = 100 ).

Original: 2 for e = 1 . . .
Revised: 2 for e = 1 . . .

Original: episodes max do
Revised: episodes max do

Original: D lab , D pool ← split( D ) while |D lab | <
Revised: D lab , D pool ← split( D ) while |D lab | <

Original: B do
Revised: B do

Original: M ← initAndTrain ( M, D lab )
Revised: M ← initAndTrain ( M, D lab )

Original: D sub ← sample ( D pool , n ) e ∗ ← bestExpert ( E , M, D sub , D val )
Revised: D sub ← sample ( D pool , n ) e ∗ ← bestExpert ( E , M, D sub , D val )

Original: D sel ← e ∗ .
Revised: D sel ← e ∗ .

Original: SelectQuery ( M, D sub , acq )
Revised: SelectQuery ( M, D sub , acq )

Original: S , A ← toState ( D sub , D lab ) , toAction ( D sel ) if Rnd (0 , 1) ≥ p then //
Revised: S , A ← toState ( D sub , D lab ) , toAction ( D sel ) if Rnd (0 , 1) ≥ p then //

Original: We may choose π ’s selection
Revised: We may choose π ’s selection

Original: D sel ← π.
Revised: D sel ← π.

Original: SelectQuery ( M, D sub , acq )
Revised: SelectQuery ( M, D sub , acq )

Original: D lab ← D lab ∪ D sel
Revised: D lab ← D lab ∪ D sel

Original: D pool ← D pool \ D sel
Revised: D pool ← D pool \ D sel

Original: Update policy using {S , A}
Revised: Update policy using {S , A}

Original: Our policy training builds on the intuition behind DA GGER , which is a well-known algorithm for IL that aims to train a policy by iteratively growing a dataset for supervised learning.
Revised: Our policy training builds on the intuition behind DA GGER , which is a well-known algorithm for IL that aims to train a policy by iteratively growing a dataset for supervised learning.

Original: The key idea is that the dataset includes the states that are likely to be visited over the course of solving a problem (in other words, those state and action encodings that would have been visited if we would follow a hardcoded AL strategy).
Revised: The key idea is that the dataset includes the states that are likely to be visited over the course of solving a problem (in other words, those state and action encodings that would have been visited if we would follow a hardcoded AL strategy).

Original: To this end, it is common when using DA GGER to determine a policy’s next state by either following the current policy or an available expert (Ross et al., 2011).
Revised: To this end, it is common when using DA GGER to determine a policy’s next state by either following the current policy or an available expert (Ross et al., 2011).

Original: We thus grow a list of state and action pairs, and randomly either choose expert or policy selections as the action.
Revised: We thus grow a list of state and action pairs, and randomly either choose expert or policy selections as the action.

Original: Each episode of the IL cycle lasts until the AL labeling budget is reached for episodes max iterations.
Revised: Each episode of the IL cycle lasts until the AL labeling budget is reached for episodes max iterations.

Original: We aggregate the states and actions over all episodes, and continually train the policy on the pairs.
Revised: We aggregate the states and actions over all episodes, and continually train the policy on the pairs.

Original: [Figure]
Revised: [Figure]

Original: na ya ha
Revised: na ya ha

Original: We use DA GGER to further randomize the exploration of D .
Revised: We use DA GGER to further randomize the exploration of D .

Original: Instead of always following the best expert’s advice, we randomly follow the policy’s prediction, and thus enrich the possible states.
Revised: Instead of always following the best expert’s advice, we randomly follow the policy’s prediction, and thus enrich the possible states.

Original: Our IL approach for training π is given in Algorithm 1.
Revised: Our IL approach for training π is given in Algorithm 1.

Original: At each AL cycle, we randomly sample a subset D sub of n = 100 samples from the unlabeled pool D pool (line 3).
Revised: At each AL cycle, we randomly sample a subset D sub of n = 100 samples from the unlabeled pool D pool (line 3).

Original: We ﬁnd the best expert e ∗ from a set of experts E (line 8) by extending the training dataset by the expert selections (from
Revised: We ﬁnd the best expert e ∗ from a set of experts E (line 8) by extending the training dataset by the expert selections (from

Original: D sub ) and train a classiﬁer each.
Revised: D sub ) and train a classiﬁer each.

Original: This means that each expert constructs one batch according to its heuristic, e.g., a batch composition could maximize model-change, and queries the oracle for labels.
Revised: This means that each expert constructs one batch according to its heuristic, e.g., a batch composition could maximize model-change, and queries the oracle for labels.

Original: We choose the best expert by comparing the resulting classiﬁers’ accuracies on the labeled validation dataset.
Revised: We choose the best expert by comparing the resulting classiﬁers’ accuracies on the labeled validation dataset.

Original: We next set its acquisition as this iteration’s chosen target and store state and action for the policy training (line 10).
Revised: We next set its acquisition as this iteration’s chosen target and store state and action for the policy training (line 10).

Original: According to DA GGER we then ﬂip a coin and depending on the probability p (line 11) either use the policy or the best expert to increase D lab for the next iteration (line 14).
Revised: According to DA GGER we then ﬂip a coin and depending on the probability p (line 11) either use the policy or the best expert to increase D lab for the next iteration (line 14).

Original: After each episode we retrain π on the state and action pairs (line 16).
Revised: After each episode we retrain π on the state and action pairs (line 16).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We ﬁrst describe our experimental setup (Section 4.1).
Revised: We ﬁrst describe our experimental setup (Section 4.1).

Original: Next, we describe how we trained our policy (Section 4.2) and evaluate our approach by transferring it to test datasets, i.e., to FMNIST and
Revised: Next, we describe how we trained our policy (Section 4.2) and evaluate our approach by transferring it to test datasets, i.e., to FMNIST and

Original: KMNIST (Section 4.3).
Revised: KMNIST (Section 4.3).

Original: Finally, we end with a discussion of our ablation studies and the limitations of our approach (Section 4.4).
Revised: Finally, we end with a discussion of our ablation studies and the limitations of our approach (Section 4.4).

Original: The source code is publicly at https://github.com/ crispchris/iale and can be used to reproduce our experimental results.
Revised: The source code is publicly at https://github.com/ crispchris/iale and can be used to reproduce our experimental results.

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTAL S ETUP

Original: Datasets.
Revised: Datasets.

Original: We use the image classiﬁcation datasets MNIST (LeCun et al., 1998), Fashion-MNIST (FMNIST) (Xiao et al., 2017), and Kuzushiji-MNIST (KMNIST) (Clanuwat et al., 2018) for our evaluation.
Revised: We use the image classiﬁcation datasets MNIST (LeCun et al., 1998), Fashion-MNIST (FMNIST) (Xiao et al., 2017), and Kuzushiji-MNIST (KMNIST) (Clanuwat et al., 2018) for our evaluation.

Original: They all consist of 70 , 000 grey-scale images (28 × 28px) in total for 10 classes.
Revised: They all consist of 70 , 000 grey-scale images (28 × 28px) in total for 10 classes.

Original: MNIST contains the handwritten digits 0 − 9 , FMNIST contains images of clothing (i.e., bags, shoes, etc.), and KMNIST consists of Hiragana characters, see Figure 2.
Revised: MNIST contains the handwritten digits 0 − 9 , FMNIST contains images of clothing (i.e., bags, shoes, etc.), and KMNIST consists of Hiragana characters, see Figure 2.

Original: To evaluate IALE we train a policy π and run it (3 repetitions) on unseen datasets along with the baselines.
Revised: To evaluate IALE we train a policy π and run it (3 repetitions) on unseen datasets along with the baselines.

Original: The similarity between FMNIST and MNIST (that has previously been shown (Nalisnick et al., 2019)) and the difﬁculty of FMNIST (it has been shown to be a demanding dataset for
Revised: The similarity between FMNIST and MNIST (that has previously been shown (Nalisnick et al., 2019)) and the difﬁculty of FMNIST (it has been shown to be a demanding dataset for

Original: AL methods (Hahn et al., 2019b)) make these datasets a perfect combination to evaluate IALE .
Revised: AL methods (Hahn et al., 2019b)) make these datasets a perfect combination to evaluate IALE .

Original: Please ﬁnd more results for transfering π ’s AL strategy learned from MNIST to CIFAR-10 in Appendix A.3.3.
Revised: Please ﬁnd more results for transfering π ’s AL strategy learned from MNIST to CIFAR-10 in Appendix A.3.3.

Original: Architectures of classiﬁer M .
Revised: Architectures of classiﬁer M .

Original: We use the same model used to evaluate AL on MNIST data that has been used in previous research (Gal & Ghahramani, 2016).
Revised: We use the same model used to evaluate AL on MNIST data that has been used in previous research (Gal & Ghahramani, 2016).

Original: Our model has two convolutional layers, followed by a max pooling and dense layer.
Revised: Our model has two convolutional layers, followed by a max pooling and dense layer.

Original: We add dropout layers after the convolution and dense layers and use ReLU activations.
Revised: We add dropout layers after the convolution and dense layers and use ReLU activations.

Original: A softmax layer allows for classiﬁcation.
Revised: A softmax layer allows for classiﬁcation.

Original: We also provide additional results for all the methods with a Resnet-18 (He et al., 2016) and with a two-layer MLP with a dense layer ( 256 neurons) followed by a soft-max layer (Ash et al., 2020) in Appendix A.3.2.
Revised: We also provide additional results for all the methods with a Resnet-18 (He et al., 2016) and with a two-layer MLP with a dense layer ( 256 neurons) followed by a soft-max layer (Ash et al., 2020) in Appendix A.3.2.

Original: Architecture of policy π .
Revised: Architecture of policy π .

Original: Our policy model π uses an MLP with three dense layers withneurons each.
Revised: Our policy model π uses an MLP with three dense layers withneurons each.

Original: The ﬁrst two dense layers are followed by a ReLU activation layer, whereas the ﬁnal layer has only one neuron and the output of this layer is passed onto a sigmoid function to constrain the outputs to the range [0 , 1] and to further process it into an aggregating top-k operation.
Revised: The ﬁrst two dense layers are followed by a ReLU activation layer, whereas the ﬁnal layer has only one neuron and the output of this layer is passed onto a sigmoid function to constrain the outputs to the range [0 , 1] and to further process it into an aggregating top-k operation.

Original: Baselines.
Revised: Baselines.

Original: For the evaluation of the performance of our AL method, we implemented different well-known AL approaches from literature: Random Sampling , MC-Dropout , Ensemble ,
Revised: For the evaluation of the performance of our AL method, we implemented different well-known AL approaches from literature: Random Sampling , MC-Dropout , Ensemble ,

Original: CoreSet , BADGE , Confidence -sampling, Entropy -sampling and ALIL (which we adapted
Revised: CoreSet , BADGE , Confidence -sampling, Entropy -sampling and ALIL (which we adapted

Original: [Figure]
Revised: [Figure]

Original: to work on image classiﬁcation tasks).
Revised: to work on image classiﬁcation tasks).

Original: For a more detailed description of the baselines please see
Revised: For a more detailed description of the baselines please see

Original: Appendix A.1.1.
Revised: Appendix A.1.1.

Original: 4. P OLICY T RAINING
Revised: 4. P OLICY T RAINING

Original: We use the MNIST dataset as our source dataset on which we train our policy for 100 episodes, with each episode containing data from an AL cycle.
Revised: We use the MNIST dataset as our source dataset on which we train our policy for 100 episodes, with each episode containing data from an AL cycle.

Original: The initial amount of labeled training data issamples (class-balanced).
Revised: The initial amount of labeled training data issamples (class-balanced).

Original: At each step of the active learning process, 10 samples are labeled and added to the training data until a labeling budget B of 1 , 000 is reached.
Revised: At each step of the active learning process, 10 samples are labeled and added to the training data until a labeling budget B of 1 , 000 is reached.

Original: We use the AL heuristics
Revised: We use the AL heuristics

Original: MC-Dropout , Ensemble , CoreSet , BADGE , Confidence and Entropy as experts, and use
Revised: MC-Dropout , Ensemble , CoreSet , BADGE , Confidence and Entropy as experts, and use

Original: D val with 100 labeled samples to score the acquisitions of the experts.
Revised: D val with 100 labeled samples to score the acquisitions of the experts.

Original: The pool dataset is sampled with K = 100 at each AL iteration.
Revised: The pool dataset is sampled with K = 100 at each AL iteration.

Original: We choose p = 0 .
Revised: We choose p = 0 .

Original: 5 for DA GGER for means of comparison with the baselines (based on preliminary experiments (see Appendix A.2.1 on Exploration-Exploitation ).
Revised: 5 for DA GGER for means of comparison with the baselines (based on preliminary experiments (see Appendix A.2.1 on Exploration-Exploitation ).

Original: We train the policy’s MLP on the current and growing list of state and action pairs using the binary cross entropy loss from Equation 2 and use the Adam optimizer (Kingma & Ba, 2015) for 30 epochs with a learning rate of 1 e − 3 , β 1 = 0 .
Revised: We train the policy’s MLP on the current and growing list of state and action pairs using the binary cross entropy loss from Equation 2 and use the Adam optimizer (Kingma & Ba, 2015) for 30 epochs with a learning rate of 1 e − 3 , β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , (cid:15) = 1 e − 8 , and without any weight decay.
Revised: 999 , (cid:15) = 1 e − 8 , and without any weight decay.

Original: Figure 3a shows the results of our method in comparison to all the baseline approaches on the
Revised: Figure 3a shows the results of our method in comparison to all the baseline approaches on the

Original: MNIST dataset, on which the policy was trained on.
Revised: MNIST dataset, on which the policy was trained on.

Original: Our method consistently outperforms or is at least en par (towards the end, when enough representatives samples are labeled) with all the other methods.
Revised: Our method consistently outperforms or is at least en par (towards the end, when enough representatives samples are labeled) with all the other methods.

Original: IALE performs better acquisitions than Ensemble and MC-Dropout for the important ﬁrst half of the labeling budget, where it matters the most.
Revised: IALE performs better acquisitions than Ensemble and MC-Dropout for the important ﬁrst half of the labeling budget, where it matters the most.

Original: Moreover, IALE is faster (see
Revised: Moreover, IALE is faster (see

Original: Appendix A.3.3).
Revised: Appendix A.3.3).

Original: While MC-Dropout requires 20 forward passes to decide which samples it acquires, and Ensembles N = 5 forward passes, one for each model, our approach requires only 2 inferences for only n = 100 samples plus the labeled pool.
Revised: While MC-Dropout requires 20 forward passes to decide which samples it acquires, and Ensembles N = 5 forward passes, one for each model, our approach requires only 2 inferences for only n = 100 samples plus the labeled pool.

Original: Confidence -sampling performs similar as the two more complex methods, even though it uses only the simple soft-max probabilities.
Revised: Confidence -sampling performs similar as the two more complex methods, even though it uses only the simple soft-max probabilities.

Original: While Entropy beats random it is still not competitive.
Revised: While Entropy beats random it is still not competitive.

Original: BADGE performs similar to random sampling, which is due to the small acquisition size of 10 (the better performance of BADGE was reported with much larger acquisition sizes of 100 to 10,000 in Ash et al.
Revised: BADGE performs similar to random sampling, which is due to the small acquisition size of 10 (the better performance of BADGE was reported with much larger acquisition sizes of 100 to 10,000 in Ash et al.

Original: (2020) as its mix of uncertainty and diversity heuristic beneﬁts from these).
Revised: (2020) as its mix of uncertainty and diversity heuristic beneﬁts from these).

Original: The same applies to CoreSet , however, here it performs worst on average over all experiments.
Revised: The same applies to CoreSet , however, here it performs worst on average over all experiments.

Original: This ﬁnding is in line with previous research (Sinha et al., 2019; Hahn et al., 2019a) and can be attributed to a weakness of the utilized p -norm distance metric regarding high-dimensional data, called the distance concentration phenomenon .
Revised: This ﬁnding is in line with previous research (Sinha et al., 2019; Hahn et al., 2019a) and can be attributed to a weakness of the utilized p -norm distance metric regarding high-dimensional data, called the distance concentration phenomenon .

Original: The accuracy of ALIL on MNIST is similar to CoreSet , however, ALIL is designed to add only one sample to the training data at a time (no batch-mode).
Revised: The accuracy of ALIL on MNIST is similar to CoreSet , however, ALIL is designed to add only one sample to the training data at a time (no batch-mode).

Original: Construction of acquisition.
Revised: Construction of acquisition.

Original: We compare IALE’s chosen samples with the ones chosen by the baselines, see Figure 4.
Revised: We compare IALE’s chosen samples with the ones chosen by the baselines, see Figure 4.

Original: We show this overlap in relation to the baselines in percent, and plot the values over the 100 AL cycles using the MNIST dataset (more results can be found in Appendix A.2.2).
Revised: We show this overlap in relation to the baselines in percent, and plot the values over the 100 AL cycles using the MNIST dataset (more results can be found in Appendix A.2.2).

Original: We plot second-order polynomials, ﬁt to the percentages (given as dots) over 100 acquisitions of size 10 .
Revised: We plot second-order polynomials, ﬁt to the percentages (given as dots) over 100 acquisitions of size 10 .

Original: π mostly imitates uncertainty-based heuristics, i.e., soft-max heuristics and MC-Dropout , and the uncertainty-/diversity-heuristic BADGE (close behind).
Revised: π mostly imitates uncertainty-based heuristics, i.e., soft-max heuristics and MC-Dropout , and the uncertainty-/diversity-heuristic BADGE (close behind).

Original: Interestingly, Ensemble is overlapping mostly at the beginning.
Revised: Interestingly, Ensemble is overlapping mostly at the beginning.

Original: CoreSet has the lowest overlap.
Revised: CoreSet has the lowest overlap.

Original: Note that IALE ’s acquisitions are build
Revised: Note that IALE ’s acquisitions are build

Original: [Figure]
Revised: [Figure]

Original: from combinations of the heuristics (instead of single votes).
Revised: from combinations of the heuristics (instead of single votes).

Original: The percentages do not sum up to 1 as the experts are independent and may also overlap with each other.
Revised: The percentages do not sum up to 1 as the experts are independent and may also overlap with each other.

Original: 4. P OLICY T RANSFER
Revised: 4. P OLICY T RANSFER

Original: [Figure]
Revised: [Figure]

Original: We investigate how π works on a different dataset than the one that it has been trained on.
Revised: We investigate how π works on a different dataset than the one that it has been trained on.

Original: Hence, we train π on the source dataset MNIST as in Section 4.2 and use it for the AL problem on FMNIST and KMNIST .
Revised: Hence, we train π on the source dataset MNIST as in Section 4.2 and use it for the AL problem on FMNIST and KMNIST .

Original: For the AL we again use an initial class-balanced labeled training dataset of 20 samples and add 10 samples per
Revised: For the AL we again use an initial class-balanced labeled training dataset of 20 samples and add 10 samples per

Original: AL acquisition cycle until we reach a labeling budget of 1,000 samples.
Revised: AL acquisition cycle until we reach a labeling budget of 1,000 samples.

Original: All the baselines are evaluated along with our method for comparison.
Revised: All the baselines are evaluated along with our method for comparison.

Original: Figures 3b and 3c show the performance of IALE along with the baselines on FMNIST and KMNIST .
Revised: Figures 3b and 3c show the performance of IALE along with the baselines on FMNIST and KMNIST .

Original: IALE consistently outperforms the baselines on both datasets.
Revised: IALE consistently outperforms the baselines on both datasets.

Original: We can see that it learns a combined and improved policy that outperforms the individual experts consistently and (sometimes) even with large margins.
Revised: We can see that it learns a combined and improved policy that outperforms the individual experts consistently and (sometimes) even with large margins.

Original: On
Revised: On

Original: FMNIST IALE is the only method that actually beats Random Sampling (similar ﬁndings have previously been reported by Hahn et al.
Revised: FMNIST IALE is the only method that actually beats Random Sampling (similar ﬁndings have previously been reported by Hahn et al.

Original: (2019b)).
Revised: (2019b)).

Original: IALE is consistently 1 − 3% better than Random
Revised: IALE is consistently 1 − 3% better than Random

Original: Sampling on FMNIST , on the harder KMNIST dataset IALE is even 7 − 9% ahead.
Revised: Sampling on FMNIST , on the harder KMNIST dataset IALE is even 7 − 9% ahead.

Original: The baselines give a mixed picture.
Revised: The baselines give a mixed picture.

Original: ALIL does not achieve competitive performance on any task and actually never beats a random sampling strategy.
Revised: ALIL does not achieve competitive performance on any task and actually never beats a random sampling strategy.

Original: We also see unstable performance for MC-Dropout and
Revised: We also see unstable performance for MC-Dropout and

Original: Ensemble , that generally perform similarly well.
Revised: Ensemble , that generally perform similarly well.

Original: The simple soft-max heuristics Entropy and
Revised: The simple soft-max heuristics Entropy and

Original: Confidence fail on FMNIST .
Revised: Confidence fail on FMNIST .

Original: CoreSet lags far behind, especially on KMNIST .
Revised: CoreSet lags far behind, especially on KMNIST .

Original: BADGE always performs like random sampling, due to the aforementioned problematic acquisition size.
Revised: BADGE always performs like random sampling, due to the aforementioned problematic acquisition size.

Original: Please ﬁnd additional experiments in Appendix A.3 including a wider range of classiﬁer models (MLP, CNN, and Resnet-18) and datasets ( CIFAR-10) .
Revised: Please ﬁnd additional experiments in Appendix A.3 including a wider range of classiﬁer models (MLP, CNN, and Resnet-18) and datasets ( CIFAR-10) .

Original: 4. A BLATION S TUDIES
Revised: 4. A BLATION S TUDIES

Original: Hyperparameters.
Revised: Hyperparameters.

Original: Two important parameters are the acquisition size acq and the size of D sub .
Revised: Two important parameters are the acquisition size acq and the size of D sub .

Original: Figures 5a and 5b show results for acq of 1 to 40 and size of D sub between n = 10 and n = 10 , 000 .
Revised: Figures 5a and 5b show results for acq of 1 to 40 and size of D sub between n = 10 and n = 10 , 000 .

Original: As expected, IALE performs best at acq =1 and worst at acq =40, if n is unchanged, because n limits the available choices, e.g., bad samples have to be chosen.
Revised: As expected, IALE performs best at acq =1 and worst at acq =40, if n is unchanged, because n limits the available choices, e.g., bad samples have to be chosen.

Original: Increasing n to 1 , 000 alleviates this issue.
Revised: Increasing n to 1 , 000 alleviates this issue.

Original: However, there is an upper limit to the size of D sub after which performance deteriorates again, see
Revised: However, there is an upper limit to the size of D sub after which performance deteriorates again, see

Original: Figure 5b.
Revised: Figure 5b.

Original: This could be because
Revised: This could be because

Original: random sub-sampling actually simpliﬁes the selection of diverse, uncertain samples.
Revised: random sub-sampling actually simpliﬁes the selection of diverse, uncertain samples.

Original: The lower limit becomes apparent again when n is smaller than 10 times acq , with n = acq essentially being a random sampling.
Revised: The lower limit becomes apparent again when n is smaller than 10 times acq , with n = acq essentially being a random sampling.

Original: From our observations, n should be 10 −times acq .
Revised: From our observations, n should be 10 −times acq .

Original: From the small differences within this value range, it is suggested that our method is suitable for larger acquisition sizes for batch-mode AL, as its performance is not affected much.
Revised: From the small differences within this value range, it is suggested that our method is suitable for larger acquisition sizes for batch-mode AL, as its performance is not affected much.

Original: Varying experts.
Revised: Varying experts.

Original: To investigate the inﬂuence of experts, we leave out some types of experts: We categorize them into 4 groups, i.e., uncertainty ( McdropEns ), soft-max uncertainty ( EntrConf ), diversity ( Coreset ) and hybrid ( Badge ), and leave one subset out.
Revised: To investigate the inﬂuence of experts, we leave out some types of experts: We categorize them into 4 groups, i.e., uncertainty ( McdropEns ), soft-max uncertainty ( EntrConf ), diversity ( Coreset ) and hybrid ( Badge ), and leave one subset out.

Original: We fully train each method on MNIST with B = 1 , 000 and an acquisition size of 10 , and present the results of the evaluation on KMNIST in Figure 5c (more results including the ablation of state elements can be found in Appendixes A.4.4 and A.4.3).
Revised: We fully train each method on MNIST with B = 1 , 000 and an acquisition size of 10 , and present the results of the evaluation on KMNIST in Figure 5c (more results including the ablation of state elements can be found in Appendixes A.4.4 and A.4.3).

Original: We see that most combinations perform well compared to the baselines.
Revised: We see that most combinations perform well compared to the baselines.

Original: However, leaving out uncertainty or soft-max uncertainty experts can decrease performance.
Revised: However, leaving out uncertainty or soft-max uncertainty experts can decrease performance.

Original: Even though training time is longer with MC-Dropout , the gain in performance can be worth it.
Revised: Even though training time is longer with MC-Dropout , the gain in performance can be worth it.

Original: In contrast, the soft-max uncertainty based heuristics are computationally cheap and yield good policies.
Revised: In contrast, the soft-max uncertainty based heuristics are computationally cheap and yield good policies.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a novel imitation learning approach for active learning.
Revised: We proposed a novel imitation learning approach for active learning.

Original: Our method learns to imitate the behavior of different active learning heuristics, such as uncertainty-, diversity-, model changeand query-by-committee-based heuristics, on one initial dataset and model, and transfers the obtained knowledge to work on other (types of) datasets and models (that share an embedding space).
Revised: Our method learns to imitate the behavior of different active learning heuristics, such as uncertainty-, diversity-, model changeand query-by-committee-based heuristics, on one initial dataset and model, and transfers the obtained knowledge to work on other (types of) datasets and models (that share an embedding space).

Original: Our policy network is a simple MLP that learns to imitate the experts based on embeddings of the dataset samples.
Revised: Our policy network is a simple MLP that learns to imitate the experts based on embeddings of the dataset samples.

Original: Our experiments on well-known datasets show that we outperform the state of the art consistently (despite being a batch-mode AL approach).
Revised: Our experiments on well-known datasets show that we outperform the state of the art consistently (despite being a batch-mode AL approach).

Original: An ablation study and analysis of the inﬂuence of certain hyper-parameters also shows the limitations of our approach.
Revised: An ablation study and analysis of the inﬂuence of certain hyper-parameters also shows the limitations of our approach.

Original: Future work investigates the relationship between acquisition sizes and sub-pools, and an analysis on how the state (embedding) enables π to transfer its active learning strategy, potentially leading to the use of artiﬁcial data for ﬁnding optimal AL strategies suitable for speciﬁc active learning scenarios.
Revised: Future work investigates the relationship between acquisition sizes and sub-pools, and an analysis on how the state (embedding) enables π to transfer its active learning strategy, potentially leading to the use of artiﬁcial data for ﬁnding optimal AL strategies suitable for speciﬁc active learning scenarios.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: In this section we provide an extension of the experiments section (Section 4) and feature additional results that support a more complete evaluation of IALE .
Revised: In this section we provide an extension of the experiments section (Section 4) and feature additional results that support a more complete evaluation of IALE .

Original: We adhere to the same section structure.
Revised: We adhere to the same section structure.

Original: A. E XPERIMENTAL S ETUP A.1. B ASELINES
Revised: A. E XPERIMENTAL S ETUP A.1. B ASELINES

Original: In the following is a short explanation of the baselines and experts that we used in our experiments:
Revised: In the following is a short explanation of the baselines and experts that we used in our experiments:

Original: Random Sampling randomly samples data points from the unlabeled pool.
Revised: Random Sampling randomly samples data points from the unlabeled pool.

Original: MC-Dropout (Gal et al., 2017) approximates the sample uncertainty of the model by repeatedly computing inferences of the sample, i.e., 20 times, with dropout enabled in the classiﬁcation model 3.
Revised: MC-Dropout (Gal et al., 2017) approximates the sample uncertainty of the model by repeatedly computing inferences of the sample, i.e., 20 times, with dropout enabled in the classiﬁcation model 3.

Original: Ensemble (Beluch et al., 2018) trains an ensemble of 5 classiﬁers with different weight initializations.
Revised: Ensemble (Beluch et al., 2018) trains an ensemble of 5 classiﬁers with different weight initializations.

Original: The uncertainty of the samples is quantiﬁed by the disagreement between the model predictions.
Revised: The uncertainty of the samples is quantiﬁed by the disagreement between the model predictions.

Original: CoreSet (Sener & Savarese, 2018) solves the k -center problem using the poolembeddings of the last dense layer (128 neurons) before the softmax output to pick samples for labeling.
Revised: CoreSet (Sener & Savarese, 2018) solves the k -center problem using the poolembeddings of the last dense layer (128 neurons) before the softmax output to pick samples for labeling.

Original: BADGE (Ash et al., 2020) uses the gradient of the loss (given pseudo labels), both its magnitude and direction, for k-means++ clustering, to select uncertain and diverse samples from a batch.
Revised: BADGE (Ash et al., 2020) uses the gradient of the loss (given pseudo labels), both its magnitude and direction, for k-means++ clustering, to select uncertain and diverse samples from a batch.

Original: Confidence -sampling (Wang & Shang, 2014) selects samples with the lowest class probability of the soft-max predictions.
Revised: Confidence -sampling (Wang & Shang, 2014) selects samples with the lowest class probability of the soft-max predictions.

Original: Entropy -sampling (Wang & Shang, 2014) calculates the soft-max class probabilities’ entropy and then selects samples with the largest entropy, i.e., where the model is least certain.
Revised: Entropy -sampling (Wang & Shang, 2014) calculates the soft-max class probabilities’ entropy and then selects samples with the largest entropy, i.e., where the model is least certain.

Original: ALIL (Liu et al., 2018): we modify ALIL’s implementation (that is initially intended for NLP tasks) to work on image classiﬁcation task.
Revised: ALIL (Liu et al., 2018): we modify ALIL’s implementation (that is initially intended for NLP tasks) to work on image classiﬁcation task.

Original: Due to the high runtime costs of running ALIL (as the acquisition size is 1), we perform the training of ALIL for 20 episodes.
Revised: Due to the high runtime costs of running ALIL (as the acquisition size is 1), we perform the training of ALIL for 20 episodes.

Original: We trained the ALIL policy network with a labeling budget B of 1 , 000 and an up-scaled policy network comparable to that of our method along with a similar
Revised: We trained the ALIL policy network with a labeling budget B of 1 , 000 and an up-scaled policy network comparable to that of our method along with a similar

Original: M as we use to evaluate the other AL approaches.
Revised: M as we use to evaluate the other AL approaches.

Original: We left the coin-toss parameter at 0 . 5 , and the k parameter for sequential selections from a random subset of D pool at 10 .
Revised: We left the coin-toss parameter at 0 . 5 , and the k parameter for sequential selections from a random subset of D pool at 10 .

Original: We use the variation ratio metric (Gal et al., 2017) to quantify and select the data samples for labeling from the uncertainty obtained from MC-Dropout and Ensemble heuristics.
Revised: We use the variation ratio metric (Gal et al., 2017) to quantify and select the data samples for labeling from the uncertainty obtained from MC-Dropout and Ensemble heuristics.

Original: The variation ratio metric is given by its Bayesian deﬁnition (Gal et al., 2017) for a data sample x ∈ D pool in Equationand for an ensemble expert (Beluch et al., 2018) in Equation 4:
Revised: The variation ratio metric is given by its Bayesian deﬁnition (Gal et al., 2017) for a data sample x ∈ D pool in Equationand for an ensemble expert (Beluch et al., 2018) in Equation 4:

Original: [Equation]
Revised: [Equation]

Original: where m is number of occurrences of the mode and N is the number of forward passes or number of models in the ensemble
Revised: where m is number of occurrences of the mode and N is the number of forward passes or number of models in the ensemble

Original: . A. P OLICY T RAINING A.2. E XPLORATION -E XPLOITATION IN DA GGER
Revised: . A. P OLICY T RAINING A.2. E XPLORATION -E XPLOITATION IN DA GGER

Original: DA GGER uses a hyper-parameter p that determines how likely π predicts the next action, and thereby setting the next state, instead of using the best expert from E .
Revised: DA GGER uses a hyper-parameter p that determines how likely π predicts the next action, and thereby setting the next state, instead of using the best expert from E .

Original: In this preliminary study we compare the inﬂuence it has to either ﬁx p to 0 .
Revised: In this preliminary study we compare the inﬂuence it has to either ﬁx p to 0 .

Original: 5 or to use an exponential decay parameterized by the number of the current episode epi : 1 − 0 .
Revised: 5 or to use an exponential decay parameterized by the number of the current episode epi : 1 − 0 .

Original: 9 epi .
Revised: 9 epi .

Original: We train the policy on MNIST for 100 episodes with a labeling budget of 1 , 000 and an acquisition size of 10 (as before).
Revised: We train the policy on MNIST for 100 episodes with a labeling budget of 1 , 000 and an acquisition size of 10 (as before).

Original: Our result is that the ﬁxed policy outperforms the exponential one by a small margin for the transfer of the policy to another dataset than the trained one, which is in line with previous ﬁndings (Liu et al., 2018).
Revised: Our result is that the ﬁxed policy outperforms the exponential one by a small margin for the transfer of the policy to another dataset than the trained one, which is in line with previous ﬁndings (Liu et al., 2018).

Original: A balanced (i.e., ﬁxed) ratio does not emphasize one over the other, whereas an exponentially decay quickly relies on the policy for selecting new states of the dataset
Revised: A balanced (i.e., ﬁxed) ratio does not emphasize one over the other, whereas an exponentially decay quickly relies on the policy for selecting new states of the dataset

Original: [Figure]
Revised: [Figure]

Original: and thus it trains on too few optimal states over the AL cycle
Revised: and thus it trains on too few optimal states over the AL cycle

Original: [Figure]
Revised: [Figure]

Original: A.2. O VERLAP RATIOS
Revised: A.2. O VERLAP RATIOS

Original: We additionally show FMNIST and KMNIST overlap ratios in Fig.
Revised: We additionally show FMNIST and KMNIST overlap ratios in Fig.

Original: The policy chooses about half of the samples differently from any single baseline.
Revised: The policy chooses about half of the samples differently from any single baseline.

Original: The overlap is lower on FMNIST , where our method is the only one that beats random sampling.
Revised: The overlap is lower on FMNIST , where our method is the only one that beats random sampling.

Original: CoreSet overlaps least often and performs worst.
Revised: CoreSet overlaps least often and performs worst.

Original: [Figure]
Revised: [Figure]

Original: A. P OLICY T RANSFER
Revised: A. P OLICY T RANSFER

Original: In this section we provide further studies on how our method performs in regard to applying it to unseen scenarios.
Revised: In this section we provide further studies on how our method performs in regard to applying it to unseen scenarios.

Original: These include that we (1) train the policy on other source datasets than MNIST , that we (2) apply the policy onto other classiﬁers than the one it was trained on
Revised: These include that we (1) train the policy on other source datasets than MNIST , that we (2) apply the policy onto other classiﬁers than the one it was trained on

Original: [Figure]
Revised: [Figure]

Original: and that we (3) use different datasets and classiﬁer models in training and application of the policy.
Revised: and that we (3) use different datasets and classiﬁer models in training and application of the policy.

Original: We aim to show that π learns a (relatively) task-agnostic AL strategy, that works in inference and outperforms the baselines, as long as the state can be expressed similarly to the one that has been used during training.
Revised: We aim to show that π learns a (relatively) task-agnostic AL strategy, that works in inference and outperforms the baselines, as long as the state can be expressed similarly to the one that has been used during training.

Original: A.3. V ARYING SOURCE DATASETS
Revised: A.3. V ARYING SOURCE DATASETS

Original: Our policy does not coincidentally perform well due to a speciﬁc source dataset that is most suitable.
Revised: Our policy does not coincidentally perform well due to a speciﬁc source dataset that is most suitable.

Original: In Figure 7, we show two additional policies that were trained on FMNIST or KMNIST (with unchanged hyper-parameter settings).
Revised: In Figure 7, we show two additional policies that were trained on FMNIST or KMNIST (with unchanged hyper-parameter settings).

Original: We see comparable performance that indicates that IALE actually learns to actively learn and not just remembers the source datasets as it makes not difference which datasets are chosen as the source and the target dataset
Revised: We see comparable performance that indicates that IALE actually learns to actively learn and not just remembers the source datasets as it makes not difference which datasets are chosen as the source and the target dataset

Original: [Figure]
Revised: [Figure]

Original: A.3. C LASSIFIER ARCHITECTURES .
Revised: A.3. C LASSIFIER ARCHITECTURES .

Original: Our method is not bound to a speciﬁc classiﬁer architecture.
Revised: Our method is not bound to a speciﬁc classiﬁer architecture.

Original: To show this, we evaluate our approach with MLP and ResNet-18 classiﬁers.
Revised: To show this, we evaluate our approach with MLP and ResNet-18 classiﬁers.

Original: We use the same hyper-parameters as with the CNN and only switch out the underlying classiﬁer.
Revised: We use the same hyper-parameters as with the CNN and only switch out the underlying classiﬁer.

Original: All results for the experiments are given in Figure 8.
Revised: All results for the experiments are given in Figure 8.

Original: We see the robustness of π over fundamentally different classiﬁer architectures ( 2 to 18 layers).
Revised: We see the robustness of π over fundamentally different classiﬁer architectures ( 2 to 18 layers).

Original: The deviations for ResNet-18 are very large due to the very deep architecture and the modest amount of training data.
Revised: The deviations for ResNet-18 are very large due to the very deep architecture and the modest amount of training data.

Original: We use median ﬁltering in Figures 8g, 8h
Revised: We use median ﬁltering in Figures 8g, 8h

Original: [Figure]
Revised: [Figure]

Original: and 8i.
Revised: and 8i.

Original: These experiments show that π can learn AL strategies for both very small and very deep architectures and still outperform baselines.
Revised: These experiments show that π can learn AL strategies for both very small and very deep architectures and still outperform baselines.

Original: Even though the strongest baselines, i
Revised: Even though the strongest baselines, i

Original: [Figure]
Revised: [Figure]

Original: e., CoreSet and MC
Revised: e., CoreSet and MC

Original: Dropout , come close to our method in accuracy, they are less versatile and require more computational resources, that is especially noticeable on deeper architectures
Revised: Dropout , come close to our method in accuracy, they are less versatile and require more computational resources, that is especially noticeable on deeper architectures

Original: [Figure]
Revised: [Figure]

Original: a cc u r a cy sc o r e
Revised: a cc u r a cy sc o r e

Original: IALERandomBADGE
Revised: IALERandomBADGE

Original: ConfidenceEntropyCoreSet
Revised: ConfidenceEntropyCoreSet

Original: EnsembleMC-Dropout (
Revised: EnsembleMC-Dropout (

Original: [Figure]
Revised: [Figure]

Original: A.3. G ENERALIZATION
Revised: A.3. G ENERALIZATION

Original: To show that π learns active learning independent from task and classiﬁer, we conduct experiments where we mix both the source datasets and the classiﬁers, as only the state retains the same formulation.
Revised: To show that π learns active learning independent from task and classiﬁer, we conduct experiments where we mix both the source datasets and the classiﬁers, as only the state retains the same formulation.

Original: Since these states are the same for CNN and Resnet-18 classiﬁers (independant of the datasets) we can transfer a policy trained using a Resnet-18 classiﬁer to a CNN classiﬁer and vice versa.
Revised: Since these states are the same for CNN and Resnet-18 classiﬁers (independant of the datasets) we can transfer a policy trained using a Resnet-18 classiﬁer to a CNN classiﬁer and vice versa.

Original: We report the results for applying π (trained on Resnet-18 and MNIST ) to a CNN and all the datasets in Figure 9.
Revised: We report the results for applying π (trained on Resnet-18 and MNIST ) to a CNN and all the datasets in Figure 9.

Original: IALE is always performing at the top.
Revised: IALE is always performing at the top.

Original: These results convincingly show that our method learns a model- and task-agnostic active learning strategy that transfers knowledge between datasets and even between classiﬁer architectures.
Revised: These results convincingly show that our method learns a model- and task-agnostic active learning strategy that transfers knowledge between datasets and even between classiﬁer architectures.

Original: [Figure]
Revised: [Figure]

Original: We conclude the transfer studies by applying pre-trained π to a Resnet-18 classiﬁer on the
Revised: We conclude the transfer studies by applying pre-trained π to a Resnet-18 classiﬁer on the

Original: CIFAR-10 (Krizhevsky, 2009) (color) image classiﬁcation dataset.
Revised: CIFAR-10 (Krizhevsky, 2009) (color) image classiﬁcation dataset.

Original: Each image in CIFARis 32 × 32 pixels and has 3 color channels.
Revised: Each image in CIFARis 32 × 32 pixels and has 3 color channels.

Original: The transfer of π is possible as the classiﬁer’s state is both invariant to the models and independent from the number of color channels of the image dataset and the input size.
Revised: The transfer of π is possible as the classiﬁer’s state is both invariant to the models and independent from the number of color channels of the image dataset and the input size.

Original: For our experiments we use two different π : π 1 was trained using Resnet-18 and
Revised: For our experiments we use two different π : π 1 was trained using Resnet-18 and

Original: MNIST ( IALE Resnet ) and π 2 was trained using CNN and MNIST ( IALE CNN ).
Revised: MNIST ( IALE Resnet ) and π 2 was trained using CNN and MNIST ( IALE CNN ).

Original: We train the classiﬁers with an acquisition size of 10 until the labeling budget of 10 . 000 is reached.
Revised: We train the classiﬁers with an acquisition size of 10 until the labeling budget of 10 . 000 is reached.

Original: The results are presented in Figure 10.
Revised: The results are presented in Figure 10.

Original: As the acquisition size of 10 results in noisy curves we report the raw learning curves (Figure 10a)and median ﬁltered learning curves (Figure 10b).
Revised: As the acquisition size of 10 results in noisy curves we report the raw learning curves (Figure 10a)and median ﬁltered learning curves (Figure 10b).

Original: We report the interesting segment of the learning curve in more detail (ﬁltered) in Figures 10c.
Revised: We report the interesting segment of the learning curve in more detail (ﬁltered) in Figures 10c.

Original: The results generally show the feasibility of transferring π to both different classiﬁers and datasets.
Revised: The results generally show the feasibility of transferring π to both different classiﬁers and datasets.

Original: IALE is en par or better than Random , and the other baselines are either en par or worse than Random (some of them considerably).
Revised: IALE is en par or better than Random , and the other baselines are either en par or worse than Random (some of them considerably).

Original: Moreover, besides IALE ’s better accuracy compared to all other methods, its run time is highly competitive.
Revised: Moreover, besides IALE ’s better accuracy compared to all other methods, its run time is highly competitive.

Original: Per experiment iteration (with 10 . 000 samples, acquired over 998 steps of size 10 , each step including a complete re-training of Resnet-18 for 100 epochs), run times for IALE are 10:17:31 (h:m:s) on a high performance GPU (NVidia V100) (9:45:12 for Random sampling), compared to 49:15:23 for Ensembles , 14:23:18 for MC-Dropout and 11:58:47 for BADGE .
Revised: Per experiment iteration (with 10 . 000 samples, acquired over 998 steps of size 10 , each step including a complete re-training of Resnet-18 for 100 epochs), run times for IALE are 10:17:31 (h:m:s) on a high performance GPU (NVidia V100) (9:45:12 for Random sampling), compared to 49:15:23 for Ensembles , 14:23:18 for MC-Dropout and 11:58:47 for BADGE .

Original: Only
Revised: Only

Original: Conf (10:12:01) and Entropy (10:05:21) are quicker (however, they both also perform worse than
Revised: Conf (10:12:01) and Entropy (10:05:21) are quicker (however, they both also perform worse than

Original: Random ).
Revised: Random ).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: t
Revised: t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: While more experiments are certainly required to further emphasize these initial claims of generalizability to more diverse types of datasets
Revised: While more experiments are certainly required to further emphasize these initial claims of generalizability to more diverse types of datasets

Original: [Figure]
Revised: [Figure]

Original: these ﬁndings are already very promising
Revised: these ﬁndings are already very promising

Original: [Figure]
Revised: [Figure]

Original: A. A BLATION S TUDIES A.4. H YPERPARAMETERS .
Revised: A. A BLATION S TUDIES A.4. H YPERPARAMETERS .

Original: We report ﬁne-granular steps of acquisition sizes in Figure 11a, with values between 1 and 10 , plus 20 and 40 , for | D sub | of 100 .
Revised: We report ﬁne-granular steps of acquisition sizes in Figure 11a, with values between 1 and 10 , plus 20 and 40 , for | D sub | of 100 .

Original: Overall, a clear difference is not visible below 10 samples.
Revised: Overall, a clear difference is not visible below 10 samples.

Original: For enhanced readability, we show a magniﬁed section of the varied acquisition sizes and | D sub | in
Revised: For enhanced readability, we show a magniﬁed section of the varied acquisition sizes and | D sub | in

Original: Fig. 11b
Revised: Fig. 11b

Original: [Figure]
Revised: [Figure]

Original: that clearly shows the beneﬁts of tuning | D sub | to a suitable value for the acquisition size
Revised: that clearly shows the beneﬁts of tuning | D sub | to a suitable value for the acquisition size

Original: [Figure]
Revised: [Figure]

Original: Acquisition sizes for baselines:
Revised: Acquisition sizes for baselines:

Original: We additionally compare the baseline active learning methods with our approach, as these exhibit different performance at different acquisition sizes, see Fig.
Revised: We additionally compare the baseline active learning methods with our approach, as these exhibit different performance at different acquisition sizes, see Fig.

Original: We have included comparisons with acquisition sizes of either 1 or 100 (1 or 3 repetitions).
Revised: We have included comparisons with acquisition sizes of either 1 or 100 (1 or 3 repetitions).

Original: For our method, for an acquisition size of 1 we chose | D sub | = 100 and for acquisition size of 100 we chose | D sub | = 2 , 000 .
Revised: For our method, for an acquisition size of 1 we chose | D sub | = 100 and for acquisition size of 100 we chose | D sub | = 2 , 000 .

Original: While the results show that IALE outperforms the baselines they also highlight the large effect that the acquisition size has on some of the baseline methods.
Revised: While the results show that IALE outperforms the baselines they also highlight the large effect that the acquisition size has on some of the baseline methods.

Original: For instance, CoreSet constructs better set covers with larger batches, and BADGE increases its accuracy by constructing a representative sampling as well.
Revised: For instance, CoreSet constructs better set covers with larger batches, and BADGE increases its accuracy by constructing a representative sampling as well.

Original: At the same time the uncertainty-based methods, apart from
Revised: At the same time the uncertainty-based methods, apart from

Original: Entropy , remain unaffected.
Revised: Entropy , remain unaffected.

Original: [Figure]
Revised: [Figure]

Original: A.4. V ARYING E XPERTS
Revised: A.4. V ARYING E XPERTS

Original: We present more results for variations of sets of experts in Figure 13.
Revised: We present more results for variations of sets of experts in Figure 13.

Original: We leave out some types of experts, and train the policies with the unchanged hyper-parameters and the CNN classiﬁer.
Revised: We leave out some types of experts, and train the policies with the unchanged hyper-parameters and the CNN classiﬁer.

Original: The results for all three datasets show that the generally high performance of IALE holds for the leaveone-out sets of experts, with the full set of experts being consistently among the best performing policies.
Revised: The results for all three datasets show that the generally high performance of IALE holds for the leaveone-out sets of experts, with the full set of experts being consistently among the best performing policies.

Original: [Figure]
Revised: [Figure]

Original: A.4. V ARYING S TATE E LEMENTS
Revised: A.4. V ARYING S TATE E LEMENTS

Original: Next, we study the state more closely.
Revised: Next, we study the state more closely.

Original: For unlabeled samples, the state contains two types of representations for predictive uncertainty: the statistics on predicted labels M ( x n ) and the gradient representations g
Revised: For unlabeled samples, the state contains two types of representations for predictive uncertainty: the statistics on predicted labels M ( x n ) and the gradient representations g

Original: ( M e ( x n )) .
Revised: ( M e ( x n )) .

Original: In this study, we focus on leaving out one or the other.
Revised: In this study, we focus on leaving out one or the other.

Original: To get the full picture
Revised: To get the full picture

Original: [Figure]
Revised: [Figure]

Original: we again train sets of experts for reduced states.
Revised: we again train sets of experts for reduced states.

Original: In Figure 14 we see that dropping gradients generally decreases performance (bottom row), while dropping predicted labels M ( x n ) affects performance very little (top row).
Revised: In Figure 14 we see that dropping gradients generally decreases performance (bottom row), while dropping predicted labels M ( x n ) affects performance very little (top row).

Original: However, the inﬂuence of different sets of experts is more important.
Revised: However, the inﬂuence of different sets of experts is more important.

Original: We cannot see that a particular set of states and experts generally outperforms others consistently (while the negative effect of leaving out g ( M e ( x n )) is consistently visible).
Revised: We cannot see that a particular set of states and experts generally outperforms others consistently (while the negative effect of leaving out g ( M e ( x n )) is consistently visible).

Original: Overall, we ﬁnd that using as many experts as available, combined with a full state both performs well and works reliably.
Revised: Overall, we ﬁnd that using as many experts as available, combined with a full state both performs well and works reliably.

Original: Even though training a policy this way does not guarantee the best performance, it always performs among with the group of best policies
Revised: Even though training a policy this way does not guarantee the best performance, it always performs among with the group of best policies

Original: [Figure]
Revised: [Figure]

Original: A.4. A DDITIONAL D ATASETS
Revised: A.4. A DDITIONAL D ATASETS

Original: We additionally run experiment on harder datasets, i.e., SVHN and CIFAR-100 in Fig.
Revised: We additionally run experiment on harder datasets, i.e., SVHN and CIFAR-100 in Fig.

Original: SVHN: We train a Resnet18 on SVHN with an acquisition size of 1,000 and a labeling budget of 16,000.
Revised: SVHN: We train a Resnet18 on SVHN with an acquisition size of 1,000 and a labeling budget of 16,000.

Original: We initially label 1,000 samples and evaluate over 5 repetitions.
Revised: We initially label 1,000 samples and evaluate over 5 repetitions.

Original: While the results in Fig.exhibit some variance, we can clearly see that IALE performs best (and is the only AL methods that is actually able to beat a random sampling.
Revised: While the results in Fig.exhibit some variance, we can clearly see that IALE performs best (and is the only AL methods that is actually able to beat a random sampling.

Original: CIFAR-100: Next, we evaluate datasets with a larger number of classes by generalizing the composition of the state (removing prediction and empirical class distribution ).
Revised: CIFAR-100: Next, we evaluate datasets with a larger number of classes by generalizing the composition of the state (removing prediction and empirical class distribution ).

Original: We train two policies, one with a CNN classiﬁer on MNIST and the second with a Resnet18 classiﬁer on CIFAR-10 .
Revised: We train two policies, one with a CNN classiﬁer on MNIST and the second with a Resnet18 classiﬁer on CIFAR-10 .

Original: Then, we apply the resulting policies in the following experiment.
Revised: Then, we apply the resulting policies in the following experiment.

Original: We train a Resnet18 on the very hard CIFAR-100 dataset, with an acquisition size of 1,000 and a labeling budget of 20000, and initially label 1,000 samples.
Revised: We train a Resnet18 on the very hard CIFAR-100 dataset, with an acquisition size of 1,000 and a labeling budget of 20000, and initially label 1,000 samples.

Original: We evaluate over 5 repetitions.
Revised: We evaluate over 5 repetitions.

Original: Fewer samples or less powerful network architectures tend to fail to converge in our experiments.
Revised: Fewer samples or less powerful network architectures tend to fail to converge in our experiments.

Original: The results in Fig.
Revised: The results in Fig.

Original: 15 show that both policies, i.e., transferred from CNN/MNIST and from Restnet18/CIFAR10 with an arbitrary number of classes, perform best among all heuristics, and beat random sampling.[Figure] e s t da t a a
Revised: 15 show that both policies, i.e., transferred from CNN/MNIST and from Restnet18/CIFAR10 with an arbitrary number of classes, perform best among all heuristics, and beat random sampling.

Original: [Figure]
Revised: [Figure]

Original: u ra [Figure] o r e
Revised: r

Original: [Figure]
Revised: [Figure]