Original: L EARNING A CTIVE L EARNING IN THE B ATCH -M ODE S ETUP WITH E NSEMBLES OF A CTIVE L EARNING A GENTS
Revised: L EARNING A CTIVE L EARNING IN THE B ATCH -M ODE S ETUP WITH E NSEMBLES OF A CTIVE L EARNING A GENTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Supervised learning models perform best when trained on a lot of data, but annotating training data is very costly in some domains.
Revised: Supervised learning models perform best when trained on a lot of data, but annotating training data is very costly in some domains.

Original: Active learning aims to choose only the most informative subset of unlabelled samples for annotation, thus saving annotation cost.
Revised: Active learning aims to choose only the most informative subset of unlabelled samples for annotation, thus saving annotation cost.

Original: Several heuristics for choosing this subset have been developed, which use ﬁxed policies for their choice.
Revised: Several heuristics for choosing this subset have been developed, which use ﬁxed policies for their choice.

Original: They are easily understandable and applied.
Revised: They are easily understandable and applied.

Original: However, there is no heuristic performing optimal in all settings.
Revised: However, there is no heuristic performing optimal in all settings.

Original: This led to the development of agents learning the best selection policy from data.
Revised: This led to the development of agents learning the best selection policy from data.

Original: They formulate active learning as a Markov decision process and apply reinforcement learning (RL) methods to it.
Revised: They formulate active learning as a Markov decision process and apply reinforcement learning (RL) methods to it.

Original: Their advantage is that they are able to use many features and adapt to the speciﬁc task.
Revised: Their advantage is that they are able to use many features and adapt to the speciﬁc task.

Original: Our paper proposes a new approach combining these advantages of learning active learning and heuristics: We propose to learn active learning using a parameterized ensemble of agents, where the parameters are learned using Monte Carlo policy search.
Revised: Our paper proposes a new approach combining these advantages of learning active learning and heuristics: We propose to learn active learning using a parameterized ensemble of agents, where the parameters are learned using Monte Carlo policy search.

Original: As this approach can incorporate any active learning agent into its ensemble, it allows to increase the performance of every active learning agent by learning how to combine it with others.
Revised: As this approach can incorporate any active learning agent into its ensemble, it allows to increase the performance of every active learning agent by learning how to combine it with others.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Supervised machine learning systems perform best when trained on a large amount of training data.
Revised: Supervised machine learning systems perform best when trained on a large amount of training data.

Original: Obtaining this data by labelling can cause huge time and cost efforts in some domains.
Revised: Obtaining this data by labelling can cause huge time and cost efforts in some domains.

Original: Active learning in the selective scenario overcomes this bottleneck by selecting a subset of all unlabelled samples to be labelled such that the model trained on them learns as much as possible and achieves a high accuracy (Cohn et al., 1994).
Revised: Active learning in the selective scenario overcomes this bottleneck by selecting a subset of all unlabelled samples to be labelled such that the model trained on them learns as much as possible and achieves a high accuracy (Cohn et al., 1994).

Original: Heuristic active learning agents choose the samples to be labelled using a ﬁxed policy.
Revised: Heuristic active learning agents choose the samples to be labelled using a ﬁxed policy.

Original: They have a known and predictable behaviour and their policy is easy to understand (Settles et al., 2008).
Revised: They have a known and predictable behaviour and their policy is easy to understand (Settles et al., 2008).

Original: However, they have two main disadvantages: First, they rarely combine different features.
Revised: However, they have two main disadvantages: First, they rarely combine different features.

Original: Second, it was found that the best heuristic highly depends on the dataset and supervised learning model used (Lowell et al., 2018).
Revised: Second, it was found that the best heuristic highly depends on the dataset and supervised learning model used (Lowell et al., 2018).

Original: More recently, these shortcomings have been addressed by learning active learning directly from data (Konyushkova et al., 2017; 2018; Bachman et al., 2017; Fang et al., 2017; Liu et al., 2018b;a).
Revised: More recently, these shortcomings have been addressed by learning active learning directly from data (Konyushkova et al., 2017; 2018; Bachman et al., 2017; Fang et al., 2017; Liu et al., 2018b;a).

Original: The authors formulate active learning as a Markov decision process and apply reinforcement learning (RL) methods like Q-Learning and imitation learning to it.
Revised: The authors formulate active learning as a Markov decision process and apply reinforcement learning (RL) methods like Q-Learning and imitation learning to it.

Original: While this approach promises to overcome the advantages of heuristics it introduces new problems: There is the credit assignment problem (Minsky, 1961), the training is computationally very costly (Amodei et al.), and many results are not signiﬁcant and hard to reproduce (Henderson et al., 2017).
Revised: While this approach promises to overcome the advantages of heuristics it introduces new problems: There is the credit assignment problem (Minsky, 1961), the training is computationally very costly (Amodei et al.), and many results are not signiﬁcant and hard to reproduce (Henderson et al., 2017).

Original: Learning active learning with RL in the batch-mode setting has only received little attention in literature.
Revised: Learning active learning with RL in the batch-mode setting has only received little attention in literature.

Original: One of the reasons is that choosing a batch of samples instead of a single one makes the action space exponentially bigger and thus ﬁnding the action maximizing a value function can not be done by iterating over all actions anymore.
Revised: One of the reasons is that choosing a batch of samples instead of a single one makes the action space exponentially bigger and thus ﬁnding the action maximizing a value function can not be done by iterating over all actions anymore.

Original: Furthermore, it makes it harder to attribute the reward to a speciﬁc parameter of the policy.
Revised: Furthermore, it makes it harder to attribute the reward to a speciﬁc parameter of the policy.

Original: Our paper addresses these shortcomings by proposing an active learning agent being a weighted ensemble of other agents.
Revised: Our paper addresses these shortcomings by proposing an active learning agent being a weighted ensemble of other agents.

Original: The weights are learned using Monte Carlo policy search and blackbox optimization.
Revised: The weights are learned using Monte Carlo policy search and blackbox optimization.

Original: As several different agents can be used as part of the ensemble, many different features can be included.
Revised: As several different agents can be used as part of the ensemble, many different features can be included.

Original: By learning the weights of each agent, the ensemble can adapt to the dataset, model and optimization metric.
Revised: By learning the weights of each agent, the ensemble can adapt to the dataset, model and optimization metric.

Original: Nonetheless, the learning is very robust and global, as blackbox optimization over a small number of parameters is much easier than reinforcement learning.
Revised: Nonetheless, the learning is very robust and global, as blackbox optimization over a small number of parameters is much easier than reinforcement learning.

Original: Furthermore, the policy is easily understandable and interpretable.
Revised: Furthermore, the policy is easily understandable and interpretable.

Original: The ensemble should not be seen as an alternative approach to current approaches, but rather as an extension of them allowing to learn how to combine several approaches to further increase their performance.
Revised: The ensemble should not be seen as an alternative approach to current approaches, but rather as an extension of them allowing to learn how to combine several approaches to further increase their performance.

Original: We evaluate our approach using active learning tasks from different domains and with random forests, CNNs and LSTMs as classiﬁers.
Revised: We evaluate our approach using active learning tasks from different domains and with random forests, CNNs and LSTMs as classiﬁers.

Original: The experiments show that the ensemble consistently performs at least as good as the best agent it includes and sometimes even outperforms it by a signiﬁcant margin.
Revised: The experiments show that the ensemble consistently performs at least as good as the best agent it includes and sometimes even outperforms it by a signiﬁcant margin.

Original: Our main contributions are:
Revised: Our main contributions are:

Original: • We propose an approach to learning active learning using an ensemble of heuristics.
Revised: • We propose an approach to learning active learning using an ensemble of heuristics.

Original: It combines the advantages of heuristics and approaches learning active learning.
Revised: It combines the advantages of heuristics and approaches learning active learning.

Original: The experiments show that the theoretical advantages also translate into a high empirical performance.
Revised: The experiments show that the theoretical advantages also translate into a high empirical performance.

Original: • We show with our experiments that it is very important to train the ensemble on a similar task it is evaluated on.
Revised: • We show with our experiments that it is very important to train the ensemble on a similar task it is evaluated on.

Original: This is contrary to assumptions in earlier literature that a learning agent trained on a synthetic dataset works well in completely different domains.
Revised: This is contrary to assumptions in earlier literature that a learning agent trained on a synthetic dataset works well in completely different domains.

Original:  R ELATED WORK 2. A CTIVE LEARNING HEURISTICS
Revised:  R ELATED WORK 2. A CTIVE LEARNING HEURISTICS

Original: Heuristic frameworks are active learning frameworks relying on engineered, ﬁxed policies.
Revised: Heuristic frameworks are active learning frameworks relying on engineered, ﬁxed policies.

Original: Their performance depends highly on the active learning task, with no heuristic being able to outperform the others in all cases.
Revised: Their performance depends highly on the active learning task, with no heuristic being able to outperform the others in all cases.

Original: There are three core ideas which kind of samples should be chosen to be labelled.
Revised: There are three core ideas which kind of samples should be chosen to be labelled.

Original: They can be clearly distinguished, as they rely on disjoint sets of features:
Revised: They can be clearly distinguished, as they rely on disjoint sets of features:

Original: Informativeness sampling: One group of heuristics prefers to choose informative samples, which can be expected to change the supervised learning model a lot when added to the labelled set.
Revised: Informativeness sampling: One group of heuristics prefers to choose informative samples, which can be expected to change the supervised learning model a lot when added to the labelled set.

Original: Their features for a sample can be calculated only given the current supervised learning model and the sample itself.
Revised: Their features for a sample can be calculated only given the current supervised learning model and the sample itself.

Original: The most popular heuristic in this group is uncertainty sampling (Lewis & Gale, 1994;
Revised: The most popular heuristic in this group is uncertainty sampling (Lewis & Gale, 1994;

Original: Scheffer et al., 2001; Shannon, 1948), while many others exist.
Revised: Scheffer et al., 2001; Shannon, 1948), while many others exist.

Original: Diversity sampling: This approach chooses samples which are dissimilar to already labelled samples.
Revised: Diversity sampling: This approach chooses samples which are dissimilar to already labelled samples.

Original: Its features are similarity metrics from a sample to the current labelled set.
Revised: Its features are similarity metrics from a sample to the current labelled set.

Original: It is nearly always combined with uncertainty based sampling.
Revised: It is nearly always combined with uncertainty based sampling.

Original: Representative sampling: This approach, also called density-based sampling, chooses samples which are representative for the whole dataset.
Revised: Representative sampling: This approach, also called density-based sampling, chooses samples which are representative for the whole dataset.

Original: In particular that means that outliers should not be chosen.
Revised: In particular that means that outliers should not be chosen.

Original: Like diversity sampling, it is usually combined with uncertainty sampling.
Revised: Like diversity sampling, it is usually combined with uncertainty sampling.

Original: The combination of features or heuristics from these three disjoint groups is used by many researchers to increase the performance of active learning agents (Wang et al., 2017; Sener & Savarese, 2017; Zhu et al., 2009).
Revised: The combination of features or heuristics from these three disjoint groups is used by many researchers to increase the performance of active learning agents (Wang et al., 2017; Sener & Savarese, 2017; Zhu et al., 2009).

Original: There are many ways to combine heuristics, e.g. it is possible to ﬁrst choose a subset of candidate points using one heuristic and then choose the ﬁnal points to be labelled using the other heuristic, or one could directly deﬁne a heuristic combining two existing ones.
Revised: There are many ways to combine heuristics, e.g. it is possible to ﬁrst choose a subset of candidate points using one heuristic and then choose the ﬁnal points to be labelled using the other heuristic, or one could directly deﬁne a heuristic combining two existing ones.

Original: While our approach with a learned ensemble is the ﬁrst known approach to learn the best combination of heuristics, there exist many ideas in current literature, which could improve the performance of the proposed ensemble even further:
Revised: While our approach with a learned ensemble is the ﬁrst known approach to learn the best combination of heuristics, there exist many ideas in current literature, which could improve the performance of the proposed ensemble even further:

Original: Wang et al.
Revised: Wang et al.

Original: (2017) have added a diversity constraint to uncertainty sampling, and use a Quadratic
Revised: (2017) have added a diversity constraint to uncertainty sampling, and use a Quadratic

Original: Programming approach to enforce this constraint.
Revised: Programming approach to enforce this constraint.

Original: Sener & Savarese (2017) have described active learning as a core-set-selection problem.
Revised: Sener & Savarese (2017) have described active learning as a core-set-selection problem.

Original: Thus, they try to choose samples which are dissimilar both to each other and the labelled set and representative of the unlabelled set.
Revised: Thus, they try to choose samples which are dissimilar both to each other and the labelled set and representative of the unlabelled set.

Original: These two approaches share that they propose advanced non-greedy methods for choosing a batch of samples.
Revised: These two approaches share that they propose advanced non-greedy methods for choosing a batch of samples.

Original: A greedy choice is only a 1 − 1 e -algorithm, as already pointed out by Kirsch et al.
Revised: A greedy choice is only a 1 − 1 e -algorithm, as already pointed out by Kirsch et al.

Original: Adapting our approach such that the samples are chosen non-greedily using the method by Wang et al.
Revised: Adapting our approach such that the samples are chosen non-greedily using the method by Wang et al.

Original: or Sener and Savarese is thus a promising future research direction.
Revised: or Sener and Savarese is thus a promising future research direction.

Original: Zhu et al.
Revised: Zhu et al.

Original: (2009) have proposed a sampling by uncertainty and density approach choosing the samples whose product of their uncertainty and density metric is highest.
Revised: (2009) have proposed a sampling by uncertainty and density approach choosing the samples whose product of their uncertainty and density metric is highest.

Original: This idea can be used to extend the simple linear combination of heuristics we proposed:
Revised: This idea can be used to extend the simple linear combination of heuristics we proposed:

Original: The product of the utility values of two heuristics or any other combination of them can easily be included as the utility of an additional heuristic of the ensemble.
Revised: The product of the utility values of two heuristics or any other combination of them can easily be included as the utility of an additional heuristic of the ensemble.

Original: Ash et al.
Revised: Ash et al.

Original: (2019) have proposed a batch-mode AL agent choosing samples such that their gradients are diverse and used the k-means++ algorithm to choose the batch to be labelled.
Revised: (2019) have proposed a batch-mode AL agent choosing samples such that their gradients are diverse and used the k-means++ algorithm to choose the batch to be labelled.

Original: The diversity of gradients could be added as an additional feature for the ensemble we propose.
Revised: The diversity of gradients could be added as an additional feature for the ensemble we propose.

Original: Active learning agents may perform better if they use different strategies in the beginning than in the end of the active learning episode.
Revised: Active learning agents may perform better if they use different strategies in the beginning than in the end of the active learning episode.

Original: Tang & Huang (2019) have shown that it helps if the active learning agent chooses easier samples in the beginning and harder samples only in the end.
Revised: Tang & Huang (2019) have shown that it helps if the active learning agent chooses easier samples in the beginning and harder samples only in the end.

Original: This idea could also be used to extend our approach, by letting it learn a schedule of policies instead of having a constant policy.
Revised: This idea could also be used to extend our approach, by letting it learn a schedule of policies instead of having a constant policy.

Original: 2. L EARNING ACTIVE LEARNING
Revised: 2. L EARNING ACTIVE LEARNING

Original: As the MDP of active learning is unknown for new tasks and datasets, methods to ﬁnd the optimal policy given this MDP must rely on data.
Revised: As the MDP of active learning is unknown for new tasks and datasets, methods to ﬁnd the optimal policy given this MDP must rely on data.

Original: Thus, dynamic programming is not feasible, instead reinforcement learning algorithms are the suitable method for this problem.
Revised: Thus, dynamic programming is not feasible, instead reinforcement learning algorithms are the suitable method for this problem.

Original: In the sequential active learning setting, the action space is a categorical one-dimensional space allowing to efﬁciently calculate the action maximizing a function given the action.
Revised: In the sequential active learning setting, the action space is a categorical one-dimensional space allowing to efﬁciently calculate the action maximizing a function given the action.

Original: This makes the common reinforcement learning framework of Q-Learning suitable.
Revised: This makes the common reinforcement learning framework of Q-Learning suitable.

Original: It ﬁts a regression function Q(observation, action) to the training data, which tries to predict the reward given the observation for each action.
Revised: It ﬁts a regression function Q(observation, action) to the training data, which tries to predict the reward given the observation for each action.

Original: Its policy π ( observation ) is simply choosing the action maximizing the Q-Value given this observation.
Revised: Its policy π ( observation ) is simply choosing the action maximizing the Q-Value given this observation.

Original: This approach is used by Konyushkova et al.
Revised: This approach is used by Konyushkova et al.

Original: ; Bachman et al.
Revised: ; Bachman et al.

Original: (2017); Fang et al.
Revised: (2017); Fang et al.

Original: Another approach is used by Liu et al.
Revised: Another approach is used by Liu et al.

Original: (2018b;a): They use imitation learning and train the agent to choose the action maximizing the reward directly without estimating the reward.
Revised: (2018b;a): They use imitation learning and train the agent to choose the action maximizing the reward directly without estimating the reward.

Original: Their policy to imitate is a so-called algorithmic expert: It is a policy already knowing the true labels of the samples to choose and is thus able to calculate the improvement of the model if a sample is added to the labelled set.
Revised: Their policy to imitate is a so-called algorithmic expert: It is a policy already knowing the true labels of the samples to choose and is thus able to calculate the improvement of the model if a sample is added to the labelled set.

Original: This equals evaluating the reward function without actually performing the step.
Revised: This equals evaluating the reward function without actually performing the step.

Original: They have applied this approach not only in the sequential, but also in the batch-mode active learning setting: Their agent sequentially chooses the samples to add to the batch.
Revised: They have applied this approach not only in the sequential, but also in the batch-mode active learning setting: Their agent sequentially chooses the samples to add to the batch.

Original: Our approach is also based on this idea of sequential batch-ﬁlling active learning.
Revised: Our approach is also based on this idea of sequential batch-ﬁlling active learning.

Original: The error decay can also be assumed to only depend on the number of samples in a cluster (Chang et al., 2020).
Revised: The error decay can also be assumed to only depend on the number of samples in a cluster (Chang et al., 2020).

Original: They predicted the error decay using a parameterized policy depending only on this single value.
Revised: They predicted the error decay using a parameterized policy depending only on this single value.

Original: Because this policy is mostly a parameterized heuristic, it only needs a small number of samples to be trained on a dataset.
Revised: Because this policy is mostly a parameterized heuristic, it only needs a small number of samples to be trained on a dataset.

Original: Learning batch-mode active learning has also partly been covered by Ravi & Larochelle (2018):
Revised: Learning batch-mode active learning has also partly been covered by Ravi & Larochelle (2018):

Original: They used a Q-learning algorithm to train a regression model to predict the improvement of the accuracy if a sample is added to the labelled set.
Revised: They used a Q-learning algorithm to train a regression model to predict the improvement of the accuracy if a sample is added to the labelled set.

Original: Then they multiply this quality metric with a diversity metric to gain a ﬁnal expert metric.
Revised: Then they multiply this quality metric with a diversity metric to gain a ﬁnal expert metric.

Original: The diversity metric takes the similarity of a sample to other unlabelled samples into account.
Revised: The diversity metric takes the similarity of a sample to other unlabelled samples into account.

Original: The policy is to choose the samples having the highest expert metric.
Revised: The policy is to choose the samples having the highest expert metric.

Original: While the quality metric is a learned metric, it is a design choice to maximize the product of the quality metric and the diversity metric.
Revised: While the quality metric is a learned metric, it is a design choice to maximize the product of the quality metric and the diversity metric.

Original: Thus, this approach can be seen as a mixture of an engineered heuristic and a learning agent.
Revised: Thus, this approach can be seen as a mixture of an engineered heuristic and a learning agent.

Original: Their idea to use not only a combination of heuristics, but in particular a combination of a learned active learning agent and a heuristic can also be used to improve the ensemble we propose: Any learned active learning agent can be added to the set of agents making up the ensemble.
Revised: Their idea to use not only a combination of heuristics, but in particular a combination of a learned active learning agent and a heuristic can also be used to improve the ensemble we propose: Any learned active learning agent can be added to the set of agents making up the ensemble.

Original: 2. E NSEMBLES
Revised: 2. E NSEMBLES

Original: In supervised learning, combinations of several different models can be combined to form an ensemble.
Revised: In supervised learning, combinations of several different models can be combined to form an ensemble.

Original: They combine the output of these models to calculate the ﬁnal output.
Revised: They combine the output of these models to calculate the ﬁnal output.

Original: The combination itself might be learned by another supervised learning algorithm using the predictions of the other models as input features.
Revised: The combination itself might be learned by another supervised learning algorithm using the predictions of the other models as input features.

Original: This approach is called stacking and usually performs better than the best method being part of the ensemble (Wolpert, 1992).
Revised: This approach is called stacking and usually performs better than the best method being part of the ensemble (Wolpert, 1992).

Original: Learning the weights of an ensemble of agents can also be seen as a form of stacking, which further validates our claim that an ensemble of active learning agents performs better than the single best agent being part of it.
Revised: Learning the weights of an ensemble of agents can also be seen as a form of stacking, which further validates our claim that an ensemble of active learning agents performs better than the single best agent being part of it.

Original: While ensembles of supervised learning models are widely used, there has been very little research on ensembles of agents.
Revised: While ensembles of supervised learning models are widely used, there has been very little research on ensembles of agents.

Original: Nonetheless, it was found that ensembles of reinforcement learning algorithms using majority voting and Boltzmann multiplication perform signiﬁcantly better than single algorithms (Wiering & Van Hasselt, 2008).
Revised: Nonetheless, it was found that ensembles of reinforcement learning algorithms using majority voting and Boltzmann multiplication perform signiﬁcantly better than single algorithms (Wiering & Van Hasselt, 2008).

Original: Gao et al.
Revised: Gao et al.

Original: (2018) have proposed to learn AL by learning to choose one of three heuristics at each step of the active learning process.
Revised: (2018) have proposed to learn AL by learning to choose one of three heuristics at each step of the active learning process.

Original: Different to our approach, the heuristics they used and their experiments are restricted to learning network representations.
Revised: Different to our approach, the heuristics they used and their experiments are restricted to learning network representations.

Original: Furthermore, their approach is only applied in the sequential AL setting.
Revised: Furthermore, their approach is only applied in the sequential AL setting.

Original:  F RAMEWORK FOR LEARNING BATCH - MODE ACTIVE LEARNING
Revised:  F RAMEWORK FOR LEARNING BATCH - MODE ACTIVE LEARNING

Original: Our framework for learning batch-mode active learning is split into three parts: First, batch-mode active learning is formalized as a Markov decision process.
Revised: Our framework for learning batch-mode active learning is split into three parts: First, batch-mode active learning is formalized as a Markov decision process.

Original: Second, an ensemble of several different active learning is deﬁned.
Revised: Second, an ensemble of several different active learning is deﬁned.

Original: Third, the parameters of this ensemble are learned using Monte Carlo policy and its objective function and optimizer are deﬁned.
Revised: Third, the parameters of this ensemble are learned using Monte Carlo policy and its objective function and optimizer are deﬁned.

Original: 3. M ARKOV DECISION PROCESS FOR SEQUENTIAL BATCH - FILLING ACTIVE LEARNING
Revised: 3. M ARKOV DECISION PROCESS FOR SEQUENTIAL BATCH - FILLING ACTIVE LEARNING

Original: The goal of every active learning agent is to choose the next batch of samples to be labelled based on features of the current active learning process such that the supervised learning model learns most.
Revised: The goal of every active learning agent is to choose the next batch of samples to be labelled based on features of the current active learning process such that the supervised learning model learns most.

Original: The MDP for pool-based active learning is described by the 4-tuple ( S, A, P, R ) :
Revised: The MDP for pool-based active learning is described by the 4-tuple ( S, A, P, R ) :

Original: The state space S consists of the labelled and unlabelled set of samples and the model trained on the labelled set.
Revised: The state space S consists of the labelled and unlabelled set of samples and the model trained on the labelled set.

Original: As this space is extremely huge and can include millions of variables, it is very hard for an agent to learn from this state.
Revised: As this space is extremely huge and can include millions of variables, it is very hard for an agent to learn from this state.

Original: Thus, we deﬁne an observation space, which includes features of the state which might be relevant to the active learning agent.
Revised: Thus, we deﬁne an observation space, which includes features of the state which might be relevant to the active learning agent.

Original: Common metrics in this space are the predictions of a sample and metrics computed out of it, the uncertainty of the prediction and similarities or distances of unlabelled samples to each other and labelled samples.
Revised: Common metrics in this space are the predictions of a sample and metrics computed out of it, the uncertainty of the prediction and similarities or distances of unlabelled samples to each other and labelled samples.

Original: Most active learning heuristics deﬁne a single metric to rely on and do not combine them.
Revised: Most active learning heuristics deﬁne a single metric to rely on and do not combine them.

Original: The action space A is the categorical choice of one sample or a batch of samples in the unlabelled set.
Revised: The action space A is the categorical choice of one sample or a batch of samples in the unlabelled set.

Original: As the unlabelled set becomes smaller each step, the action space is variable.
Revised: As the unlabelled set becomes smaller each step, the action space is variable.

Original: Following the frameworks for learning sequential active learning (Konyushkova et al., 2017; 2018;
Revised: Following the frameworks for learning sequential active learning (Konyushkova et al., 2017; 2018;

Original: Bachman et al., 2017; Fang et al., 2017)
Revised: Bachman et al., 2017; Fang et al., 2017)

Original: we are using reward shaping (Ng & Jordan, 2003) and set the reward to be the improvement of the model accuracy after a full batch is labelled and the supervised learning model is re-trained on the new labelled set.
Revised: we are using reward shaping (Ng & Jordan, 2003) and set the reward to be the improvement of the model accuracy after a full batch is labelled and the supervised learning model is re-trained on the new labelled set.

Original: This makes the MDP a minimal extension of existing MDPs to batch-mode AL and facilitates comparison with them.
Revised: This makes the MDP a minimal extension of existing MDPs to batch-mode AL and facilitates comparison with them.

Original: Instead of deﬁning the probability P ( s (cid:48) | s, a ) of a new state given the current state and action, we deﬁne a transition function directly mapping the current state and action to the new state.
Revised: Instead of deﬁning the probability P ( s (cid:48) | s, a ) of a new state given the current state and action, we deﬁne a transition function directly mapping the current state and action to the new state.

Original: In the case of active learning this step includes the annotation of a chosen sample or batch of samples, the re-training of the model with it, the change of both the labelled and unlabelled dataset, and the increase of the currently spent annotation budget.
Revised: In the case of active learning this step includes the annotation of a chosen sample or batch of samples, the re-training of the model with it, the change of both the labelled and unlabelled dataset, and the increase of the currently spent annotation budget.

Original: Choosing a batch of samples to be annotated next is a very hard task: If one wants to choose b samples out of a pool of n unlabelled samples, there are (cid:0) nk (cid:1) options.
Revised: Choosing a batch of samples to be annotated next is a very hard task: If one wants to choose b samples out of a pool of n unlabelled samples, there are (cid:0) nk (cid:1) options.

Original: Thus, we set the deﬁnition of a step of one MDP for batch-mode active learning to be the choice of one sample which is added to the batch.
Revised: Thus, we set the deﬁnition of a step of one MDP for batch-mode active learning to be the choice of one sample which is added to the batch.

Original: The corresponding pseudocode for this sequential batch-ﬁlling MDP is shown in Algorithm 1.
Revised: The corresponding pseudocode for this sequential batch-ﬁlling MDP is shown in Algorithm 1.

Original: In each step, the agent chooses the next sample to be added to the batch based on the current observation.
Revised: In each step, the agent chooses the next sample to be added to the batch based on the current observation.

Original: As the observation includes similarity measures of unlabelled samples to samples in the batch, the agent can choose samples which are not too close to samples already in the batch.
Revised: As the observation includes similarity measures of unlabelled samples to samples in the batch, the agent can choose samples which are not too close to samples already in the batch.

Original: While the unlabelled set becomes smaller every step as one sample leaves it and enters the batch, the labelled set is only updated if the batch is full (i.e. the if-condition in the pseudocode is fulﬁlled):
Revised: While the unlabelled set becomes smaller every step as one sample leaves it and enters the batch, the labelled set is only updated if the batch is full (i.e. the if-condition in the pseudocode is fulﬁlled):

Original: Then the labelled set is extended by all samples in the batch, thus all samples in the batch are annotated.
Revised: Then the labelled set is extended by all samples in the batch, thus all samples in the batch are annotated.

Original: As the labelled set was updated, the supervised learning model can be retrained on it, changing the observation signiﬁcantly.
Revised: As the labelled set was updated, the supervised learning model can be retrained on it, changing the observation signiﬁcantly.

Original: Algorithm function stepFunction ( sampleToLabel )
Revised: Algorithm function stepFunction ( sampleToLabel )

Original: D unlabelled ← D unlabelled \ { sampleToLabel } batch ← batch ∪ { sampleToLabel } if | batch | ==
Revised: D unlabelled ← D unlabelled \ { sampleToLabel } batch ← batch ∪ { sampleToLabel } if | batch | ==

Original: batchSize then
Revised: batchSize then

Original: D labelled ← D labelled ∪ batch batch ← {} newAccuracy ← supervisedLearningModel.trainOn ( D labelled ) reward ← newAccuracy − oldAccuracy oldAccuracy ← newAccuracy else reward ←end if observation ← getObservation ( D unlabelled , D labelled , batch ) epochFinished ← | D labelled | + | batch | > = annotationBudget return observation, reward, epochFinished end function
Revised: D labelled ← D labelled ∪ batch batch ← {} newAccuracy ← supervisedLearningModel.trainOn ( D labelled ) reward ← newAccuracy − oldAccuracy oldAccuracy ← newAccuracy else reward ←end if observation ← getObservation ( D unlabelled , D labelled , batch ) epochFinished ← | D labelled | + | batch | > = annotationBudget return observation, reward, epochFinished end function

Original: This deﬁnition of sequential batch-ﬁlling active learning combines the advantages of sequential and batch-mode active learning.
Revised: This deﬁnition of sequential batch-ﬁlling active learning combines the advantages of sequential and batch-mode active learning.

Original: The action per step is still one categorical choice like for sequential active learning, making it much easier for a learning agent to learn which sample to choose next.
Revised: The action per step is still one categorical choice like for sequential active learning, making it much easier for a learning agent to learn which sample to choose next.

Original: As the labelling is only performed for a full batch, the practical advantages of batch-mode learning are still preserved.
Revised: As the labelling is only performed for a full batch, the practical advantages of batch-mode learning are still preserved.

Original: However, there is a theoretical disadvantage:
Revised: However, there is a theoretical disadvantage:

Original: Compared to true sequential learning, the supervised learning model is not trained on the samples just chosen, and thus the features depending on the supervised learning model are not updated.
Revised: Compared to true sequential learning, the supervised learning model is not trained on the samples just chosen, and thus the features depending on the supervised learning model are not updated.

Original: Thus, it can be expected that a sequential batch-ﬁlling agent performs slightly worse than a true sequential agent, with the difference monotonically increasing with the batch size.
Revised: Thus, it can be expected that a sequential batch-ﬁlling agent performs slightly worse than a true sequential agent, with the difference monotonically increasing with the batch size.

Original: One shortcoming of this deﬁnition of the MDP is that the reward is only non-zero if a full batch was labelled.
Revised: One shortcoming of this deﬁnition of the MDP is that the reward is only non-zero if a full batch was labelled.

Original: This makes reinforcement learning approaches like Q-learning difﬁcult to apply, as it is hard to attribute the reward to the choice of a speciﬁc sample, i.e. a single action.
Revised: This makes reinforcement learning approaches like Q-learning difﬁcult to apply, as it is hard to attribute the reward to the choice of a speciﬁc sample, i.e. a single action.

Original: This could be solved by re-training the supervised learning model after each sample added to the batch to calculate a ﬁctive reward during the training of the RL approach.
Revised: This could be solved by re-training the supervised learning model after each sample added to the batch to calculate a ﬁctive reward during the training of the RL approach.

Original: After this, the model has to be set back to the previous state to keep the state and observation independent of this step.
Revised: After this, the model has to be set back to the previous state to keep the state and observation independent of this step.

Original: Such an approach would make it possible to attribute each reward to exactly one sample, however, it has the disadvantage of needing —batch— times more re-trainings of the supervised learning model, thus increasing the computational complexity by the batch size.
Revised: Such an approach would make it possible to attribute each reward to exactly one sample, however, it has the disadvantage of needing —batch— times more re-trainings of the supervised learning model, thus increasing the computational complexity by the batch size.

Original: Hence, Monte Carlo policy search is more effective than approaches like Q-Learning.
Revised: Hence, Monte Carlo policy search is more effective than approaches like Q-Learning.

Original: A general challenge of learning active learning is that each computation of a reward needs the retraining of a supervised learning model, which is computationally very expensive for large models and complex problems.
Revised: A general challenge of learning active learning is that each computation of a reward needs the retraining of a supervised learning model, which is computationally very expensive for large models and complex problems.

Original: This property makes sample-efﬁciency an important criterion for choosing a suitable framework to learn active learning.
Revised: This property makes sample-efﬁciency an important criterion for choosing a suitable framework to learn active learning.

Original: Consequently, Q-Learning and Monte Carlo policy gradient are unsuited methods.
Revised: Consequently, Q-Learning and Monte Carlo policy gradient are unsuited methods.

Original: Model-based RL is also unsuited, because predicting the next state is difﬁcult for two reasons: First, the state space has multiple features for each of many samples and is thus very large.
Revised: Model-based RL is also unsuited, because predicting the next state is difﬁcult for two reasons: First, the state space has multiple features for each of many samples and is thus very large.

Original: Second, predicting the next state essentially equals predicting the outcome of a neural network which is not easier than training the neural network itself.
Revised: Second, predicting the next state essentially equals predicting the outcome of a neural network which is not easier than training the neural network itself.

Original: 3. W EIGHTED ENSEMBLE OF ACTIVE LEARNING AGENTS
Revised: 3. W EIGHTED ENSEMBLE OF ACTIVE LEARNING AGENTS

Original: We propose an active learning policy that is represented by an ensemble of different active learning heuristics.
Revised: We propose an active learning policy that is represented by an ensemble of different active learning heuristics.

Original: The basic idea is to have several different active learning agents which all assign a measure for the utility of choosing a certain data point to be labelled next and each choose the data point with the highest utility.
Revised: The basic idea is to have several different active learning agents which all assign a measure for the utility of choosing a certain data point to be labelled next and each choose the data point with the highest utility.

Original: We evaluate this approach using an ensemble of ﬁve agents:
Revised: We evaluate this approach using an ensemble of ﬁve agents:

Original: The random sampling agent samples the utility of each sample independently from the same Gaussian distribution.
Revised: The random sampling agent samples the utility of each sample independently from the same Gaussian distribution.

Original: The uncertainty sampling agent deﬁnes the utility of a sample as the entropy of
Revised: The uncertainty sampling agent deﬁnes the utility of a sample as the entropy of

Original: its prediction vector.
Revised: its prediction vector.

Original: The diversity sampling agent chooses samples which are dissimilar to samples already chosen for labelling.
Revised: The diversity sampling agent chooses samples which are dissimilar to samples already chosen for labelling.

Original: We deﬁned its utility measure to be the minimum euclidean distance of a sample to a sample already in the labelled or batch set.
Revised: We deﬁned its utility measure to be the minimum euclidean distance of a sample to a sample already in the labelled or batch set.

Original: The representative sampling agent chooses samples which are representative of the whole data distribution.
Revised: The representative sampling agent chooses samples which are representative of the whole data distribution.

Original: In particular it aims not to choose outliers.
Revised: In particular it aims not to choose outliers.

Original: We deﬁned its utility as -

Revised: We deﬁned its utility as -

Original: * the 5 % -percentile of the euclidean distance of a sample to all other samples.
Revised: * the 5 % -percentile of the euclidean distance of a sample to all other samples.

Original: The uncertainty diversity sampling agent deﬁnes its utility as the product of the utilities of the uncertainty and diversity sampling agent (Zhu et al., 2009).
Revised: The uncertainty diversity sampling agent deﬁnes its utility as the product of the utilities of the uncertainty and diversity sampling agent (Zhu et al., 2009).

Original: The ensemble agent chooses the sample having the highest overall utility deﬁned as the weighted sum of the utility measure of each heuristic.
Revised: The ensemble agent chooses the sample having the highest overall utility deﬁned as the weighted sum of the utility measure of each heuristic.

Original: The corresponding function is given in Equation 1.
Revised: The corresponding function is given in Equation 1.

Original: [Equation]
Revised: [Equation]

Original: The beta-parameters could be chosen such that the agent performs exactly like one single heuristic by setting one β to a high value and the others to zero.
Revised: The beta-parameters could be chosen such that the agent performs exactly like one single heuristic by setting one β to a high value and the others to zero.

Original: Thus, assuming this agent learned the best β -parameters for a task, its worst-case performance is the one of the best heuristic on this task, but it can also perform signiﬁcantly better.
Revised: Thus, assuming this agent learned the best β -parameters for a task, its worst-case performance is the one of the best heuristic on this task, but it can also perform signiﬁcantly better.

Original: We do not claim that an ensemble made of these four simple heuristics outperforms state-of-the-art batch-mode active learning agents.
Revised: We do not claim that an ensemble made of these four simple heuristics outperforms state-of-the-art batch-mode active learning agents.

Original: We do claim, however, that the ensemble performs at least as good as all agents it contains, but can perform signiﬁcantly better.
Revised: We do claim, however, that the ensemble performs at least as good as all agents it contains, but can perform signiﬁcantly better.

Original: Thus, any state-of-the-art active learning agent can be added to the set of agents making up the ensemble, which allows the ensemble to further improve the performance of this agent by combining it with others.
Revised: Thus, any state-of-the-art active learning agent can be added to the set of agents making up the ensemble, which allows the ensemble to further improve the performance of this agent by combining it with others.

Original: 3. C HOICE OF PARAMETERS OF ENSEMBLE
Revised: 3. C HOICE OF PARAMETERS OF ENSEMBLE

Original: The linear combination of active learning agents is a policy parameterized by the β -parameters.
Revised: The linear combination of active learning agents is a policy parameterized by the β -parameters.

Original: We decided to learn which parameters are good using Monte Carlo policy search.
Revised: We decided to learn which parameters are good using Monte Carlo policy search.

Original: The objective is to maximize the performance of the supervised learning model after a complete episode.
Revised: The objective is to maximize the performance of the supervised learning model after a complete episode.

Original: The objective function is deﬁned in Eqn.
Revised: The objective function is deﬁned in Eqn.

Original: 2 and Algorithm 

Revised: 2 and Algorithm

Original: and calculates the performance of one choice of β .
Revised: and calculates the performance of one choice of β .

Original: Taking the mean of several active learning runs reduces the noise of the objective function.
Revised: Taking the mean of several active learning runs reduces the noise of the objective function.

Original: It is very hard to attribute the performance at the end of an episode to the choice of a speciﬁc action in a speciﬁc state or to the choice of β .
Revised: It is very hard to attribute the performance at the end of an episode to the choice of a speciﬁc action in a speciﬁc state or to the choice of β .

Original: Thus we do not solve the optimization problem using policy gradients, but treat it as a black-box function or hyperparameter optimization problem instead.
Revised: Thus we do not solve the optimization problem using policy gradients, but treat it as a black-box function or hyperparameter optimization problem instead.

Original: Methods like Bayesian optimization, random search, grid search, tree parzen estimators, or population-based training could be used to ﬁnd the optimal solution.
Revised: Methods like Bayesian optimization, random search, grid search, tree parzen estimators, or population-based training could be used to ﬁnd the optimal solution.

Original: Among them, we chose tree parzen estimators (Bergstra et al., 2013) as they are very sample-efﬁcient and search for the global optimum.
Revised: Among them, we chose tree parzen estimators (Bergstra et al., 2013) as they are very sample-efﬁcient and search for the global optimum.

Original: objectiveFunction ( β ) : R n → R with n = number of heuristics making up the ensemble (2)
Revised: objectiveFunction ( β ) : R n → R with n = number of heuristics making up the ensemble (2)

Original: Algorithm 2 Pseudocode for the objective function of the ensemble agent
Revised: Algorithm 2 Pseudocode for the objective function of the ensemble agent

Original: Require: trainingTask, activeLearningEnvironment, EnsembleAgent function objectiveFunction ( β ) agent ← EnsembleAgent ( β ) for i ← 1 to numberEpisodes do results ← activeLearningEnvironment.run ( trainingTask, agent ) performances [ i ] ← results.getFinalPerformance () end for meanPerformance ← mean ( performances ) return meanPerformance end function
Revised: Require: trainingTask, activeLearningEnvironment, EnsembleAgent function objectiveFunction ( β ) agent ← EnsembleAgent ( β ) for i ← 1 to numberEpisodes do results ← activeLearningEnvironment.run ( trainingTask, agent ) performances [ i ] ← results.getFinalPerformance () end for meanPerformance ← mean ( performances ) return meanPerformance end function

Original:  E XPERIMENTS AND RESULTS 4. S ETTINGS
Revised:  E XPERIMENTS AND RESULTS 4. S ETTINGS

Original: We compare our ensemble of ﬁve heuristics with each of these ﬁve heuristics and an ensemble not using the learned weights, but setting all weighs to the same value.
Revised: We compare our ensemble of ﬁve heuristics with each of these ﬁve heuristics and an ensemble not using the learned weights, but setting all weighs to the same value.

Original: We set the annotation budget such that the performance differences between the agents are clearly visible and set the batch size such that between 3 and 5 iterations could be performed.
Revised: We set the annotation budget such that the performance differences between the agents are clearly visible and set the batch size such that between 3 and 5 iterations could be performed.

Original: The starting set size was set to be proportional to the number of classes.
Revised: The starting set size was set to be proportional to the number of classes.

Original: For training the β -parameters, we ﬁxed the β -parameter for random sampling at 1 and chose a lognormal distribution with a mean of 1 and a standard deviation of 2 as prior for the other parameters.
Revised: For training the β -parameters, we ﬁxed the β -parameter for random sampling at 1 and chose a lognormal distribution with a mean of 1 and a standard deviation of 2 as prior for the other parameters.

Original: The later parts of this section show the performance of the active learning agent on the tasks as plots of the classiﬁcation accuracy over the number of labelled samples.
Revised: The later parts of this section show the performance of the active learning agent on the tasks as plots of the classiﬁcation accuracy over the number of labelled samples.

Original: The plots for the different evaluation tasks all share the same structure:
Revised: The plots for the different evaluation tasks all share the same structure:

Original: The dark shaded area around the curves of each agent report the 95% conﬁdence intervals, the light shaded areas are the standard deviations.
Revised: The dark shaded area around the curves of each agent report the 95% conﬁdence intervals, the light shaded areas are the standard deviations.

Original: A sufﬁx to the agent names in the form of ’ batchSize’ denotes the batch size the agent used.
Revised: A sufﬁx to the agent names in the form of ’ batchSize’ denotes the batch size the agent used.

Original: There is no such sufﬁx for the random, diversity and representative sampling agent, as their features are independent of the batch.
Revised: There is no such sufﬁx for the random, diversity and representative sampling agent, as their features are independent of the batch.

Original: 4. E XPERIMENTS ON UCI DATASETS
Revised: 4. E XPERIMENTS ON UCI DATASETS

Original: We used 11 datasets from the UCI machine learning repository (Dua & Graff, 2017).
Revised: We used 11 datasets from the UCI machine learning repository (Dua & Graff, 2017).

Original: The datasets 2breast cancer, 3-diabetes, 4-ﬂare solar, 5-heart, 6-german, 7-mushrooms, 8-waveform, 9-wdbc were used for training and 0-adult, 1-australian, and 10-spam for evaluation, similar to Konyushkova et al.
Revised: The datasets 2breast cancer, 3-diabetes, 4-ﬂare solar, 5-heart, 6-german, 7-mushrooms, 8-waveform, 9-wdbc were used for training and 0-adult, 1-australian, and 10-spam for evaluation, similar to Konyushkova et al.

Original: The objective function was set to be the harmonic mean of the ﬁnal accuracies the ensemble reached on each of the 7 training tasks.
Revised: The objective function was set to be the harmonic mean of the ﬁnal accuracies the ensemble reached on each of the 7 training tasks.

Original: This allows training on multiple datasets easily and thus improves generalization.
Revised: This allows training on multiple datasets easily and thus improves generalization.

Original: The results for the 1-australian evaluation task is shown in Figure 2, the results for the other evaluation tasks are quite similar.
Revised: The results for the 1-australian evaluation task is shown in Figure 1, the results for the other evaluation tasks are quite similar.

Original: 4. E XPERIMENTS ON CHECKERBOARD DATASETS
Revised: 4. E XPERIMENTS ON CHECKERBOARD DATASETS

Original: The datasets checkerboard 2x2 and rotated checkerboard 2x2 were used as training tasks to train the learning agents, the checkerboard 4x4 task was used as evaluation task.
Revised: The datasets checkerboard 2x2 and rotated checkerboard 2x2 were used as training tasks to train the learning agents, the checkerboard 4x4 task was used as evaluation task.

Original: All checkerboard tasks are
Revised: All checkerboard tasks are

Original: XOR-like binary classiﬁcation tasks with two-dimensional input data.
Revised: XOR-like binary classiﬁcation tasks with two-dimensional input data.

Original: We used a random forest as a classiﬁer and the results are shown in Fig.
Revised: We used a random forest as a classiﬁer and the results are shown in Fig.

Original: 4. E XPERIMENTS ON ( FASHION -) MNIST TASKS
Revised: 4. E XPERIMENTS ON ( FASHION -) MNIST TASKS

Original: The image classiﬁcation tasks use the MNIST (LeCun et al., 2010) and fashion-MNIST (Xiao et al., 2017) dataset and a three-layer convolutional neural network implemented in keras (Chollet et al., 2015a) as classiﬁer.
Revised: The image classiﬁcation tasks use the MNIST (LeCun et al., 2010) and fashion-MNIST (Xiao et al., 2017) dataset and a three-layer convolutional neural network implemented in keras (Chollet et al., 2015a) as classiﬁer.

Original: Because Munjal et al.
Revised: Because Munjal et al.

Original: (2020) have shown that performance differences between different active learning agents may vanish when more regularization techniques are used, we tried to maximize the performance of the random baseline using following techniques: random data augmentation, dropout, L1- and L2 regularization and hyperparameter optimization using tree parzen estimators (Bergstra et al., 2013).
Revised: (2020) have shown that performance differences between different active learning agents may vanish when more regularization techniques are used, we tried to maximize the performance of the random baseline using following techniques: random data augmentation, dropout, L1- and L2 regularization and hyperparameter optimization using tree parzen estimators (Bergstra et al., 2013).

Original: We did not use more advanced networks like the ResNet or VGG16 because they are computationally much more expensive and tend to overﬁt their huge number of parameters on the very small training set size.
Revised: We did not use more advanced networks like the ResNet or VGG16 because they are computationally much more expensive and tend to overﬁt their huge number of parameters on the very small training set size.

Original: Instead of calculating the distance metrics using the raw images, we used tSNE embeddings (Maaten & Hinton, 2008).
Revised: Instead of calculating the distance metrics using the raw images, we used tSNE embeddings (Maaten & Hinton, 2008).

Original: We trained the ensemble agent on the MNIST task and evaluated it on the fashion-MNIST task, the results are shown in
Revised: We trained the ensemble agent on the MNIST task and evaluated it on the fashion-MNIST task, the results are shown in

Original: Fig.
Revised: Fig.

Original: 4. R ESULTS ON QUESTION ANSWERING TASKS
Revised: 4. E XPERIMENTS ON QUESTION ANSWERING TASKS

Original: The bAbI dataset (Weston et al., 2015) is a collection of question answering datasets of which two different challenges are used: one with a single supporting fact and one with two supporting facts.
Revised: The bAbI dataset (Weston et al., 2015) is a collection of question answering datasets of which two different challenges are used: one with a single supporting fact and one with two supporting facts.

Original: The classiﬁer used is using a long short term memory and  taken from a keras example, which follows Sukhbaatar et al.(2015).
Revised: The classiﬁer used is using a long short term memory and is taken from a keras example, which follows Sukhbaatar et al.

Original: Like for the image classiﬁcation tasks, the hyperparameters were optimized to minimize the loss on a small random subset.
Revised: Like for the image classiﬁcation tasks, the hyperparameters were optimized to minimize the loss on a small random subset.

Original: We used the bAbI task with a single supporting fact as a training task and the one with two supporting facts as an evaluation task.
Revised: We used the bAbI task with a single supporting fact as a training task and the one with two supporting facts as an evaluation task.

Original: For calculating the distances, we used word2vec embeddings (Mikolov et al., 2015).
Revised: For calculating the distances, we used word2vec embeddings (Mikolov et al., 2015).

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Revised: 4. A NALYSIS

Revised: The performance of the ensemble on the checkerboard and bAbI task was found to equal the one of the best heuristic, whereas it showed a signiﬁcant performance increase compared to the best heuristic on the UCI and fashion-MNIST task.

Revised: The ensemble with all weights set to 1 performed equal or worse than the one with learned weights, which shows the importance of learning the optimal weights on a training task.

Original:  I NTERPRETATION OF LEARNED PARAMETERS
Revised: Our approach has also shown to have the same behaviour with respect to varying batch sizes as uncertainty sampling, see Fig. 3: It performs better with smaller batch sizes and slightly worse with larger ones. The effect of larger batch sizes depends on the weight of uncertainty and uncertainty diversity sampling, as the other two heuristics do not depend on the batch size at all. 4. I NTERPRETATION OF LEARNED PARAMETERS

Original: The ensemble learns the weights of the agents being part of it.
Revised: The ensemble learns the weights of the agents being part of it.

Original: These parameters can be easily interpreted and thus allow us to understand the ensemble’s policy better.
Revised: These parameters can be easily interpreted and thus allow us to understand the ensemble’s policy better.

Original: Furthermore they might be used to ﬁnd out which kind of samples could be additionally generated, if possible.
Revised: Furthermore they might be used to ﬁnd out which kind of samples could be additionally generated, if possible.

Original: The weights learned for the four different tasks or tasks groups are given in Table 4.6.
Revised: The weights learned for the four different tasks or tasks groups are given in Table 4.7.

Original: [Table]
Revised: The weights were normalized by the sum of all weights, allowing a better comparison of the weights learned on the different tasks.

Original: For the UCI datasets, diversity sampling is very unimportant, which is indicated by the weight of diversity sampling being nearly zero.
Revised: For the UCI datasets, diversity sampling is very unimportant, which is indicated by the weight of diversity sampling being nearly zero.

Original: Thus, it can be assumed that the samples in the UCI datasets are already quite dissimilar to each other.
Revised: Thus, it can be assumed that the samples in the UCI datasets are already quite dissimilar to each other.

Original: The ensemble trained on the checkerboard datasets has a weight of representative sampling being zero, which can be well explained by the fact that it does  not contain any outliers.
Revised: The ensemble trained on the checkerboard datasets has a weight of representative sampling being zero, which can be well explained by the fact that it does [Table] not contain any outliers.

Original: Both MNIST and bAbI have much more samples in the unlabelled set, explaining why representative sampling is quite important for them.
Revised: Both MNIST and bAbI have much more samples in the unlabelled set, explaining why representative sampling is quite important for them.

Original: The weights differ a lot between the different training tasks, indicating the importance of training the parameters of the ensemble on a similar dataset it is later applied on.a
Revised: The weights differ a lot between the different training tasks, indicating the importance of training the parameters of the ensemble on a similar dataset it is later applied on.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we proposed a novel approach for active learning using an ensemble of different active learning heuristics or trained agents.
Revised: In this paper, we proposed a novel approach for active learning using an ensemble of different active learning heuristics or trained agents.

Original: By learning the weights of a linear combination of different agents in the ensemble, this approach can be easily adapted to different datasets and supervised learning models.
Revised: By learning the weights of a linear combination of different agents in the ensemble, this approach can be easily adapted to different datasets and supervised learning models.

Original: To train the ensemble we formalized batch-mode active learning as a Markov decision process.
Revised: To train the ensemble we formalized batch-mode active learning as a Markov decision process.

Original: Experiments in different domains and using different supervised learning models have shown the effectiveness of our approach:
Revised: Experiments in different domains and using different supervised learning models have shown the effectiveness of our approach:

Original: As expected, the worst-case performance of the ensemble is the one of the best heuristic, while it outperformed the best one signiﬁcantly on some datasets.
Revised: As expected, the worst-case performance of the ensemble is the one of the best heuristic, while it outperformed the best one signiﬁcantly on some datasets.

Original: Furthermore, it was shown that it is very important to adapt the active learning agent to the dataset it is applied on.
Revised: Furthermore, it was shown that it is very important to adapt the active learning agent to the dataset it is applied on.

Original: The most natural extension of our ensemble is to make it up with more and more advanced active learning agents.
Revised: The most natural extension of our ensemble is to make it up with more and more advanced active learning agents.

Original: Additionally, we assume following future research directions to be very promising:
Revised: Additionally, we assume following future research directions to be very promising:

Original: First, different agent weights could be learned for different time points of the episode, allowing to change the policy with increasing accuracy.
Revised: First, different agent weights could be learned for different time points of the episode, allowing to change the policy with increasing accuracy.

Original: Second, the weights could be learned on a small subset of the dataset it is later applied on.
Revised: Second, the weights could be learned on a small subset of the dataset it is later applied on.

Original: This allows its application on tasks where no similar training task is available or where the ensemble generalizes badly between the training and application task.
Revised: This allows its application on tasks where no similar training task is available or where the ensemble generalizes badly between the training and application task.

Original: A A PPENDIX A. D ETAILS OF CLASSIFIERS
Revised: A A PPENDIX A. D ETAILS OF CLASSIFIERS

Original: The classiﬁer used for both the UCI and the checkerboard datasets is the same used by Konyushkova et al.
Revised: The classiﬁer used for both the UCI and the checkerboard datasets is the same used by Konyushkova et al.

Original: A random forest implemented with scikit-learn (Pedregosa et al., 2011) and 50 estimators.
Revised: A random forest implemented with scikit-learn (Pedregosa et al., 2011) and 50 estimators.

Original: The other hyperparameters were set to the default value.
Revised: The other hyperparameters were set to the default value.

Original: We chose this classiﬁer for easier comparison with the work by Konyushkova et al.
Revised: We chose this classiﬁer for easier comparison with the work by Konyushkova et al.

Original: The classiﬁer used for the (fashion-)MNIST task is a CNN with two convolutional layers, one intermediate dense layer and a softmax output layer.
Revised: The classiﬁer used for the (fashion-)MNIST task is a CNN with two convolutional layers, one intermediate dense layer and a softmax output layer.

Original: Intermediate dropout and max-pooling layers were added.
Revised: Intermediate dropout and max-pooling layers were added.

Original: The exact structure and the setting of hyperparameters can be found in the additional material in /supervised_learning_tasks/tasks_vision/task_Vision_CNN.py:108ff .
Revised: The exact structure and the setting of hyperparameters can be found in the additional material in /supervised_learning_tasks/tasks_vision/task_Vision_CNN.py:108ff .

Original: We used random data augmentation.
Revised: We used random data augmentation.

Original: We had also tried the ResNet16 architecture, but found it to have worse performance, probably because its large number of parameters was overﬁtting on the few hundred images it was trained on.
Revised: We had also tried the ResNet16 architecture, but found it to have worse performance, probably because its large number of parameters was overﬁtting on the few hundred images it was trained on.

Original: Furthermore, it was computationally much more expensive.
Revised: Furthermore, it was computationally much more expensive.

Original: We chose a CNN with data augmentation as classiﬁer as it is the state-of-the-art architecture for image classiﬁcation.
Revised: We chose a CNN with data augmentation as classiﬁer as it is the state-of-the-art architecture for image classiﬁcation.

Original: The classiﬁer used for the bAbI task is a memory network (Sukhbaatar et al., 2015).
Revised: The classiﬁer used for the bAbI task is a memory network (Sukhbaatar et al., 2015).

Original: The code is largely based on a keras example (Chollet et al., 2015b).
Revised: The code is largely based on a keras example (Chollet et al., 2015b).

Original: We chose this architecture as it provides state-of-the-art performance.
Revised: We chose this architecture as it provides state-of-the-art performance.

Original: A. P ERFORMANCE OF AL AGENTS WHEN TRAINED ON THE FULL DATASET
Revised: A. P ERFORMANCE OF AL AGENTS WHEN TRAINED ON THE FULL DATASET

Revised: As shown in Fig 2, the worst agents need about 3 times more samples to reach the same accuracy as the best agents.

Revised: They also need much longer to reach almost the same accuracy as trained on the full dataset.

Revised: A. P ERFORMANCE OF E NSEMBLE WITH OTHER BATCH SIZES

Revised: The plots in Fig.

Revised: 3 include the ensemble with both a lower and a higher batch size than the one it was trained on.

Revised: Across all task, the performance of the ensemble is monotonically decreasing with the batch size.

Revised: On the UCI dataset, the ensemble performs always slightly better than uncertainty sampling with the same batch size.

Revised: On the checkerboard task, the ensembles with a batch size of 2 and 32 respectively have almost the same peformance, the one with a batch size of 160 only performs slightly worse.

Revised: On the fashion-MNIST task, the ensemble with a batch size of 16 performs much better than the ones with bigger batch sizes.

Revised: On the bAbI task, the ensemble had learned to rely almost completely on the representative sampling agent, whose features are independent of the batch size, thus the ensemble’s performance is also independent of the batch size.

Revised: A. D ETAILS OF TRAINING TASKS

Revised: The ensemble trained on the UCI datasets 2-breast-cancer through 9-wdbc was trained with a starting size of 8, an annotation budget of 32, a batch size of 4 and the objective was set to the geometric mean of 64 episodes (8 per dataset).

Revised: The ensemble trained on the checkerboard datasets was trained with a starting size of 8, an annotation budget of 104, a batch size of 32 and the objective was set to the geometric mean of 64 episodes (32 per dataset).

Revised: The ensemble trained on the MNIST dataset was trained with a starting size of 40, an annotation budget of 296, a batch size of 64 and the objective was set to the arithmetic mean of 2 episodes.

Revised: The ensemble trained on the bAbI - single supporting fact dataset was trained with a starting size of 40, an annotation budget of 296, a batch size of 64 and the objective was set to the arithmetic mean of 2 episodes.

Revised: The starting size was set to 4 times the number of classes for each task.

Revised: The annotation budget and batch size were set such that the accuracy does not converge yet and 3 to 4 batches are chosen per episode.

Revised: The number of episodes run for each evaluation of the objective function was set such that the variance of the objective is approximately the same across all tasks.

Revised: A geometric mean was chosen if multiple datasets were combined for training, so that the difﬁculty of each dataset does not inﬂuence its weight relative to the other ones.

Revised: Otherwise the arithmetic mean was chosen.

Revised: A. D IFFICULTY OF FINDING THE OPTIMAL BETA - PARAMETERS

Revised: The question how likely it is, that the optimal β -parameters are found, can be broken down into to subquestions:

Revised: • What does the objective function look like?

Revised: The dataset, classiﬁer and set of heuristics to choose from determine the form of the objective function mapping the β -parameters to the ﬁnal accuracy of the classiﬁer after a complete episode.

Revised: Important characteristics of this function are how noisy it is and whether it contains local minima or saddle points making it

Revised: harder to ﬁnd its optimum.

Revised: We found for all four training tasks, that the objective function is very noisy, but does not contain any local minima.

Revised: • How likely is it, that a black-box optimizer ﬁnds the optimum of the objective function?

Revised: Given the fact that only 4 continuous parameters can be optimized and the simple structure of the objective function, it can be expected that most black-box optimizers can ﬁnd an optimum quite easily.

Revised: However, they should be able to handle the noisy nature of the objective function.

Revised: A.5. O BJECTIVE FUNCTION OF UCI TASK

Revised: The plots in Fig.

Revised: 4 show the datapoints of many random evaluations of the objective function of the

Revised: UCI task.

Revised: The β -parameters are again normalized by their sum.

Revised: The four smaller plots show the accuracy over each of the 4 β -parameters and include a ﬁtted polynomial of second degree.

Revised: It shows that high weights for uncertainty sampling should be preferred, which ﬁts well to the fact, that the optimizer found that uncertainty sampling should be assigned the highest weight.

Revised: The larger graph shows the accuracy over three of the four parameters.

Revised: For all tasks, additional plots showing the accuracy over two parameters can be found in the supplementary material.

Revised: A.5. O BJECTIVE FUNCTION OF CHECKERBOARD TASKS

Revised: The plots in Fig.

Revised: 5 show the datapoints of many random evaluations of the objective function of the checkerboard tasks.

Revised: It is easily visible, that assigning higher weights to Diversity and UncertaintyDiversity sampling increases the performance.

Revised: This further validates that the parameters found by the optimizer are indeed good.

Revised: A.5. O BJECTIVE FUNCTION OF MNIST TASK

Revised: The plots in Fig.

Revised: 6 show the datapoints of many random evaluations of the objective function of the

Revised: MNIST task.

Revised: It is clearly visible that uncertainty sampling should be assigned a low weight.

Revised: A.5. O BJECTIVE FUNCTION OF B A B I TASK

Revised: The plots in Fig.

Revised: 7 show the datapoints of many random evaluations of the objective function of the bAbI task.

Revised: It is clearly visible that representative sampling performs best.