Meta-Learning Transferable Active Learning Policies by Deep Reinforcement Learning
Anonymous authors
Paper under double-blind review
Abstract
Active learning (AL) aims to enable training high performance classiﬁers with low annotation cost by predicting which subset of unlabelled instances would be most beneﬁcial to label.
The importance of AL has motivated extensive research, proposing a wide variety of manually designed AL algorithms with diverse theoretical and intuitive motivations.
In contrast to this body of research, we propose to treat active learning algorithm design as a meta-learning problem and learn the best criterion from data.
We model an active learning algorithm as a deep neural network that inputs the base learner state and the unlabelled point set and predicts the best point to annotate next.
Training this active query policy network with reinforcement learning, produces the best non-myopic policy for a given dataset.
The key challenge in achieving a general solution to AL then becomes that of learner generalisation, particularly across heterogeneous datasets.
We propose a multi-task dataset-embedding approach that allows dataset-agnostic active learners to be trained.
Our evaluation shows that AL algorithms trained in this way can directly generalize across diverse problems.
Introduction
In many applications, supervision is costly relative to the volume of data.
In these settings active query selection methods can be invaluable to predict which instances a base classiﬁer would ﬁnd it informative to label.
By carefully choosing the training data, the classiﬁer can perform well even with relatively sparse supervision.
This vision has motivated a large body of work in active learning that has collectively proposed dozens of query criteria based on diﬀerent theoretical or intuitive motivations, such as margin (Tong & Koller, 2002) and uncertainty-based (Kapoor et al., 2007) sampling, expected error reduction (Roy &
McCallum, 2001), representative and diversity-based (Chattopadhyay et al., 2012) sampling, or combinations thereof (Hsu & Lin, 2015).
It is hard to pick a clear winner all these methods, because each is based on a reasonable and appealing – but completely diﬀerent – motivation; and there is no consistent winner in terms of performance across all datasets.
Rather than hand-designing a criterion and hoping that it performs well, we take a datadriven learning-based approach.
We treat active learning algorithm development as a metalearning problem and train an active learning policy represented by a neural network using deep reinforcement learning (DRL).
It is natural to represent AL as a sequential decision making problem since each action (queried point) aﬀects the context (available query points, state of the base learner) successively for the next decision.
In this way the active query policy trained by RL can potentially learn a powerful and non-myopic policy.
By treating the increasing accuracy of the base learner as the reward, we optimise for the actual goal: the accuracy of a classiﬁer with a small number of labels.
As the class of deep neural network (DNN) models we use includes many classic criteria as special cases, we can expect this approach should be at least as good as existing methods and likely better due to exploiting more information and non-myopic optimisation of the actual evaluation metric.
This idea of learning the best criterion within a very general function class is appealing, and other very recent research has had similar inspiration (Bachman et al., 2017).
However it does not provide a general solution to AL unless the learned criterion generalizes across diverse datasets/learning problems.
With DRL we can likely learn an excellent query policy for any given dataset.
But this is not necessarily useful alone: if we had the labels required to train the policy on a speciﬁc problem, we would not need to do AL on that problem in the ﬁrst place.
Thus the research question for AL moves from “what is a good criterion?”
to “how to learn a criterion that generalizes?”.
In this paper we investigate how to train
AL query criteria that generalize across tasks/datasets.
Our approach is to deﬁne a DNN query criterion policy that is paramaterized by a dataset embedding .
By multi-task training of our DNN policy on a diverse batch of source tasks/datasets, the network learns how to calibrate its strategy according to the statistics of a given dataset.
Speciﬁcally we are inspired by the recently proposed auxiliary network idea (Romero et al., 2016) to deﬁne a meta-network that provides paramaterized domain adaptation.
The meta network generates a dataset embedding and produces the weight matricies that parameterize the main policy.
Besides enabling the policy to adapt to datasets with diﬀerent statistics, this also means that our policy beneﬁts from end-to-end processing of raw features while being transferrable to datasets of any feature space dimensionality.
Finally, unlike Woodward & Finn (2017);
Bachman et al.
our framework is agnostic to the base classiﬁer.
Treating the underlying learner as part of the environment to be optimised means our framework can be applied to improve the label eﬃciency of any existing learning architecture or algorithm.
Preliminaries
Reinforcement Learning (RL)
In a general model-free reinforcement learning setting, an agent interacts with an environment E over a number of discrete time step t .
At each time step, the agent receives the state s t ∈ S from environment and select an action a t ∈ A based on its policy π ( a t
| s t ) which is a mapping from state to action.
Then, agent will receive a new states s t +1 and immediate reward r t from E .
The aim of RL is to maximise the return
R = (cid:80) ∞ t =1 γ t − 1 r t where the return is the accumulated immediate reward from time step t with discount factor γ ∈ (0 , 1].
There are multiple approaches to learning the policy π (Kober & Peters, 2009; Mnih et al., 2015).
We use direct policy search based RL, which learns π by gradient ascent on the objective function J π ( θ )
=
(cid:80) s ∈S d ( s ) (cid:80) a ∈A π θ ( a | s ) R , where d ( s ) is stationary distribution of Markov chain for π θ .
Active Learning (AL)
A dataset D = { ( x i , y i ) }
Ni =1 contains N instances x i ∈ R D and labels y i ∈ { 1 , 2 } , most or all of which are unknown in advance.
In active learning, at any moment the data is split between a labelled set
L and unlabelled set U
= D \ L where |L| (cid:28) |U| and a classiﬁer f has been trained on L so far.
In each iteration, a pool-based active learner τ selects an instance from unlabelled pool
U to query its label τ : { ( L , U , f ) → i } , where i ∈ { 1 , . .
. , |U|} .
Then the selected instance i is removed from the unlabeled set
U and added to the labeled set L along with its label, and the classiﬁer f is retrained based on the updated L .
Connection between RL and AL
In order to go beyond the many existing heuristic criteria, we propose to model an active learning algorithm as a neural network, and formalize discovery of the ideal criterion as a deep reinforcement learning problem.
Let the state of the world s t consist of a featurisation of the dataset and the state of the base classiﬁer s t =
{L t , U t , f } .
Let an active learning criterion be a policy π ( a i | s ) where index of actions i ∈ { 1 , . .
. , |U|} select a point in unlabelled set to query.
Upon querying a point the world state is updated to s t +
as that point is moved from U to L and f is updated as the base classiﬁer is retrained.
Assume the policy is a neural network paramaterized by weights θ , that selects actions as π ( a i | s t )
∝ exp Φ θ ( a i ,s t ) , where i ∈ { 1 , . .
. , |U|} is the index of the unlabelled instances.
Finally, we deﬁne the reward of an episode to be the quantity we wish to maximize.
E.g., If the budget is N queries and we only care about the accuracy after the N th query, then we let R =
Acc N where Acc N is the accuracy after the N th query.
Alternatively, if we care about the performance during all the N queries, we can use R =
(cid:80)
Nt =1 γ t − 1 Acc t .
(This illustrates an important advantage of the learning active learning approach: we can tune the learned criterion to suit the requirements of the AL application.)
By training θ to maximize the objective J ( θ ) we obtain the optimal active learning policy.
In interpreting AL criterion learning as a DRL problem, there is the special consideration that unlike general RL problems, each action can only be chosen once in an
[Figure]
episode.
We will achieve this by deﬁning a fully convolutional policy network architecture where the dimensionality of the output softmax π ( a i | s t ) can vary with t .
Methods
Recall that our aim is to obtain the parameters θ of an eﬀective dataset-agnostic active query policy π θ ( a | s ).
The two key challenges are how to learn such a policy given that: (i) the testing dataset statistics may be diﬀerent from training dataset statistics, and moreover (ii) diﬀerent datasets have diﬀerent feature dimensionality d .
This challenge is addressed by deﬁning the overall policy
π θ ( a | s ) in terms of two sub-networks – a policy network and meta network – described as follows.
Policy Network
Overall the policy network π inputs all N unlabelled instance Z u ∈ R N × d and its output is an N -way softmax distribution for selecting choice of instance to query.
We assume the policy models actions via the softmax π ( a i | s ) ∝ exp Φ θp ( W Te z i ) , where z i ∈ R
d is the i th unlabelled instance in Z u and W e ∈ R d × k encodes the pool of instances.
Although dimensionality d varies by dataset, the encoding u i =
W Te z i ∈
R k does not, so the rest of the policy network π ( a i | s )
∝ exp Φ θp ( u i ) is independent of dataset dimension.
The key is then how to obtain encoder W e which will be provided by the meta network.
Following previous work (Bachman et al., 2017; Konyushkova et al., 2017) we also allow the instances to be augmented by instance-level expert features so Z
=
[ X , ξ ( X )] where X are the raw instances and ξ ( X ) are the expert features of each raw instance.
Meta Network
The encoding parameters W e of the policy network is obtained from the meta network:
Ψ θ em : { ( L , U , f ) → W e ; θ em } .
The meta network inputs a featurisation of
L , U and f and produces W e ∈ R d × k to allow the policy network to process d -dimensional inputs into a ﬁxed k -dimnesional hidden representation.
Following Romero et al.
we also synthesise and use the W d ∈ R k × d dimensional decoder Ψ θ dm : { ( L , U , f ) → W d ; θ dm } to regularise theis process by reconstructing the input features.
The meta network synthesises these weight matricies based on dataset-embeddings of Z T described in the following section.
3. Achieving Cross Dataset Generalization
The idea of auxiliary networks to predict weights for a target network was recently used in Romero et al.
There the auxiliary network inputs an embedding of X T and predicts the weights for a main network that inputs X , with the purpose of reducing the total number of parameters if X is high dimensional.
In Romero et al.
all the training and testing is performed on the same dataset.
Here we are inspired by this idea in proposing a meta-network strategy for achieving end-to-end learning of multiple-domains.
By multitask training on multiple datasets, the meta-network learns to generate dataset-speciﬁc weights for the policy network such that it performs eﬀectively on all training problems and generalises well to new testing problems based on their embedding.
Dimension Embedding Strategy
The auxiliary meta-network requires a feature embedding that produces a ﬁxed size description of each dimension across all datasets.
The meta network takes ( L , U , f ) as input, treating each feature as an example.
It extracts an embedding from each input (feature) and then predicts the policy network’s weights for the corresponding feature.
All together, the auxiliary network predicts the weight matrix W e ∈ R d × k , which the policy network can use to map each feature dimension to a k dimensional embedding, as
[Equation]
Here e is a non-linear feature embedding, j indexes features, selecting the j th embedded feature and the j th row of W e , and Ψ is the non-linear mapping of the meta-network, which outputs a vector of dimension k .
Similarly, the meta-network also predicts the weight matrix
W d used for auto-encoding reconstruction (Fig 1).
Although d is dataset dependent, the meta network generates weights for a policy network of appropriate dimensionality ( d × k ) to the target problem.
The speciﬁc embeddings used are explained next.
Choice of Embeddings
We use two ‘representative’ and ‘discriminative’ histogram style embeddings.
The dimension-level embedding is to embed each feature dimension into a h histogram.
Representative ( e 1 j ( Z Tu ) and e 1 j ( Z Tl ))
For the representative embedding , we encode each feature dimension as a histogram over the instances in that dimension.
For example, we may rescale the i th dimension features into [0 , 1] and divide the dimension into 10 bins.
Then we count the proportion of labelled and unlabelled data for each bin.
This would give a 1 × 20 embedding for each dimension.
Discriminative ( e 2 j ([ Z Tu , Z Tl ] , f ))
In this case we create a 2-D histogram of 10 bins per dimension.
In this histogram we count the frequency of instances with feature values within each bin as per the previous embedding jointly with the frequency of instances with posterior values within each bin.
Finally we get counts in a a 10 × 10 grid, which we vectorize to 1 × 100.
Concatenating these two kind of embeddings we have we have that [ e 1 j ( Z Tu ) , e 1 j ( Z Tl ) , e 2 j ([ Z Tu , Z Tl ] , f )] provides a E =dimensional representation of each feature dimension for processing by the meta network.
Training for Cross Dataset Generalization We train policy networks and meta networks using the policy gradient method REINFORCE (Williams, 1992) to ensure that the generated policies maximise the return (active learning accuracy) with the desired reward discounting.
To ensure that our pair of networks achieve the desired dataset (active learning problem) invariance, we perform multi-task training on multiple source datasets: (i)
In every mini batch we sample a random subset of source datasets, and set the return to the average return over all the sampled datasets.
Thus achieving a high return means the meta network has learned to synthesise suitable per-dataset weights for the policy network based on the dataset embedding, and that together they generalize across multiple tasks/datasets.
(ii) To further promote cross-dataset generalization, we apply the baseline method to standardize the return from each episode which compensates the variant return scale in diﬀerent datasets.
This relative return alleviates the risk of domination by a single dataset with large return due to diﬀering scale of accuracy increments among datasets of varying diﬃculty.
The overall training algorithm is summarised in Alg. 1.
3. Reinforcement Learning Training and Objective Functions
The ideal active learner should query the instance that maximally improves the base learner performance.
The reward that reﬂects the quantity we care about is therefore the increase of test split accuracy r t =
Acc t − Acc t − 1 .
To optimise this quantity non-myopically, we deﬁne the return of an active learning session as J ( θ ) =
E [ (cid:80) ∞ t =1 γ t − 1 r t ( s, π θ ( · , s ))].
We then use policy gradient to train the policy and meta-networks to optimise the objective J ( θ ).
Input: 1: for < each iteration > do (cid:46) 1 . . .
20 ,2: for < each episode > do (cid:46)
Collect batch 3:
Pick source dataset randomly 4:
Initialise label and unlabelled pool 5: for < each time step to time T > do 6:
Sample action π ( a i | s ) ∝ exp Φ θp ( W Te z i ) 7:
Update the Z u , Z l and base learner f 8:
Record the triplet < Z u , a, r > (cid:46) state, action, reward 9: end for 10:
Standardize episode-collected return 11: end for 12:
Update Policy with standardized return 13: end for 14: return Trained Active Query Policy
Auxliary Regularization Losses
Besides optimising the obtained reward, we also optimise for two auxliary regularisation losses.
Reconstruction: The policy network should reconstruct the unlabeled input data using W d prediced by the meta-network (Romero et al., 2016).
Entropy Regularization:
Following Mnih et al.
, we also prefer a policy that maintains a high-entropy posterior over actions so as to continue to explore and avoid pre-emptive convergence to an over-conﬁdent solution.
Integrating the main RL and two auxiliary supervised tasks together, we train both networks end-to-end.
We maximize the whole objective function F by reversing the sign of reconstruction loss:
[Equation]
(2) where θ = { θ p , θ em } .
The network (Fig. 1) trained by Eq.
2 using Alg.
1 learns to synthesise policies that are eﬀective active query criteria (high return J ) on any domain/dataset (synthesising domain speciﬁc network parameters via auxiliary network), adapting to the statistics of the dataset and independent of the dimensionality of the dataset.
Experiments 4. Datasets and Settings Datasets
Overall we experiment with a diverse set of 14 datasets from UCI machine learning repository.
These include austra , heart , german , ILPD , ionospheres , pima , wdbc , breast , diabetes , fertility , fourclass , habermann , livers , planning .
We split these into 7 for multi-task training, and 7 for testing.
Architecture
The auxiliary network for encoder has fully connected layers with of size 120 , 100 , 100 ( E = 120 , k = 100) and decoder auxiliary network has analogous structure.
The policy network has layers of size N × d ( N × d input matrix Z u ), N × 100 N × 50, N × 10, N × 1 ( N -way output).
All penultimate layers use ReLU activation.
Transition of the input to ﬁrst hidden layer of policy network is provided by the auxiliary network.
Thereafter for eﬃcient implementation with few parameters and to deal with the variable sized input and output, it is implemented convolutionally.
We convolve a h 1 × h 2 sized matrix across the N dimension of each N × h 1 matrix shaped layer to obtain the next N × h 2 layer.
Experiment Settings
We train using Adam optimiser with initial learning rate 0.
and hyperparameters set to α = λ = 0 .
005 and discount factor γ = 0 .
During RL training, we use two tricks to stabilize the policy gradient.
We use a relatively large batch size of 32 episodes.
We smooth the gradient by accumulated time-step G t =
(1 − α )
G t − 1 + αg t where g t is the gradient of the a t in time step t and the G t is the accumulated gradient.
Intuitively, the accumulated gradient G t puts more emphasis on early time step actions.
We active learning
Expert Features: To enhance the low-level feature of each instance in X we deﬁne expert features ξ ( X ) to include distance furthest ﬁrst and uncertainty as the augmented feature.
In most experiments we use linear SVM as the base learner.
Alternatives
We compare our learning approach to AL with two classic approaches uncertainty/margin-based sampling (US) (Tong & Koller, 2002; Kapoor et al., 2007) and furthest-ﬁrst-based sampling (DFF) (Baram et al., 2004), as well as to random sampling (RAND) as a lower bound.
Uncertainty sampling is a simple deterministic approach that queries the instance with minimum certainty (maximum entropy).
While simple, and not the most state of the art criteria, it is consistently very competitive with more sophisticated criteria and more robust in the sense of hardly ever being a very poor criteria.
As a representative more sophisticated approach, we compare with QUIRE (Huang et al., 2010).
We denote our method meta-learned policy for general active learning (MLP-GAL). As a related alternative we propose SingleRL.
This is our RL approach, but without the metanetwork, so a single model is learned over all datasets.
Without the meta-network it can only use expert features ξ ( X ) so that dimensionality is ﬁxed over datasets.
To give SingleRL an advantage we concatenate some extra global features to the input space 1 .
This method can also be seen as a version of one of the few state of the art learning-based alternatives (Konyushkova et al., 2017).
But upgraded in that we learn it with reinforcement learning instead of the more myopic supervised learning used in (Konyushkova et al., 2017).
4. Multi-Task Training Evaluation
In the ﬁrst experiment we verify that it is indeed possible to learn a single policy that generalizes across multiple training datasets.
Here we train on a random split of all seven of the training datasets, and test on a random split of the same seven datasets.
The results in Table 1 shows that our approach learns an eﬀective criterion that outperforms the given baselines.
Although there are diﬀerent random splits there is possibly some overﬁtting here as the policy has seen all these datasets at some during training (datasets randomly selected within minibatches).
However it is reassuring that it works because it means that it is possible to learn a single query policy that performs well on a diverse set of datasets.
4. Cross-Task Generalization
In the next experiment we apply our multi-task trained method to the held out datasets.
According to the results in Table 2, we see that our MLP-GAL approach is comparable or better than alternatives, even when applied on a diverse suite of datasets diﬀerent to those it was trained on.
Fig.
2 shows examples of the resulting learning curves we can see that our MLP-GAL can indeed learn query policies that improve the base learner visibly more rapidly than alternatives, even on novel datasets.
SingleRL is generally quite eﬀective compared to prior methods, showing the eﬃcacy of training a policy with RL.
However it does not beneﬁt from a meta network, so is not as eﬀective as our MLP-GAL .
From the table it is also interesting to see that while sophisticated methods such as QUIRE sometimes perform very well, they also sometimes perform very badly scoring even worse than random (red).
Meanwhile although the simple and classic uncertainty-sampling is never best, it is also never worst: performing more consistently than more elaborate QUIRE.
This illustrates that the challenge in building sophisticated AL algorithms that generalize to datasets that they were not engineered on.
In contrast, although our approach has not seen these datasets it performs consistently well due to adapting to each dataset via the meta-network.
An advantage of our approach compared to related methods such as Bachman et al.
Woodward & Finn (2017) is that it treats the base learner as part of the environment to be optimised against rather than tying the user to a particular method.
To demonstrate this, we repeat the previous experiment using RBF-SVM as the base learner rather than linear
SVM.
After multi-task training on source datasets, the results for held out testing datasets [Table] [Figure] are shown in Table 3.
We can see that the results of hard-crafted policies are similar to linear SVM (expected given the diﬃculty of learning a non-linear model in a budget ofpoints).
However our learning-based approach is again consistently high performing and eﬀective overall – it is able to learn a policy customised for this new type of base learner.
Related Work Active Learning by Learning
A few papers have very recently appeared that also approach ﬁnding an AL criterion as a learning problem.
Konyushkova et al.
(2017) proposes to learn a criterion based on a vector of expert features (e.g., classiﬁer conﬁdence, label imbalance).
However by using expert features, this misses the chance to learn the representation from raw features as in our approach; and by using supervised rather than reinforcement learning to train the policy, it is not optimally non-myopic.
Bachman et al.
(2017) and Woodward & Finn (2017) use RL to train a single model that provides both the base classiﬁer and the active learner.
This tight integration has the drawback that the frameworks are constrained to a speciﬁc base learner, so cannot be used to improve the training of an arbitrary base learner as per our framework.
More importantly, while these methods learn eﬀective non-myopic policies, they are trained and tested on diﬀerent classes within the same dataset, so the generalization challenge and evaluation is minimal.
There is no mechanism to ensure eﬀective transfer across datasets of diﬀerent statics or to allow any transfer at all across datasets of diﬀerent dimensionality.
Active Learning Ensembles
Diﬀerent AL algorithms perform well on diﬀerent datasets, or at diﬀerent learning stages.
For this reason studies have proposed heuristics to switch criteria from early to late stage learning (Donmez et al., 2007; Baram et al., 2004), or use multi-armed bandit (MAB) approaches to estimate the best criterion for a given dataset within an ensemble (Hsu & Lin, 2015).
But aside from being myopic, MAB learners do not learn transferrable knowledge: They perform all their learning within a single rollout, and their need to explore/learn online is fundamentally at odds with active learning. Chu
[Table] [Table]
& Lin (2016) ameliorate this somewhat with regularization, but still need dataset-speciﬁc learning.
Our approach can address these issues: Besides non-myopic policy learning with RL, a DNN has capacity to encode multiple criteria and apply diﬀerent ones at diﬀerent stages of learning. By learning a meta-policy that paramaterizes a dataset-speciﬁc policy, it customizes the overall active learning strategy to the target dataset; thus transferring knowledge for immediate eﬃcacy on a new dataset without dataset speciﬁc learning.
Domain Generalisation and Adaptation
Our task-agnostic AL goal is related to
Domain Generalisaton (DG) (Muandet et al., 2013) and Domain Adpatation (DA) (Ganin & Lempitsky, 2015) in supervised learning in that we would like to train on one dataset and perform well when testing on another dataset.
Our framework has aspects of DG (multi-task training to increase generality) and DA (adapting to target data, via dataset embedding meta network) methods.
But we are not aware of any dataset embedding approaches to achieving DA within supervised learning.
Related Methods
Models that predict the parameters of other models are increasingly widely used (Ha et al., 2017).
In a reinforcement learning context, this is known as contextual (Kupcsik et al., 2013) or paramaterised (Stulp et al., 2013) policies.
Romero et al. (2016) used auxiliary networks for parameter reduction when training and testing within one dataset.
Discussion
We have proposed a learning-based perspective on the problem of active query criteria design.
Such learning-based algorithm design elegantly obtains AL models by optimising the ultimate goal of classiﬁcation performance with few labels.
However aside from the widely-shared questions of good network architecture and RL training algorithms, the key challenge is then whether general enough policies can be learned to be widely useful in diﬀerent applications, rather than requiring dataset-speciﬁc training which contradicts the motivation of AL.
Our key contribution is to provide the ﬁrst solution to this issues through multi-task training of a meta-network that synthesises dataset-speciﬁc active query policies.
Our study thus far has the main limitation that we have only evaluated our method on a binary base classiﬁer (binary assumption shared by Konyushkova et al.
In future work we would like to evaluate our method on deep multi-class classiﬁers by designing embeddings which can represent the state of such learners, as well as explore application to the stream-based AL setting.