Hierarchical Expert Networks for Meta-Learning
Abstract
The goal of meta-learning is to train a model on a variety of learning tasks, such that it can adapt to new problems within only a few iterations.
Here we propose a principled informationtheoretic model that optimally partitions the underlying problem space such that specialized expert decision-makers solve the resulting subproblems.
To drive this specialization we impose the same kind of information processing constraints both on the partitioning and the expert decision-makers.
We argue that this specialization leads to efﬁcient adaptation to new tasks.
To demonstrate the generality of our approach we evaluate three meta-learning domains: image classiﬁcation, regression, and reinforcement learning.
1. Introduction
Recent machine learning research has shown impressive results on incredibly diverse tasks from problem classes such as pattern recognition, reinforcement learning and generative model learning (Devlin et al., 2018; Mnih et al., 2015;
Schmidhuber, 2015).
These success stories typically have two computational luxuries in common: a large database with thousands or even millions of training samples and a long and extensive training period.
Applying these pretrained models to new tasks na¨ıvely usually leads to poor performance, as with each new incoming batch of data, expensive and slow re-learning.
In contrast to this, humans can learn from few examples and excel at adapting quickly (Jankowski et al., 2011), for example in motor tasks (Braun et al., 2009) or at learning new visual concepts (Lake et al., 2015).
Sample-efﬁcient adaptation to new tasks can is a form of meta-learning or “learning to learn” (Thrun & Pratt, 2012;
Schmidhuber et al., 1997; Caruana, 1997) and is an ongoing and active ﬁeld of research–see e.g., (Koch et al., 2015; Correspondence to: Heinke Hihn
Vinyals et al., 2016; Finn et al., 2017; Ravi & Larochelle, 2017; Ortega et al., 2019; Botvinick et al., 2019; Yao et al., 2019).
Meta-learning has multiple deﬁnitions, but a common point is that the system learns on two levels, each with different time scales: slow meta-learning across different tasks, and fast learning to adapt to each task individually.
Here, we propose a novel information-theoretic learning paradigm for hierarchical meta-learning systems.
The method we propose comes from a single uniﬁed framework and can be readily applied to supervised meta-learning and to meta-reinforcement learning.
Our method ﬁnds an optimal soft partitioning of the problem space by imposing information-theoretic constraints on both the process of expert selection and on the expert specialization.
We argue that these constraints drive an efﬁcient division of labor in systems that have limited information processing power, where we make use of information-theoretic bounded rationality (Ortega & Braun, 2013).
When the model is presented with previously unseen tasks it assigns them to experts specialized on similar tasks – see Figure 1.
Additionally, expert networks specializing in only a subset of the problem space allow for smaller neural network architectures with only a few units per layer.
To split the problem space and to assign the partitions to experts, we learn to represent tasks through a common latent embedding, that is then used by a selector network to distribute the tasks to the experts.
The outline of this paper is as follows:
ﬁrst, we introduce bounded rationality and meta-learning, next we introduce our novel approach and derive applications to classiﬁcation, regression, and reinforcement learning.
We evaluate our method against current meta-learning algorithms and perform additional ablation studies.
Finally, we conclude.
2. Information-Processing Constraints in Hierarchical Learning Systems
An important concept in decision making is the notion of utility (Von Neumann & Morgenstern, 2007), where an agent picks an action a ∗ x ∈
A such that it maximizes their utility in some context s ∈ S , i.e., a ∗ x = arg max a U ( x, a ) , where the utility is a function U ( x, a ) and the states distribution p ( s ) is known and ﬁxed.
Trying to solve this optimization problem na¨ıvely leads to an exhaustive search over all possible ( a, x ) pairs, which is in general a pro- [Figure]
hibitive strategy. Instead of ﬁnding an optimal strategy, a bounded-rational decision-maker optimally trades off expected utility and the processing costs required to adapt.
In this study we consider the information-theoretic freeenergy principle (Ortega & Braun, 2013) of bounded rationality, where an upper bound on the Kullback-Leibler divergence D KL ( p ( a | x ) || p
( a ))
= (cid:80) a p ( a | x ) log p ( a | x ) p ( a ) between the agent’s prior distribution p ( a ) and the posterior policy p ( a | x ) model the decision-maker’s resources, resulting in the following constrained optimization problem:
[Equation]
This constraint deﬁnes a regularization on p ( a | x ) .
We can transform this into an unconstrained variational problem by introducing a Lagrange multiplier β ∈ R + :
[Equation]
For β → ∞ we recover the maximum utility solution and for β → 0 the agent can only act according to the prior.
The optimal prior in this case is the marginal p ( a )
= (cid:80) x ∈ X p ( x ) p ( a | x ) (Ortega & Braun, 2013).
Aggregating bounded-rational agents by a selection policy allows for solving optimization problems that exceed the capabilities of the individual decision-makers (Genewein et al., 2015).
To achieve this, the search space is split into partitions such that each partition can be solved by a decisionmaker.
A two-stage mechanism is introduced: The ﬁrst stage is an expert selection policy p ( m | x ) that chooses an expert m given a state x and the second stage chooses an action according to the expert’s posterior policy p ( a | x, m ) .
The optimization problem given by Equation (3) can be extended to incorporate a trade-off between computational costs and utility in both stages:
[Equation]
where β 1 is the resource parameter for the expert selection stage and β 2 for the experts.
I ( · ; · ) is the mutual information between the two random variables.
To measure a decisionmaker’s performance on terms of the utility vs. cost trade off we use the free-energy difference deﬁned as
[Equation]
where p is the posterior and q the decision-maker’s prior policy.
The marginal distribution p ( a | x ) deﬁnes a mixtureof-experts (Jordan & Jacobs, 1994; Jacobs et al., 1991) policy given by the posterior distributions p ( a | s, m ) weighted by the responsibilities determined by the Bayesian posterior p ( x | m ) .
Note that p ( x | m ) is not determined by a given likelihood model, but is the result of the optimization process (4).
3. Meta Learning
We can divide Meta-learning algorithms roughly into MetricLearning (Koch et al., 2015; Vinyals et al., 2016;
Snell et al., 2017), Optimizer Learning (Ravi & Larochelle, 2017;
Finn et al., 2017; Zintgraf et al., 2018; Rothfuss et al., 2018), and
Task Decomposition Models (Lan et al., 2019; Vezhnevets et al., 2019).
Our approach depicted in Figure 1 can be seen as a member of the latter group.
3.1. Meta Supervised Learning
In a supervised learning task we are usually interested in a dataset consisting of multiple input and output pairs D
= { ( x i , y i ) } Ni [-=
-]=1 and the learner’s task is to ﬁnd a function f ( x ) that maps from input to output, for example through a deep neural network.
To do this, we split the dataset into training and test sets and ﬁt a set of parameters θ on the training data and evaluate on test data using the learned function f θ ( x ) .
In meta-learning, we are instead working with meta-datasets D , each containing regular datasets split into training and test sets.
We thus have different sets for meta-training, metavalidation, and meta-test, i.e., D = { D train , D val , D test } .
The goal is to train a learning procedure (the meta-learner) that can take as input one of its training sets D train and produce a classiﬁer (the learner) that achieves low prediction error on its corresponding test set D test . The meta-learning is then updated using performance measure based the learner’s performance on D val .
This may not always be the case, but our work (among others, e.g., (Finn et al., 2017)) follow this paradigm.
The rationale being that the meta-learner is trained such that it implicitly optimizes the base learner’s generalization capabilities.
3.2. Meta Reinforcement Learning
First, we give a short introduction to reinforcement learning in general, and then we show how this deﬁnition can be extended to meta reinforcement learning problems.
We model sequential decision problems by deﬁning a Markov
Decision Process as a tuple ( S , A , P, r ) , where S is the set of states, A the set of actions, P : S ×
A × S → [0 , 1] is the transition probability, and r : S ×
A → R is a reward function.
The aim is to ﬁnd the parameter θ of a policy π θ that maximizes the expected reward:
[Equation]
We deﬁne r ( τ ) =
(cid:80) ∞ t =0 r ( s t , a t ) as the cumulative reward of trajectory τ
= { ( s t , a t ) } ∞ i =0 , which is sampled by acting according to the policy π , i.e., ( s, a ) ∼ π ( ·| s ) , and s t +1 ∼ P ( ·| s t , a t ) .
We model learning in this environment as reinforcement learning (Sutton & Barto, 2018), where an agent interacts with an environment over some (discrete) time steps t .
At each time step t , the agent ﬁnds itself in a state s t and selects an action a t according to the policy π ( a t
| s t ) . In return, the environment transitions to the next state s t +1 and generates a scalar reward r t . This process continues until the agent reaches a terminal state after which the process restarts. The goal of the agent is to maximize the expected return from each state s t , which is typically deﬁned as the inﬁnite horizon discounted sum of the rewards.
In meta reinforcement learning the problem is given by a set of tasks t i ∈ T , where MDP t i = ( S , A , P i , r i ) deﬁne tasks t
i as described earlier. We are now interested in ﬁnding a set of policies Θ that maximizes the average cumulative reward across all tasks in T and generalizes well to new tasks sampled from a different set of tasks T (cid:48) . 1: Input : Data Distribution p ( D ) , number of samples K , batch-size M , training episodes N 2: Hyper-parameters : resource parameters β 1 , β 2 , learning rates η x , η x for selector and experts 3: Initialize parameters θ, ϑ 4: for i = 0, 1, 2, ..., N do 5:
Sample batch of M datasets D i ∼ p ( D ) , each consisting of a training dataset D meta-train and a metavalidation dataset D meta-val with 2 K samples each 6: for D ∈ D i do 7:
Find Latent
Embedding z
( D meta-train ) 8:
Select expert m ∼ p θ ( m | z ( D meta-train ) 9:
Compute ˆ f ( m, D val ) 10: end for 11:
Update selection parameters θ with ˆ f ( m, D val ) 12:
Update Autoencoder with positive samples in D i 13:
Update experts m with assigned D meta-train 14: end for 15: return θ , ϑ
[-(cid:80) ∞ t =
-]
4. Expert Networks for Meta-Learning
Information-theoretic bounded rationality postulates that hierarchies and abstractions emerge when agents have only limited access to computational resources (Genewein et al., 2015), e.g., limited sampling complexity (Hihn et al., 2018) or limited representational power (Hihn et al., 2019).
We will show that forming such abstractions equips an agent with the ability of learning the underlying problem structure and thus enables learning of unseen but similar concepts.
The method we propose comes out of a uniﬁed optimization principle and has the following important features:
A regularization mechanism to enforce the emergence of expert policies.
A task compression mechanism to extract relevant task information.
A selection mechanism to ﬁnd the most efﬁcient expert for a given task.
A regularization mechanism to improve generalization capabilities.
4.1. Latent Task Embeddings
Note that the selector assigns a complete dataset to an expert and that this can be seen as a meta-learning task, as described in (Ravi & Larochelle, 2017).
To do so, we must ﬁnd a feature vector z ( d ) of the dataset d .
This feature vector must fulﬁll the following desiderata: 1) invariance against permutation of data points in d , 2) high representational capacity, 3) efﬁcient computability, and 4) constant dimensionality regardless of sample size K .
In the following we propose such features for image classiﬁcation, regression, and reinforcement learning problems.
For image classiﬁcation we propose to pass the positive images in the dataset through a convolutional autoencoder and use the outputs of the bottleneck layer.
Convolutional Autoencoders are generative models that learn to reconstruct their inputs by minimizing the Mean-Squared-Error between the input and the reconstructed image (see e.g., (Chen et al., 2019)).
In this way we get similar embeddings z ( d ) for similar inputs belonging to the same class.
The latent representation is computed for each positive sample in d and then passed through a pooling function h ( z ( d )) to ﬁnd a single embedding for the complete dataset–see ﬁgurefor an overview of our proposed model.
While in principle functions such as mean, max, and min can be used, we found that max-pooling yields the best results.
The authors of (Yao et al., 2019) propose a similar feature set.
For regression we deﬁne a similar feature vector.
We transform the K training data points into a feature vector z ( d ) by binning the points into N bins according to their x value and collecting the y value.
If more than one point falls into the same bin, we average the y values, thus providing invariance against the order of the data points in D train .
We use this feature vector to assign each data set to an expert according to p θ ( x | z ( d )) .
In the reinforcement learning setting we use a dynamic recurrent neural network (RNN) with LSTM units (Hochreiter & Schmidhuber, 1997) to classify trajectories.
We feed the
RNN with ( s t , a t , r t , t ) tuples to describe the underlying
Markov Decision Process describing the task.
At t =
we sample the expert x according to the learned prior distribution p ( x ) , as there is no information available so far.
The authors of (Lan et al., 2019) propose a similar feature set.
4.2. Specialization in Supervised Learning
Combining multiple experts can often be beneﬁcial (Kuncheva, 2004), e.g. in Mixture-of-Experts (Yuksel et al., 2012) or Multiple Classiﬁer Systems (Bellmann et al., 2018).
Our method can be interpreted as a member of this family of algorithms.
We deﬁne the utility as the negative prediction loss, i.e.
U ( f x ( d ) , y ) =
−L ( f x ( d ) , y ) , where f x ( d ) is the prediction of the expert x given the input data point d (in the following we will use the shorthand ˆ y x ) and y is the ground truth.
We deﬁne the cross-entropy loss L (ˆ y x , y, )
= − (cid:80)
i
y i log ˆ y ix as a performance measure for classiﬁcation and the mean squared error L (ˆ y x , y )
=
(cid:80) i
(ˆ y i − y i ) 2 for regression.
The
1: Input : Environment Distributions p ( T ) and p ( T (cid:48) ) , number of roll-outs K , batch-size M , training episodes
N , number of tuples L used for expert selection 2: Hyper-parameters : resource parameters β 1 , β 2 , learning rates η x , η x for selector and experts 3: Initialize parameters θ, ϑ 4: for i = 1, 2, 3, ..., N do 5:
Sample batch of M environments E train i ∼ p
( T )
and
E val i ∼ p
( T (cid:48) ) 6: for E ∈ E train i do 7: for k = 1, 2, 3, ..., K do 8:
Collect τ = { ( x t , a t , r t , t ) } Lt =1 tuples by following random expert 9:
Select expert m ∼ p θ ( m | τ ) with RNN policy 10:
Collect trajectory τ k = { ( x t , a t , r t , t ) }
Tt =
L by following p
ϑ ( a | x, m ) 11: end for 12:
{+Compute F t = (cid:80) Tl =
+}
γ l f ( x t + l , m t + l , a t + l ) for trajectories τ 13: where f ( x, m, a ) =
r train ( x, a ) − 1 β 2 log p
ϑ ( a | x,m ) p ( a | m
[Equation]
14: end for 15:
{+Compute ¯ F t = (cid:80) Tl =
+}
γ l ¯ f ( x t + l , m t + l ) with 16: ¯ f ( x, m ) =
E p
ϑ ( a | x,m ) (cid:104) r val ( x, a ) − 1 β 2 log p ( a | x,m ) p ( a | m ) (cid:105) 17:
Update selection parameters θ with ¯ F collected in batch i 18:
Update experts m with roll-outs collected in batch i 19: end for 20: return θ , ϑ objective for expert selection thus is given by
[Equation]
where ˆ f =
[Equation]
i.e. the free energy of the expert and θ, ϑ are the parameters of the selection policy and the expert policies, respectively.
Analogously, the action selection objective for each expert x is deﬁned by
[Equation]
This regularization technique can be seen as a form of output regularization (Pereyra et al., 2017).
4.3. Specialization in Reinforcement Learning
In the following we will derive our algorithm for specialization in hierarchical reinforcement learning agents.
Note that in the reinforcement learning setup the reward function r ( x, a ) deﬁnes the utility U ( x, a ) .
In maximum entropy RL (see e.g., Haarnoja et al.
(Haarnoja et al., 2017))
the regularization penalizes deviation from a ﬁxed uniformly distributed prior, but in a more general setting we can discourage deviation from an arbitrary prior policy by optimizing for:
[Equation]
where β trades off between reward and entropy, such that β → ∞ recovers the standard RL value function and β →recovers the value function under a random policy.
To optimize the objective (9) we deﬁne two separate kinds of value function, V φ for the selector and one value function
V ϕ for each expert.
Thus, each expert is an actor-critic with separate actor and critic networks.
Similarly, the selector has an actor-critic architecture, where the actor network selects experts and the critic learns to predict the expected free energy of the experts depending on a state variable.
The selector’s policy is represented by p θ , while each expert’s policy is represented by a distribution p ϑ .
In standard reinforcement learning a common technique to update a parametric policy representation p ω ( a | x ) with parameters ω , is to use policy gradients that optimize the cumulative reward J ( ω ) =
E [ p ω ( a | x ) V ψ ( x )] expected under the critic’s prediction V ψ ( x ) , by following the gradient ∇ ω J ( ω ) =
E [ ∇ ω log p ω ( a | x ) V ψ ( x )] .
This policy gradient formulation (Sutton et al., 2000) is prone to producing high variance gradients.
A common technique to reduce the variance is to formulate the updates using the advantage function instead of the reward (Arulkumaran et al., 2017).
The advantage function A ( a, x ) is a measure of how well a certain action a performs in a state x compared to the average performance in that state, i.e., A ( a, x ) =
Q ( x, a ) − V ψ ( x ) .
Here, V ( x ) is the value function and is a measure of how well the agent performs in state x , and Q ( x, a ) is an estimate of the cumulative reward achieved in state x when the agent executes action a .
Instead of learning the value and the Q function, we can approximate the advantage function solely based on the critic’s estimate V ψ ( x ) by noting that
Q ( x t , a t )
≈ r
( x t , a t )+ γV ψ ( x t +1 ) .
Similar to the standard policy update based on the advantage function, the expert selection stage can be formulated by optimizing the expected advantage E p
ϑ ( a | x,m ) [ A m ( x, a )] for expert m with
[Equation]
Accordingly, we can deﬁne an expected advantage function E (cid:2) p
[Equation]
( m
[Equation]
( x
[Equation]
m
[Equation]
(cid:3) for the selector with
[Equation]
where m t denotes the expert m selected at time t . We estimate the double expectation by Monte Carlo sampling,
[Table]
convolution block. This corresponds to our expert network architecture. Using the suggested architectures by the respective studies, we achieve classiﬁcation accuracy ≥ 95% where in practice we use a single ( x, m, a ) tuple for ˆ f ( x, m ) , which enables us to employ our algorithm in an on-line optimization fashion.
In the RL setup the reward function r ( x, a ) deﬁnes the utility U ( x, a ) .
To optimize the objective we deﬁne value functions, V φ for the selector and one value function V ϕ for each expert.
Thus, the selector and each expert is an actor-critic architecture.
We deﬁne a discounted free energy as
[Equation]
which we learn through a value function V ϕ for each expert.
We deﬁne the selector’s discounted free energy
[Equation]
with ¯ f ( x, m ) =
E p
ϑ ( a | x,m )
[ r ( x, a ) − 1 β 2 log p ϑ
[Equation] a | x,m ) p ( a | m ) ] that is learned through the selector’s value function V φ .
Now let p θ ( m | x ) be the selection policy and
p ϑ ( a | x, m ) the expert policy.
The expert selection stage is optimizing the expected advantage E [ p ϑ ( a | x, m )
A m ( x, a )] for expert m with
[Equation]
Accordingly, we deﬁne an expected advantage function
[Equation]
selector
[Equation]
We ﬁnd the policy parameters by gradient descent on the objectives.
5. Results 5.1. Sinusoid Regression
We adopt this task from (Finn et al., 2017).
In this K -shot problem, each task consists of learning to predict a function of the form y = a · sin( x + b ) , with both a ∈ [0 . 1 , 5] and b ∈ [0 , 2 π ] chosen uniformly, and the goal of the learner is to ﬁnd y given x based on only K pairs of ( x, y ) .
Given that the underlying function changes in each iteration it is impossible to solve this problem with a single learner.
Our results show that by combing expert networks, we are able to reduce the generalization error iteratively as we add more experts to our system–see Figure 2 where we also show how the system is able to capture the underlying problem structure as we add more experts.
In Figure 4 we visualize how the selector’s partition of the problem space looks like.
The shape of the emerged clusters indicates that the selection is mainly based on the amplitude a of the current sine function, indicating that from an adaptation point-of-view it is more efﬁcient to group sine functions based on amplitude a instead of phase b .
We can also see that an expert specializes on the low values for b as it covers the upper region of the a × b space.
The selection network splits this region among multiple experts if we increase the set of experts to 8 or more.
5.2. Few-Shot Classiﬁcation
The Omniglot dataset (Lake et al., 2011) consists of over 1600 characters from 50 alphabets.
As each character has only 20 samples each drawn by a different person, this forms a challenging meta-learning benchmark. The selection policy now selects experts based on their freeenergy that is computed over datasets D val and the selection policy depends on the training datasets D train : [Equation] where
[Equation]
the free energy of expert m on dataset D val , L (ˆ y, y ) is loss function, and ( x, y ) ∈ D val .
The experts optimize their free energy objective on the training dataset D train deﬁned by [Equation] where ( x, y ) ∈ D train . We train the learner on a subset of the dataset ( ≈ 80% , i.e., ≈ 1300 classes) and evaluate the remaining ≈classes, thus investigating the generalization.
In each training episode we build the datasets D train and D val by selecting a target class c t and sampling K positive and K negative samples.
To generate negative samples, we draw K images randomly out of the remaining N − 1 classes.
We present the selection network with the feature presentation of the
K positive training samples (see Figure 1), but evaluate the experts’ performance on the 2 K test samples in D val .
In this way, the free energy of the experts becomes a generalization measure.
Using this optimization scheme, we train the networks to become experts in recognizing a subset of classes.
After a proper expert is selected we train that expert using the 2 K samples from the training dataset.
We consider three experimental setups: 1) how does a learner with only a single hidden layer perform when trained n¨ıvely compared to with sophisticated methods such as MAML (Finn et al., 2017) and Matching Nets (Vinyals et al., 2016) as a baseline? 2) does the system beneﬁt from adding more experts and if so, at what rate? and 3) how does our method compare to the aforementioned algorithms? Regarding 1) we note that introducing constraints by reducing
5 a    b K =2 E x p e r t s     5 a    b K =     5 a    b K =     5 a    b 4 E x p e r t s     5 a    b     5 a    b     5 a    b 8 E x p e r t s     5 a    b     5 a    b the representational power of the models does not facilitate specialization is it would by explicit information-processing constraints.
In the bottom row of Figure 2 we address question 2).
We can interpret this curve as the rate-utility curve showing the trade-off between information processing and expected utility (transparent area represents one standard deviation), where increasing I ( X ; M ) improves adaptation.
The improvement gain grows logarithmically, which is consistent with what rate-distortion theory would suggest.
In
Table 1 we present empirical results addressing question 3).
5.3. Meta Reinforcement Learning
We create a set of RL tasks by sampling the parameters for the Inverted Double Pendulum problem (Sutton, 1996) implemented in OpenAI Gym (Brockman et al., 2016), where the task is to balance a two-link pendulum in an upward position.
We modify inertia, motor torques, reward function, goal position and invert the control signal – see Table 7 for details.
We build the meta task set T (cid:48) on the same environment, but change the parameter distribution and range, providing new but similar reinforcement learning problems.
During training, we sample M environments in each episode and perform updates as described earlier.
During evaluation, we measure the system’s performance on tasks sampled from T (cid:48) – see results in Figure 3, where we can see improving performance as more experts are added and the increasing mutual information in the selection stage indicates precise partitioning.
The expert policies are trained on the meta-training environment policies, but evaluated on unseen but similar validation environments.
In this setting we deﬁne the discounted free energy ¯ F t with [Equation]
where r val is a reward function deﬁned by a validation environment (see Figure 3 for details).
6. Related Work
The hierarchical structure we employ is related to Mixture of Experts (MoE) models. (Jacobs et al., 1991; Jordan & Jacobs, 1994) introduced MoE as tree structured models for complex classiﬁcation and regression problems, where the underlying approach is a divide and conquer paradigm. As in our approach, three main building blocks deﬁne MoEs: gates, experts, and a probabilistic weighting to combine expert predictions.
Learning proceeds by ﬁnding a soft partitioning of the input space and assigning partitions to experts performing well on the partition. In this setting, the model response is then a sum of the experts’ outputs, weighted by how conﬁdent the gate is in the expert’s opinion (see (Yuksel et al., 2012) for an overview). This paradigm has seen recent interest in the ﬁeld of machine learning, see e.g., (Aljundi et al., 2017; Rosenbaum et al., 2019; Jerfel et al., 2019; Shazeer et al., 2017).
The approach we propose allows learning such models, but also has applications to more general decision-making settings such as reinforcement learning. Our approach belongs to a wider class of models that use information constraints for regularization to deal more efﬁciently with learning and decision-making problems (Martius et al., 2013; Leibfried et al., 2017; Grau-Moya et al., 2017; Achiam et al., 2017; Hihn et al., 2018; Grau-Moya et al., 2019). One such prominent approach is Trust Region Policy Optimization (TRPO) (Schulman et al., 2015).
The main idea is to constrain each update step to a trust region around the current state of the system.
This region is deﬁned by D KL ( π new || π old ) between the old policy and the new policy, providing a theoretic monotonic policy improvement guarantee. In our approach we deﬁne this region by D KL between the agent’s posterior and prior policy, thus allowing to learn this region and to adapt it over time. This basic idea has been extend to meta-learning by (Rothfuss et al., 2018), which we use to compare our method against in meta-rl experiments.
Most other methods for meta-learning such as the work of (Finn et al., 2017) and (Ravi & Larochelle, 2017) ﬁnd an initial parametrization of a single learner, such that the agent can adapt quickly to new problems.
This initialization represents prior knowledge and can be regarded as an abstraction over related tasks and our method takes this idea one step further by ﬁnding a possibly disjunct set of such compressed task properties.
Another way of thinking of such abstractions by lossy compression is to go from a task-speciﬁc posterior to a task-agnostic prior strategy.
By having a set of priors the task speciﬁc information is available more locally then with a single prior, as in MAML (Finn et al., 2017) and the work of (Ravi & Larochelle, 2017).
In principle, this can help to adapt within fewer iterations.
Thus our method can be seen as the general case of such monolithic metalearning algorithms.
Instead of learning similarities within a problem, we can also try to learn similarities between different problems (e.g., different classiﬁcation datasets), as is described in the work of (Yao et al., 2019).
In this way, the partitioning is governed by different tasks, where our study however focuses on discovering meta-information within the same task family, where the meta-partitioning is determined solely by the optimization process and can thus potentially discover unknown dynamics and relations within a task family.
7. Discussion
We have introduced and evaluated a novel hierarchical information-theoretic approach to meta-learning.
In particular, we leveraged an information-theoretic approach to bounded rationality.
Although our method is widely applicable, it suffers from low sample efﬁciency in the RL domain.
A research direction would be to combine our system with model-based RL which is known to improve sample efﬁciency.
Another research direction would be to investigate the performance of our system in continual adaption tasks, such as in (Yao et al., 2019).
Another limitation is the restriction to binary meta classiﬁcation tasks, which we leave for feature work.
The results show that our method can identify sub-regions of the problem set and solve them efﬁciently with expert networks. In effect, this equips the system with initializations covering the problem space and thus enables it to adapt quickly to new but similar tasks. To reliably identify such tasks, we have proposed feature extraction methods for classiﬁcation, regression, and reinforcement learning, that could be simply be replaced and improved in future work. The strength of our model is that it follows from simple principles that can be applied to a large range of problems.
Moreover, the system performance can be interpreted in terms of the information processing of the selection stage and the expert decision-makers.
We have shown empirically that our method achieves results comparable to recent metalearning algorithms, such as MAML (Finn et al., 2017),
Matching Nets (Vinyals et al., 2016), and Proximal MetaPolicy Search (Rothfuss et al., 2018).
This work was supported by the European Research Council Starting Grant BRISC , ERC-STG-2015, Project ID Appendix
Few-Shot Classiﬁcation Experimental details: We followed the design of (Vinyals et al., 2016) but reduce the number of blocks to one.
We used a single convolutional block consisting of 32 3 × 3 ﬁlters with strided convolutions followed by a batch normalization layer and a ReLu nonlinearity.
During training we used a meta-batch size of 16.
The convolutional autoencoder is a 3 layer network consisting of 16, 16, and 4 ﬁlters each with size 3 × 3 with strided convolutions followed by a leaky ReLu non-linearity.
The layers are mirrored by de-convolotional layers to reconstruct the image.
This results in an image embedding with dimensionality 64.
The selection network is a two layer network with 32 units each, followed by a ReLu non-linearity, a dropout layer (Srivastava et al., 2014) per layer.
We augment the dataset by rotating each image in 90, 180, anddegrees resulting in 80 images per class.
We also normalize the images two be in (0,1) range.
We evaluate our method by resetting the system to the state after training and allow for 10 gradient updates and report the ﬁnal accuracy.
To evaluate MAML on 2-way N -shot omniglot dataset we used a inner learning rate of α = 0 . 05 and one inner update step per iteration for all settings.
We used a single convolutional block followed by a fully connected layer with 64 units and a ReLU non-linearity.
Note, that we reduce the number of layers to make the tests comparable.
Using the suggested architectures by (Finn et al., 2017) we achieve classiﬁcation accuracy ≥ 95% .
To generate this ﬁgure, we ran 10-fold cross-validation on the whole dataset and show the averaged performance metric and the standard-deviation across the folds.
In both settings, ”0 bits” corresponds to a single expert, i.e., a single neural network trained on the task.
Meta-RL Experimental details: The selector’s actor and critic net are build of RNNs with 200 hidden units each.
The critic is trained to minimize the Huber loss between the prediction and the cumulative reward.
The experts are two layer networks with 64 units each followed by ReLu non-linearities and used to learn the parameters of a Gaussian distribution.
The critics have the same architecture (except for the output dimensionality).
The control signal a is continuous in the interval [-1,1] and is generated by neural network that outputs µ and log( σ ) of a gaussian.
The action is then sampled by re-parameterizing the distribution to p ( a ) = µ + exp( σ ) (cid:15) , where (cid:15) ∼ N (0 , 1) , so that the distribution is differentiable w.r.t to the network outputs.
We average the results over 10 random seeds and trained for 1000 episodes each with a batch of 64 environments.
[Table]
[Table]
Regression Experimental details: For regression we use a two layer selection network with 16 units each followed by tanh non-linearities.
The experts are shallow neural networks with a single hidden layer that learn log-variance and mean of a Gaussian distribution which they use for prediction.
We use the “Huber Loss” instead of MSE as it is more robust (Balasundaram & Meena, 2019).
We optimize all networks using Adam (Kingma & Ba, 2014).