Model-Based Meta Automatic Curriculum Learning
Abstract
When an agent trains for one target task, its experience is expected to be useful for training on another target task.
This paper formulates the meta curriculum learning problem that builds a sequence of intermediate training tasks, called a curriculum, which will assist the learner to train toward any given target task in general.
We propose a model-based meta automatic curriculum learning algorithm ( MM-ACL ) that learns to predict the performance improvement on one task when the policy is trained on another, given contextual information such as the history of training tasks, loss functions, rollout state-action trajectories from the policy, etc.
This predictor facilitates the generation of curricula that optimizes the performance of the learner on different target tasks.
Our empirical results demonstrate that MM-ACL outperforms a random curriculum, a manually created curriculum, and a commonly used nonstationary bandit algorithm in a G RID W ORLD domain.
1. Introduction
Curriculum Learning (CL) is the problem of generating a sequence of source tasks for a reinforcement learning agent (learner) to train on, in order to improve some objective related to the learner’s performance on a target task .
To address this problem, automatic curriculum learning (ACL) has demonstrated its success by improving sample efficiency (Schaul et al., 2015), dealing with sparse reward (Zheng et al., 2018; Durugkar et al., 2021), facilitating generalization (Akkaya et al., 2019), and finding solutions to hardexploration challenges (Ecoffet et al., 2019) .
The curriculum is realized as a sequence of intermediate tasks such as the next environment (Narvekar et al., 2017) or distribution of environments (Portelas et al., 2020a) to train on, or a sequence of learning contexts such as the discount factor, bootstrapping parameter (Xu et al., 2018), and intrinsic rewards (Zheng et al., 2018).
In an attempt to find a good curriculum, most existing work optimizes surrogate objectives such as learning progress, diversity, or intermediate difficulty (Portelas et al., 2020b).
However, this approach may lead to sub-optimal solutions or failures when the surrogate objective does not align well with the intrinsic learning objective.
Rather than solving for surrogate objectives, there is a line of ACL work that directly optimizes the learning time (Narvekar et al., 2017; Narvekar & Stone, 2018).
In this paper, we optimize a similar objective directly on a given target task, i.e., the performance on the target task, using the novel MM-ACL algorithm.
First, this algorithm learns from training different target tasks.
Specifically, it learns to predict the performance on one task while training on another, based on the offline data collected from multiple lifetimes, where at each lifetime the learner trains toward a different target task.
Based on this model, MM-ACL chooses the next intermediate task to train, such that the expected performance on the chosen target task is maximized.
This process enables directly optimizing the performance on the target task, instead of focusing on the surrogate goals.
2. Problem description
Meta curriculum learning problem Consider a domain that includes a collection of possible source tasks W ∪ V , where each task is characterized by controllable parameters w ∈ W , representing its task control or learning contexts, and uncontrolled parameters v ∈ V that remain fixed when training a target task, but may differ between different target tasks.
Since v is fixed in a curriculum, we refer to the task instantiated by the parameter set w as “task w ”.
The policy is parameterized by θ .
To find the best policy for a target task w target , a straightforward approach is to use gradientbased optimization by training exclusively on the target task.
However, in this approach the agent might not be able to learn a good policy due to reward sparsity or high variance of returns.
Instead, we train the learner on a sequence of tasks defined as a curriculum W = ( w 0 , w 1 , . . .
, w N ) generated by a teacher policy π C .
In particular, at each step t , given a policy θ t , we rollout this policy on a task w t ∼ π C multiple times, and use the data collected for a gradient update: θ
[Equation]
where L ( θ t , w t , v ) is the loss of policy θ t on task w t .
Given the policy θ N after N gradient updates following curriculum W , the learning progress can be characterized by the difference between the return of the initial policy θ 0 and the return of policy θ N on the target task w target .
Defining the return of a policy θ on task w as G ( θ, w , v ) , a curriculum learning problem is to find the best curriculum that maximize the learning progress:
[Equation]
A meta curriculum learning problem, instead, seeks to find a curriculum learning policy π C that maximizes the learning progress over multiple lifetimes.
We define a lifetime as a two-tuple ( w target , v ) that differs from other lifetimes by its target task and uncontrolled parameters.
Taking lifetimes into consideration, the objective of a meta curriculum learning problem becomes finding a curriculum policy π C that maximizes the expectation of learning progress over a distribution of lifetimes D :
[Equation]
Optimizing local learning progress Due to the intractability of directly finding a sequence of N source tasks, we select the best task locally at each time step.
To avoid insufficient training in each source task and coming back to the same source task frequently, we compute k gradient updates for each source task at each time step, ∀ j = 1 , ..., k : θ [Equation]
[Equation]
And the local objective is to choose the immediate task w t to train next, in order to maximize the learning progress
P kt ( w t , w target , v ) of this policy after k gradient updates, which is described as follows:
[Equation]
, w target , v ) Here, we use superscript val to denote the gain/losses for validating the learning progress only, and train to denote the gain/losses used for training and updating the weights.
3. MM-ACL
One common method to decide the task control parameters for the next iteration is to update the parameters by a meta gradient, or w t + k = w t − β ∇ w t P kt ( w t , w target , v ) (Xu et al., 2018).
However, in curriculum learning, the task control parameters are usually start-goal locations or deeply embedded in the transition dynamics.
Therefore, the loss functions are usually non-differentiable with respect to these parameters unless the dynamics model of the environment is given.
An alternative approach is to approximate the local learning progress P kt ( w t , w target , v ) with a black box function, and then choose the best task w t based on the approximated learning progress.
Instead of approximating the local learning progress on the target task, we learn a general function that captures the learning progress from w to any other task w ′ .
Formally, we define a learning dynamics model that models local learning progress as function f k ( c, w , w ′ , v ) , where c is the context, w is the task the agent is trained on, w ′ is the task the agent is evaluated on, and v is the set of uncontrolled parameters.
There are several ways to represent the context, such as the following.
No context : if no context is provided, the learning dynamics model is simply represented as f k ( w , w ′ , v ) , which is equivalent to modeling the curriculum learning as a non-stationary bandit problem (Sutton & Barto, 2018).
Under this representation, a function R ( w ) :
= f k ( w , w target , v ) can be thought of as the reward function for choosing a bandit arm w for only one lifetime with fixed ( w target , v ) (Matiisen et al., 2019).
Such a modeling enables learning f k adaptively in a single lifetime, 2) Training tasks and losses : the context is represented as c t = ( w 0 , L
train [Equation] [Equation] , a sequence of all the history of training tasks and associated losses.
Such a context is lightweight and does not add additional computation overhead (e.g., more rollouts or gradient updates).
3) Validation trajectories : the context is represented as c t = { τ jt }
j = J j =1 , a sequence of state-action pairs generated from evaluating the current policy θ t on the task w target .
Trajectories carry more information, but require much more training data to learn to efficiently encode the stateaction pairs (Fang et al., 2020).
For the rest of this paper, we focus on the context of training tasks and losses , with which the learning dynamics model can be trained with a relatively small amount of data, while still be applied for different lifetimes.
In addition, in practice, uncontrolled parameters are usually difficult to be encoded and are usually hidden to the teacher.
Therefore, we consider only the learning dynamics model of form f kϕ ( c, w , w ′ ) that does not take v as an input.
Once a learning dynamics model is learned, the curriculum learning policy can be created by greedily selecting the task that maximizes the local learning progress, or π C = arg max w [ f kϕ ( c, w , w target )] .
A learning dynamics model f kϕ ( c, w , w ′ ) parameterized by ϕ is a regression model that takes an input of three-tuple
1: Input:
2: Output: ϕ 3: Initialize D = {} , ϕ 4: for l = 1 to NumLifetime do 5:
Initialize θ 0 , c 0 = ∅ , ( w , v ) ∼ D . 6: while t <
NumIteration do 7: w train ← SelectTask ( c t , w target , ϕ ) 8: for j = 0 to k do 9:
L traint + j ← L ( θ t + j , w train , v ) 10: θ t + j +1 =
θ t + j − α ∇ θ t + j L traint + j 11: c t + j +1 ← c t + j ∪ ( w train , L traint + j ) 12: end for 13: for j = 1 to NumV alidationTask do 14: w val ← GetRandomTask () 15:
L valt + k ← L ( w val , θ t + k , v ) 16:
L valt ← L ( w val , θ t , v ) 17:
D ← D ∪ ( c t , w train , w val , L valt + k − L valt ) 18: end for 19: if t + 1 mod UpdateInterval =
=
then 20: ← UpdateModel ( D , ϕ ) 21: end if 22: t ← t + k 23: end while 24: end for
( c, w , w ′ ) and returns a prediction of learning progress on task w ′ after k iterations of reinforcement learning update.
The architecture of the learning dynamics model is shown in Fig.
The history training task-loss pairs are embedded by one fully-connected layer and are fed into a LSTM (Gers et al., 2000) as a sequence of task-loss embedding.
Then, the output embedding of the LSTM is concatenated with the embedding of w and w ′ followed by a MLP to compute the validation return prediction G valt + k .
Algorithm 1 shows the whole pipeline of training f kϕ .
The training data is stored in a replay buffer D collected from learners trained from scratch in multiple i.i.d lifetimes.
At an iteration t of a lifetime, a training task w train is selected based on a SelectTask function (Algorithm 2) that has a exploration probability p to select the task based on the current learning dynamics model and a probability 1 − p to select a exploratory task.
For limited discrete task space, the exploratory tasks are uniformly sampled from parameter space W .
For more complex continuous task space, the exploratory tasks are selected by ALP-GMM (Portelas et al., 2020a), an ACL algorithm that has been reported to efficiently solve more tasks compared to randomly selected curriculum in a continuous task space during the training of one single lifetime (Portelas et al., 2020a).
After a training task w train is selected, the parameters of the agent are updated for k iterations.
During the updates, the history of tasks and losses are appended to the context variable (see line 12 in
Algorithm 1).
Then, the losses L valt and L valt + k are evaluated on the same randomly sampled validation task w val before and after the updating respectively.
The learning progress, context, training and validation tasks form a new instance of training data ( c t , w train , w val , L valt + k − L valt ) which is added into a replay buffer D .
At every fixed interval of iterations, the learning dynamics model parameter ϕ is updated based on the training data in the replay buffer D .
Once the learning dynamics model is learned, the model can be deployed online following the SelectTask function with exploration probability p = 0 .
During deployment, the conext variable c t is built in the same way as line 12 in
Algorithm 1.
Notice that during deployment, lines 15 toin Algorithm 1 are not executed, which guarantees that the algorithm does not add extra rollouts and gradient updates.
4. Experimental Results
We test MM-ACL in the FourRoom domain based on the widely used MiniGrid environment by Chevalier-Boisvert et al.
As shown in Figure 2, the target task in this domain requires an agent to navigate through three rooms (labeled as room 0, 1, and 2 from left to right) and reach a goal location (green tiles) in the right-most room (room 3) by executing one of the four discrete actions: move forward, turn left, turn right and open the door.
Three types of skills are required to pass the rooms: (1) navigate around the blocks (grey tiles) which the agent cannot overlap with; (2) avoid tiles covered with lava (orange tiles with wave patterns) that cause immediate failure of an episode once the agent visit them; (3) open the door that blocks the way to the goal location.
The controllable parameter space in this domain is a four-category discrete set W = { 0 , 1 , 2 , 3 } with w equal to 0, 1, 2, 3 denoting the tasks of navigating from room 0 to 3, room 1 to 3, room 2 to 3, and room 1 to 2.
The uncontrollable parameter space V is a set of all possible room configurations.
Each configuration can have different locations of the openings and doors (see two example room configurations in Fig. 2).
Directly learning w target ( w = 0 ) is difficult because of the long navigation trajectory from room 0 to room 3 and the hard exploration due to the lava tiles that cause immediate failure.
We assume that the agent will experience multiple lifetimes, such that in each lifetime, the agent will be faced with a room configuration uniformly sampled from V with its weight reinitialized.
In the experiment, the agent is trained using PPO (Schulman et al., 2017) with its hyper-parameters kept similar to the example reinforcement learning algorithm implementation (Chevalier-Boisvert et al., 2018).
This set of hyperparameters is not guaranteed to be optimal for F OUR R OOM , but is good enough to learn the target task if a good curriculum is given.
The model is trained for 150 lifetimes.
After the training, the learning dynamics model is deployed forhold-out lifetimes that train learners from scratch with tasks selected by the learning dynamics model .
During the deployment, the agents’ performances are evaluated on all the four tasks for 8 episodes at every 4 iterations.
To investigate the dependence of performance on the amount of training data, we deployed the learning dynamics model trained with only 10 and 50 lifetimes.
The results are compared with a random curriculum, a non-stationary-bandit ACL algorithm similar to (Matiisen et al., 2019), and a manually designed curriculum that trains on tasks w = 3 , w = 1 , and w target for 500, 800, and 4000 episodes respectively.
The training curves are shown in Fig. 3(a), and the ratios of the four tasks being selected by MM-ACL and bandit algorithm for every 50 selected tasks are shown in Fig. 3(b).
As shown in Fig. 3(a), given sufficient training data oflifetimes, MM-ACL (red) can train the agent with the best sample efficiency that learns the target task with about 1.8k episodes compared to 2.8k by random curriculum and 4k by the bandit algorithm.
The sample efficiencies on tasks w = 1 and w = 3 are also marginally improved, which take about 400 less episodes to converge.
The MM-ACL s trained by 150, 50, and 10 lifetimes are shown by blue, orange and red curves in Fig. 3(a).
The sample efficiency monotonously decreases with the decreasing number of training lifetimes with MM-ACL trained by 10 lifetimes performing even worse than the random curriculum.
Surprisingly, the curriculum generated by the bandit algorithm does not benefit the training, because the bandit algorithm exploits the easiest task of w = 2 , which actually disrupts the training of harder tasks.
By carefully analyzing the tasks in this domain, we find that the skill of avoiding lava trained by task w = 3 is the key to learning more difficult task, since directly training on w = 0 and w =can easily step into the lava and cause immediate failures that make the training very difficult.
Instead, the curriculum generated by MM-ACL focuses evenly on task w = 3 and w = 0 that develop the skill of avoiding the lava and trying the target task with the developed skill.
The MM-ACL ignores the task w = 2 completely as it does not contribute to the learning of target task.
5. Conclusion
This paper first formulates a new meta curriculum learning problem, in which a teacher is required to generalize its curriculum policy, learned from multiple lifetimes, so it can train the agents toward different target tasks.
A new meta curriculum learning algorithm MM-ACL is proposed that leverages the idea of cross validations and selects the tasks that directly optimize the target task performance.
Our empirical results show an improvement of sample efficiency of the training process, tested in multiple lifetimes.