L EARNING WITH AMIG O : A DVERSARIALLY M OTIVATED I NTRINSIC G OALS
A BSTRACT
A key challenge for reinforcement learning (RL) consists of learning in environ ments with sparse extrinsic rewards
In contrast to current RL methods, humans are able to learn new skills with little or no reward by using various forms of intrinsic motivation.
We propose AMIG O , a novel agent incorporating—as form of meta-learning—a goal-generating teacher that proposes A dversarially M otivated I ntrinsic G O als to train a goal-conditioned “student” policy in the absence of (or alongside) environment reward.
Speciﬁcally, through a simple but effective “constructively adversarial” objective, the teacher learns to propose increasingly challenging—yet achievable—goals that allow the student to learn general skills for acting in a new environment, independent of the task to be solved.
We show that our method generates a natural curriculum of self-proposed goals which ultimately allows the agent to solve challenging procedurally-generated tasks where other forms of intrinsic motivation and state-of-the-art RL methods fail
I NTRODUCTION
The success of Deep Reinforcement Learning (RL) on a wide range of tasks, while impressive, has so far been mostly conﬁned to scenarios with reasonably dense rewards (e.g. Mnih et al., 2016; Vinyals et al., 2019), or to those where a perfect model of the environment can be used for search, such as the game of Go and others (e.g. Silver et al., 2016; Duan et al., 2016; Moravc ´ ık et al., 2017).
Many real-world environments offer extremely sparse rewards, if any at all.
In such environments, random exploration, which underpins many current RL approaches, is likely to not yield sufﬁcient reward signal to train an agent, or be very sample inefﬁcient as it requires the agent to stumble onto novel rewarding states by chance.
In contrast, humans are capable of dealing with rewards that are sparse and lie far in the future.
For example, to a child, the future adult life involving education, work, or marriage provides no useful reinforcement signal.
Instead, children devote much of their time to play, generating objectives and posing challenges to themselves as a form of intrinsic motivation.
Solving such self-proposed tasks encourages them to explore, experiment, and invent; sometimes, as in many games and fantasies, without any direct link to reality or to any source of extrinsic reward.
This kind of intrinsic motivation might be a crucial feature to enable learning in real-world environments (Schulz, 2012) .
To address this discrepancy between na¨ıve deep RL exploration strategies and human capabilities, we present a novel meta-learning method wherein part of the agent learns to self-propose A dversarially M otivated I ntrinsic Go als (AMIG O ).
In AMIG O , the agent is decomposed into a goal-generating teacher and a goal-conditioned student policy.
The teacher acts as a constructive adversary to the [Figure] student: the teacher is incentivized to propose goals that are not too easy for the student to achieve, but not impossible either.
This results in a natural curriculum of increasingly harder intrinsic goals that challenge the agent and encourage learning about the dynamics of a given environment.
AMIG O can be viewed as an augmentation of any agent trained with policy gradient-based methods.
Under this view, the original policy network becomes the student policy, which only requires its inputprocessing component to be adapted to accept an additional goal speciﬁcation modality.
The teacher policy can then be seen as a “bolt-on” to the original policy network, entailing that this method is—to the extent that the aforementioned goal-conditioning augmentation is possible—architecture-agnostic, and can be used on a variety of RL training model architectures and training settings.
As advocated in recent work (Cobbe et al., 2019; Zhong et al., 2020; Risi & Togelius, 2019; K¨uttler et al., 2020), we evaluate AMIG O for procedurally-generated environments instead of trying to learn to perform a speciﬁc task.
Procedurally-generated environments are challenging since agents have to deal with a parameterized family of tasks, resulting in large observation spaces where memorizing trajectories is infeasible.
Instead, agents have to learn policies that generalize across different environment layouts and transition dynamics (Rajeswaran et al., 2017; Machado et al., 2018;
Foley et al., 2018; Zhang et al., 2018).
Concretely, we use MiniGrid (Chevalier-Boisvert et al., 2018), a suite of fast-to-run procedurally-generated environments with a symbolic/discrete (expressed in terms of objects like walls, doors, keys, chests and balls) observation space which isolates the problem of exploration from that of visual perception.
MiniGrid is a widely recognized challenging benchmark for intrinsic motivation, which was used in many recent publications such as Goyal et al.
2019, Bougie et al.
2019, Raileanu and Rockt¨aschel 2020, Modhe et al.
2020 etc.
We evaluate our method on six different tasks from the MiniGrid domain with varying degrees of difﬁculties, in which the agent needs to acquire a diverse range of skills in order to succeed.
Furthermore, MiniGrid is complex and competitive baselines such as IMPALA (that achieve SOTA in other domains like
Atari) fail.
Raileanu & Rockt¨aschel (2020) found that MiniGrid presents a particular challenge for existing state-of-the-art intrinsic motivation approaches.
Here, AMIG O sets a new state-of-the-art on some of the hardest MiniGrid environments, being the only method capable of successfully obtaining extrinsic reward on some of them.
In summary, we make the following contributions: (i) we propose Adversarially Motivated Intrinsic
G O als—an approach for learning a teacher that generates increasingly harder goals, (ii) we show, through 114 experiments on 6 challenging exploration tasks in procedurally generated environments, that agents trained with AMIG O gradually learn to interact with the environment and solve tasks which are too difﬁcult for state-of-the-art methods, and (iii) we perform an extensive qualitative analysis and ablation study.
R ELATED W ORK
Our work has connections to many different research areas but due to space constraints, we will focus our discussion on the most closely related topics, namely intrinsic motivation and curriculum learning.
Intrinsic motivation (Oudeyer et al., 2007; Oudeyer & Kaplan, 2009; Schmidhuber, 1991; Barto, 2013) methods have proven effective for solving various hard-exploration tasks (Bellemare et al., 2016; Pathak et al., 2017; Burda et al., 2019b).
One prominent formulation is the use of novelty , which in its simplest form can be estimated with state visitation counts (Strehl & Littman, 2008) and has been extended to high-dimensional state spaces (Bellemare et al., 2016; Burda et al., 2019b;
Ostrovski et al., 2017).
Other sophisticated versions of curiosity (Schmidhuber, 1991) guide the agent to learn about environment dynamics by encouraging it to take actions that reduce the agent’s uncertainty (Stadie et al., 2015; Burda et al., 2019b), have unpredictable consequences (Pathak et al., 2017; Burda et al., 2019a), or a large impact on the environment (Raileanu & Rockt¨aschel, 2020).
Other forms of intrinsic motivation include empowerment (Klyubin et al., 2005) which encourages control of the environment by the agent, and goal diversity (Pong et al., 2019) which encourages maximizing the entropy of the goal distribution.
In Lair et al.
(2019), intrinsic goals are discovered from language supervision.
The optimal rewards framework presents intrinsic motivation as a mechanism that goes beyond exploration, placing its origin in an evolutionary context (Singh et al., 2009), or framing it as a meta-optimization problem of selecting internal agent goals which optimize the designer’s goals Sorg et al.
More recently Zheng et al.
(2018) extend this framework to learn parametric additive intrinsic rewards.
Our work differs from all of the above by formulating intrinsic motivation as a ”constructively adversarial” teacher that proposes increasingly harder goals for the agent.
Curriculum learning (Bengio et al., 2009) is another useful technique for tackling complex tasks but the curricula are typically handcrafted which can be time consuming.
In our work, the curriculum is generated automatically in an unsupervised fashion.
Another automatic curriculum approach learning was proposed by Schmidhuber (2011), where an agent constantly searches the space of problems for the next solvable one.
However, this method is not scalable to more complex tasks.
Florensa et al.
(2017) generate a curriculum by increasing the distance of the starting-point to a goal.
In contrast to AMIG O , this method assumes knowledge of the goal location and the ability to reset the agent in any state.
A student can also self-propose a goal by hindsight experience replay (HER,
Andrychowicz et al., 2017), which has been demonstrated to be effective in alleviating the sparse reward problem.
Recent extensions have improved goal selection by balancing the difﬁculty and diversity (Fang et al., 2019) of goals.
In contrast to our work, in HER, there is no explicit incentive for the agent to explore beyond its current reach.
Since HER is rewarded for all the states it visits, it is rewarded for easy-to-reach states, even late in the training process.
Other related work has trained a teacher to generate a curriculum of environments that maximize the learning process of the student (Portelas et al., 2020a).
The question of the complementarity between these and our work is worth pursuing in the future.
More similar to our work, Matiisen et al.
(2017) train a teacher to select tasks in which the student is improving the most or in which the student’s performance is decreasing to avoid forgetting.
Note that AMIG O uses a different objective for training the teacher, which encourages the agent to solve progressively harder tasks.
Similarly, Racaniere et al.
(2019) train a goal-conditioned policy and a goal-setter network in a non-adversarial way to propose feasible, valid and diverse goals.
Their feasibility criteria is similar to ours, but requires training an additional discriminator to rank the difﬁculty of the goals, while our teacher is directly trained to generate goals with an appropriate level of difﬁculty.
1 Recent surveys of curriculum generation in the context of RL include Narvekar et al.
and
Portelas et al.
(2020b).
Closer to our work, Sukhbaatar et al.
(2017) use an adversarial framework but require two modules that independently act and learn in the environment, where one module is encouraged to propose challenges to the other.
This setup can be costly and is restricted to only proposing goals which have already been reached by the policy.
Moreover, it requires a resettable or reversible environment.
In contrast, our method uses a single agent acting in the environment, and the teacher is less constrained in the space of goals it can propose.
Also similar to ours, Florensa et al.
(2018) present GoalGAN, a generator that proposes goals with the appropriate level of difﬁculty as determined by a learned discriminator.
While their work is similar in spirit with ours, there are several key differences.
First, GoalGAN was created for and tested on locomotion tasks with continuous goals, whereas our method is designed for discrete action and goal spaces.
While not impossible, adapting it to our setting is not trivial due to the GAN objective.
Second, the authors do not condition the generator on the observation which is necessary in procedurally-generated environments that change with each episode.
GoalGAN generates goals from a buffer, but previous goals can be unfeasible or nonsensical for the current episode.
Hence,
GoalGAN cannot be easily adapted to procedurally-generated environments.
A concurrent effort, Zhang et al.
(2020) complements ours, but in the context of continuous control, by also generating a curriculum of goals which are neither too hard nor too easy using a measure of epistemic uncertainty based on an ensemble of value functions.
This requires training multiple networks, which can become too computationally expensive for certain applications.
Finally, our approach is loosely inspired by generative adversarial networks (GANs, Goodfellow et al., 2014), where a generative model is trained to fool a discriminator which is trained to differentiate between the generated and the original examples.
In contrast with GANs, AMIG O does not require a discriminator, and is “constructively adversarial”, in that the goal-generating teacher is incentivized by its objective to propose goals which are challenging yet feasible for the student.
A DVERSARIALLY M OTIVATED I NTRINSIC G OALS
AMIG O is composed of two subsystems: a goal-conditioned student policy which “controls” the agent’s actions in the environment, and a goal-generating teacher (see Figure 1) which guides the student’s training.
The teacher proposes goals and is rewarded only when the student reaches the goal after a certain number of steps.
The student receives reward for reaching the goal proposed by the teacher (discounted by the number of steps needed to reach the goal).
The two components are trained adversarially in that the student maximizes reward by reaching goals as fast as possible, while the teacher maximizes reward by proposing goals which the student can reach, though not too quickly.
In addition to this intrinsic reward, both modules are rewarded when the agent solves the full task.
3. T RAINING THE STUDENT
We consider the traditional RL framework of a Markov Decision Process with a state space S , a set of actions A and a transition function p ( s t +1 | s t , a t ) which speciﬁes the distribution over next states given a current state and action.
At each time-step t , the agent in state s t ∈ S takes an action a t ∈ A by sampling from a goal-conditioned stochastic student policy π ( a t | s t , g ; θ π ) where g is a intrinsic goal provided by the teacher.
The teacher G ( s 0 ; θ g ) is a separate policy, operating on a different “granularity” than the student: it takes as input an initial state and outputs as actions a goal g for the student, which stays the same until a new goal is proposed.
The teacher proposes a new goal every time an episode begins or whenever the student reaches the intrinsic goal.
We assume that some goal veriﬁcation function v ( s, g ) can be speciﬁed as an indicator over whether a goal g is achieved in a state s .
We use this to deﬁne the undiscounted intrinsic reward r gt as: [Equation]
otherwise
At each time step t , the student receives a reward r t = r gt + r et , which is the sum of the intrinsic reward r gt provided by the teacher and the extrinsic reward r et provided by the environment.
The student, represented as a neural network with parameters θ π , is trained to maximize the discounted expected reward R t = E (cid:2) (cid:80)
Hk =
γ k r t + k (cid:3) where γ ∈ [0 , 1) is the discount factor.
We consider a ﬁnite time horizon H as provided by the environment.
3. T RAINING THE T EACHER
The teacher G ( s 0 ; θ g ) , represented as a neural network with parameters θ g , is trained to maximise its expected reward.
The teacher’s reward r T is a function of the student’s performance on the proposed goal and is computed every time this goal is reached (or at the end of an episode).
As a result, the teacher operates at a different temporal frequency, and thus its rewards are not discounted according to the number of steps taken by the agent.
To generate an automatic curriculum for the student, we positively reward the teacher if the student achieves the goal with suitable effort, but penalize it if the student either cannot achieve the goal, or can do so too easily.
There are different options for measuring the performance of the student here, but for simplicity we will use the number of steps t
+ it takes the student to reach an intrinsic goal since the intrinsic goal was set (with t + =
if the student does not reach the goal before the episode ends).
We deﬁne a threshold t ∗ such that the teacher is positively rewarded by r T when the student takes more steps than the threshold to reach the set goal, and negatively if it takes fewer steps or never reaches the goal before the episode ends.
We thus deﬁne the teacher reward as follows, where α and β are hyperparameters (see Section 4.2 for implementation details) specifying the weight of positive and negative teacher reward:
[Equation]
One can try to calibrate a ﬁxed target threshold t ∗ to force the teacher to propose increasingly more challenging goals as the student improves.
Initial experiments with a ﬁxed threshold indicated that the loss function was sufﬁcient to induce harder goals and curriculum learning.
However, this threshold is different across environments and has to be carefully ﬁxed (depending on the size and complexity of the environment).
A more adaptive—albeit heuristic—approach we adopt is to linearly increase the threshold t ∗ after a ﬁxed number of times in which the student successfully reaches the intrinsic goals.
Speciﬁcally, the threshold t ∗ is increased by 1 whenever the student successfully reaches an intrinsic goal in more than t ∗ steps for ten times in a row.
This increase in the target threshold provides an additional metric to visualize the improvement of the student through the “difﬁculty” of its goals (see Figure 3).
3. T YPES OF G OALS
We can conceive of variants of AMIG O whereby goals are provided in the form of linguistic instructions, images, etc.
To prove the concept, in this framework, a goal is formally deﬁned as a change in the observation on a tile, as speciﬁed by an ( x, y ) coordinate.
The agent must modify the tile before the end of an episode (e.g. by moving to it, or causing the object in it to move or change state).
The veriﬁcation function v is then trivially the indicator function of whether the cell state is different from its initial state at the beginning of the episode.
Proposing ( x, y ) coordinates as goals can present a diverse set of ways for an agent to achieve the goal, as the coordinates can not only be affected by reaching them but also by modifying what is on them.
This includes picking up keys, opening doors, and dropping objects onto empty tiles.
In some cases in our setting, moving over a square is not the simplest thing possible (e.g. when an obstacle can be removed or a door can be opened).
Similarly, in other tasks and environments it could be easier to affect a cell by throwing something at it, rather than by reaching it.
Likewise, simply navigating to a set of coordinates (say, the corner of a locked room) might require solving several non-trivial sub-problems (e.g. identifying the right key, going to it, then going to the door, unlocking it, and ﬁnally going to the target location).
We give some examples of goals proposed by the teacher, alongside the progression in their difﬁculty as the student improves, in Figure 3.
3. A UXILIARY T EACHER L OSSES
To complement our main form of intrinsic reward, we explore a few other criteria, including goal diversity, extrinsic reward, environment change and novelty.
We report, in our experiments of
Section 4, the results for AMIG O using these auxiliary losses.
We present, in Appendix D, an ablation study of the effect of these losses, alongside some alternatives to the reward structure for the teacher network.
Diverse Goals.
One desirable property is goal diversity (Pong et al., 2019; Raileanu & Rockt¨aschel, 2020).
In our implementation of AMIG O in the experiments of Section 4, we used entropy regulariza- tion to train the teacher and student, which encourages such diversity.
This regularization, along with the scheduling of the threshold, helps the teacher avoid getting stuck in local minima.
Additionally, we considered rewarding the teacher for proposing novel goals similar to count-based exploration methods (Bellemare et al., 2016; Ostrovski et al., 2017) with the difference that in our case the counts are for goals instead of states, based on the number of times the teacher presents a type of goal to the student.
This did not improve performance and is not part of our model for the rest of the paper.
Episode Boundary Awareness.
When playing in a procedurally-generated environment, humans will notice the factors of variation and exploit them.
In episodic training, RL agents and algorithms are informed if a particular state was an episode end.
To bias AMIG O towards learning the factors of variation in an environment, while not giving it any domain knowledge or any privileged information which other comparable intrinsic motivation systems and RL agents would not have access to, we positively reward the teacher if the content of the goal location it proposes changes at an episode boundary, regardless of whether this change was due to the agent.
Thus, the teacher is rewarded for selecting goals where the object type changes if the episode changes (for example a door becomes a wall, or a key becomes an empty tile due to the new episode conﬁguration).
While this heuristic is quite general and could be effective for many tasks as it encourages agents to note environmental factors of variation, we note it might not be useful in all possible domains and as such is not an essential part of AMIG O .
A comparison an extension of this loss other intrinsic motivation methods would interesting, but is not straightforward and is left for future research, we just note that this auxiliary loss on its own is not able to solve even the medium difﬁculty environments.
Extrinsic Goals.
To help transition into the extrinsic task and avoid local minima, we reward both the teacher and the student with environment reward whenever the student reaches the extrinsic goal, even if this does not coincide with the intrinsic goal set by the teacher.
This avoids the degenerate case where the student becomes good at satisfying the extrinsic goal, and the teacher is forced to encourage it “away” from it.
E XPERIMENTS
We follow Raileanu & Rockt¨aschel (2020) and evaluate our models on several challenging procedurally-generated environments from MiniGrid (Chevalier-Boisvert et al., 2018).
This environment provides a good testbed for exploration in RL since the observations are symbolic rather than high-dimensional, which helps to disentangle the problem of exploration from that of visual understanding.
We compare AMIG O with state-of-the-art methods that use various forms of exploration bonuses.
We use TorchBeast (K ¨ uttler et al., 2019), a PyTorch platform for RL research based on IMPALA (Espeholt et al., 2018) for fast, asynchronous parallel training.
The code for these experiments is included in the supplementary materials, and has also been released under ”https://anonymous” to facilitate reproduction of our method and its use in other projects.
[Figure]
4. E NVIRONMENTS
We evaluate AMIG O on the following MiniGrid environments: KeyCorrS3R3 ( KCmedium ),
ObstrMaze1Dl ( OMmedium ), ObstrMaze2Dlhb ( OMmedhard ), KeyCorrS4R3 ( KChard ), KeyCorrS5R3 ( KCharder ), and ObstrMaze1Q ( OMhard ).
The agent receives a full observation of the
MiniGrid environment.
The layout of the environment changes at every episode as it is procedurallygenerated.
Examples of these tasks can be found in Figure 2.
Each environment is a grid of size
N × N ( N being environment-speciﬁc) where each tile contains at most one of the following colored objects: wall, door, key, ball, chest.
An object in each episode is selected as an extrinsic goal.
If the agent reaches the extrinsic goal, or a maximum number of time-steps is reached, the environment is reset.
The agent can take the following actions: turn left, turn right, move forward, pick up an object, drop an object, or toggle (open doors or interact with objects).
Each tile is encoded using three integer values: the object, the color, and a type or ﬂag indicating whether doors are open or closed.
While policies could be learned from pixel observation alone, we will see below that the exploration problem is sufﬁciently complex with these semantic layers, owing to the procedurally generated nature of the tasks.
The observations are transformed before being fed to agents by embedding each tile of the observed frame into a single representation encoding the object type, color, and type/ﬂag.
The extrinsic reward provided by each environment for reaching the extrinsic goal in t steps is r et = 1 − ( . 9 · t ) /t max , where t max is the maximum episode length (which is intrinsic to each environment and set by the MiniGrid designers), if the extrinsic goal is reached at t , and 0 otherwise.
Episodes end when the goal is reached, and thus the scale of the positive reward encourages agents to reach the goal as quickly as possible.
4. AMIG O I MPLEMENTATION
The teacher is a dimensionality-preserving network of four convolutional layers interleaved with exponential linear units.
Similarly, the student consists of four convolutional layers interleaved with exponential linear units followed by two linear layers with rectiﬁed linear units.
Both the student and the teacher are trained using the TorchBeast (K ¨ uttler et al., 2019) implementation of
IMPALA (Espeholt et al., 2018), a distributed actor-critic algorithm.
But while the teacher proposes goals only at the beginning of an episode or when the student reaches a goal, the student produces an action and gets a reward at every step.
To replicate the structure of reward for reaching extrinsic goals, intrinsic reward for the student is discounted to r gt = 1 − ( . 9 · t ) /t max when v ( s t , g ) = 1 , and 0 otherwise.
The hyperparameters for the reward for the teacher r T are grid searched, and optimal values are found at α = .
7 and β = .
3 (see Appendix B for full hyperparameter search details).
4. B ASELINES AND E VALUATION
We use IMPALA (Espeholt et al., 2018) without intrinsic motivation as a standard deep RL baseline.
We then compare AMIG O to a series of methods that use intrinsic motivation to supplement extrinsic reward, as listed here.
Count is Count-Based Exploration from Bellemare et al.
(2016), which computes state visitation counts and gives higher rewards to less visited states.
RND is Random
Network Distillation Exploration by Burda et al.
(2019b) which uses a random network to compute a prediction error used as a bonus to reward novel states; ICM is Intrinsic Curiosity Module from Pathak et al.
(2017), which trains forward and inverse models to learn a latent representation used to compare the predicted and actual next states.
The Euclidean distance between the representations of predicted and actual states (as measured in the latent space) is used as intrinsic reward.
RIDE , from Raileanu & Rockt¨aschel (2020), deﬁnes the intrinsic reward as the (magnitude of the) change between two consecutive state representations.
We have noted from the literature that some of these baselines were designed for partially observable environments (Raileanu & Rockt¨aschel, 2020; Pathak et al., 2017)
so they might beneﬁt from observing an agent-centric partial view of the environment rather than a full absolute view (Ye et al., 2020).
Despite our environment being fully observable, for the strongest comparison with AMIG O we ran the baselines in each of the following four modes: full observation of the environment for both the intrinsic reward module and the policy network, full observation for the intrinsic reward and partial observation for the policy, partial view for the intrinsic reward and full view for the policy, and partial view for both.
We use an LSTM for the student policy network when it is provided with partial observations and a feed-forward network when provided with full observations.
In
Section 4.4, we report the best result (across all four modes) for each baseline and environment pair, with a full breakdown of the results in Appendix A. This, alongside a comprehensive hyperparameter search, ensures that AMIG O is compared against the baselines trained under their individually best-performing training arrangement.
We also compare AMIG O to the authors’ implementationof
Asymmetric Self-Play ( ASP ) (Sukhbaatar et al., 2017).
In their reversible mode two policies are trained adversarially:
Alice starts from a start-point and tries to reach goals, while Bob is tasked to travel in reverse from the goal to the start-point.
We ran each experiment with ﬁve different seeds, and report in Section 4.4 the means and standard deviations.
The full hyperparameter sweep for AMIG O and all baselines is reported in Appendix B, alongside best hyperparameters across experiments.
4. R ESULTS AND D ISCUSSION
We summarize the main results of our experiments in Table 1.
As discussed in Section 4.3, the reported result for each baseline and each environment is that of the best performing conﬁguration for the policy and intrinsic motivation system for that environment, as reported in Tables 2–5 of
Appendix A.
This aggregation of 114 experiments (not counting the number of times experiments were run for different seeds) ensures that each baseline is given the opportunity to perform in its best setting, in order to fairly benchmark the performance of AMIG O .
[Table]
IMPALA and Asymmetric Self-Play are unable to pass any of these medium or hard environments.
ICM and Count struggle on the “easier” medium environments, and fail to obtain any reward from the hard ones.
Only RND and RIDE perform competitively on the medium environments, but struggle to obtain any reward on the harder environments.
Our results demonstrate that AMIG O establishes a new state of the art in harder exploration problems in MiniGrid.
On environments with medium difﬁculty such as KCmedium , OMmedium , and
OMmedhard , AMIG O performs comparably to other state-of-the-art intrinsic motivation methods.
AMIG O is often able to successfully reach the extrinsic goal even on the hardest tasks.
To showcase results and sample complexity, we illustrate and discuss how mean extrinsic reward changes during training in Appendix C. To analyze which components of the teacher loss were important, we present, in Appendix D, an ablation study over the components presented in Section 3.4.
Qualitatively, the learning trajectories of AMIG O display interesting and partially adversarial dynamics.
These often involve periods in which both modules cooperate as the student becomes able to reach the proposed goals, followed by others in which the student becomes too good, forcing a drop in the teacher reward, in turn forcing the teacher to increase the difﬁculty of the proposed goals and forcing the student to further explore.
In Appendix E, we provide a more thorough qualitative analysis of AMIG O , wherein we describe the different phases of evolution in the difﬁculty of the intrinsic goals proposed by the teacher, as exempliﬁed in Figure 3.
Further goal examples are shown in Figure 6 of Appendix F. [Figure]
C ONCLUSION
In this work, we propose AMIG O , a meta-learning framework for generating a natural curriculum of goals that help train an agent as a form of intrinsic reward, to supplement extrinsic reward (or replace it if it is not available).
This is achieved by having a goal generator as a teacher that acts as a constructive adversary, and a policy that acts as a student conditioning on those goals to maximize an intrinsic reward.
The teacher is rewarded to propose goals that are challenging but not impossible.
We demonstrate that AMIG O surpasses state-of-the-art intrinsic motivation methods in challenging procedurally-generated tasks in a comprehensive comparison against multiple competitive baselines, in a series of 114 experiments across 6 tasks.
Crucially, it is the only intrinsic motivation method which allows agents to obtain any reward on some of the harder tasks, where non-intrinsic RL also fails.
The key contribution of this paper is a model-agnostic framework for improving the sample complexity and efﬁcacy of RL algorithms in solving the exploration problems they face.
In our experiments, the choice of goal type imposed certain constraints on the nature of the observation, in that both the teacher and student need to fully observe the environment, due to the goals being provided as absolute coordinates.
Technically, this method could also be applied to partially observed environments where part of the full observation is uncertain or occluded (e.g. “fog of war” in StarCraft), as the only requirement is that absolute coordinates can be provided and acted on.
However, this is not a fundamental requirement, and in future work we would wish to investigate the cases where the teacher could provide more abstract goals, perhaps in the form of language instructions which could directly specify sequences of subgoals.
Other extensions to this work worth investigating are its applicability to continuous control domains, visually rich domains, or more complex procedurally generated environments such as (Cobbe et al., 2019).
Until then, we are conﬁdent we have proved the concept in a meaningful way, which other researchers will already be able to easily adapt to their model and RL algorithm of choice, in their domain of choice. A CKNOWLEDGMENTS
We thank the anonymous reviewers for their candid and helpful feedback and discussions.
Roberta was supported by the DARPA L2M grant.
A F ULL RESULTS
Tables 2–5 show the ﬁnal performance of the intrinsic motivation baselines trained using one of four different training regimes enumerated in Section 4.3.
For each baseline, we train on KCMedium and OMmedium , and use the best hyperparameters for each task (for that particular baseline and training regime) to train it on the remaining harder versions of those environments (i.e. on KChard and KCharder or OMmedhard and OMhard , respectively).
For IMPALA, the numbers reported for KCmedium and OMmedium are from the experiments in Raileanu & Rockt¨aschel (2020), while the numbers for the harder environments are presumed to be .
because IMPALA fails to train on simpler environments.
As a sanity check, we also veriﬁed that ASP learns successfully in easier environments not considered here, such as MiniGrid-Empty-Random-5x5-v0, and MiniGrid-KeyCorridorS3R1-v0, to validate the ofﬁcial PyTorch implementation.
Tables 2–5 indicate that the best training regime for the intrinsic motivation baselines (for all the tasks they can reliably solve) is the one that uses a partially observed intrinsic reward and a partially observed policy (Table 5).
When the intrinsic reward is based on a full view of the environment,
Count and RND will consider almost all states to be ”novel” since the environment is procedurallygenerated.
Thus, the reward they provide will not be very helpful for the agent since it does not transfer knowledge from one episode to another (as is the case in ﬁxed environments (Bellemare et al., 2016; Burda et al., 2019b)).
In the case of RIDE and ICM, the change in the full view of the environment produced by one action is typically a single number in the MiniGrid observation.
For
ICM, this means that the agent can easily learn to predict the next state representation, so the intrinsic reward might vanish early in training leaving the agent without any guidance for exploring (Raileanu & Rockt¨aschel, 2020).
For RIDE, it means that the intrinsic reward will be largely uniform across all state-action pairs, thus not differentiating between more and less ”interesting” states (which it can do when the intrinsic reward is based on partial observations (Raileanu & Rockt¨aschel, 2020)).
B H YPERPARAMETER S WEEPS AND B EST V ALUES
For AMIG O , we grid search over batch size for student and teacher ∈ { 8 , 32 , 150 } , learning rate for student ∈ { .
, learning rate for teacher ∈ { . 05 , .
0001 } unroll length ∈ { 50 , 100 } , entropy cost for student ∈ { . 0005 , .
001 , . 0001 } , entropy cost for teacher ∈ { . 001 , .
embedding dimensions for the observations ∈ { 5 , 10 , 20 } , embedding dimensions for the student last linear layer ∈ { 128 , 256 } , and teacher loss function parameters α and β ∈ { 1 . 0 , 0 . 7 , 0 . 5 , 0 . 3 , 0 . 0 } .
For RND, RIDE, Count, and ICM, we used learning rate 10 − 4 , batch size 32, unroll length 100,
RMSProp optimizer with (cid:15) = 0 .
01 and momentum 0, which were the best values found for these methods on MiniGrid tasks by Raileanu & Rockt ¨ aschel (2020).
We further searched over the entropy coefﬁcient ∈ { 0 . 0005 , 0 . 001 , 0 . 0001 } and the intrinsic reward coefﬁcient ∈ { 0 . 1 , 0 . 01 , 0 . 5 } on KCmedium and OMmedium.
The results reported in Tables 2, 3 and 4 use the best values found from these experiments, while the results reported in Table 5 use the best parameter values reported by Raileanu & Rockt ¨ aschel (2020).
For ASP, we ran the authors’ implementation using its reverse mode.
We used the defaults for most hyperparameters, grid searching only over sp steps ∈ { 5 , 10 , 20 } , sp test rate ∈ { . 1 , . 5 } , and sp alice entr ∈ { . 003 , . 03 } .
The best hyperparameters for AMIG O and each baseline are reported below:
AMIG O : a student batch size of 8, a teacher batch size of 150, a student learning rate of .001, a teacher learning rate of .001, an unroll length of 100, a student entropy cost of .0005, a teacher entropy cost of .01, and observation embedding dimension of 5, a student last layer embedding dimension of 256, and ﬁnally, α = 0 .
7 and β = 0 .
RND: partially observed intrinsic reward, partially observed policy, entropy cost of .0005, intrinsic reward coefﬁcient of .
RIDE: for KCmedium , partially observed intrinsic reward, partially observed policy, entropy cost of .0005, intrinsic reward coefﬁcient of .1; for OMmedium : fully observed intrinsic reward, partially observed policy, entropy cost of .0005, intrinsic reward coefﬁcient of .
COUNT: partially observed intrinsic reward, partially observed policy, entropy cost of .0005, intrinsic reward coefﬁcient of .
ICM: for KCmedium , partially observed intrinsic reward, fully observed policy, entropy cost of .0005, intrinsic reward coefﬁcient of .1; for OMmedium : partially observed intrinsic reward, partially observed policy, entropy cost of .0005, intrinsic reward coefﬁcient of .
ASP: Best performing hyperparameters (in the easier environments) were 10 sp steps, a sp test rate of .5, and Alice entropy of .003.
All other hyperparameters used the defaults in the codebase.
C S AMPLE E FFICIENCY
We show, in Figure 4, the mean extrinsic reward over time during training for the best conﬁguration of the various methods.
The ﬁrst row consists of intermediately difﬁcult environments in which different forms of intrinsic motivation perform similarly, the ﬁrst two evironments require less than 30 million steps while OMmedhard and the three more challenging environments of the bottom rows require in the order of hundreds of millions of frames.
Any plot where a method’s line is not visible indicates that the method is consistently failing to reach reward states throughout its training. [Figure] m ean _ep i s ode _ r e t u r n [Figure]
m ea n_e p i s od e_ r e t u [Figure] The important point of note here is that on the two easiest environments, KCmedium and
OMmedium , agents need about 10 million steps to converge while on the other four more challenging environments, they need an order of 100 million steps to learn the tasks, showcasing AMIG O ’s contributions not just to solving the exploration problem, but also to improving the sample complexity of agent training.
D A BLATION S TUDY
To further explore the effectiveness and robustness of our method, in this subsection we investigate the alternative criteria discussed in Section 3.4.
We compare the F ULL M ODEL with its ablations and alternatives consisting of removing the extrinsic bonus (N O E XTRINSIC ), removing the environment
We also considered two alternative reward forms for the teacher to provide a more continuous and gradual reward than the previously introduced “all or nothing” threshold.
We consider a Gaussian p ∼ Normal ( t ∗ , σ ) reward around the target threshold t ∗ :
[Equation]
and a Linear-Exponential reward which grows linearly towards the threshold and then decays exponentially as the goal proposed becomes too hard (as measured according to the number of steps):
[Equation]
We report these two alternative forms of reward as (G AUSSIAN and L INEAR -E XPONENTIAL ) in the study below. [Table] L INEAR -E XP
Performance is shown in Table 6 where the number of steps needed to converge (to the ﬁnal extrinsic reward) is reported.
A positive number means the model learned to solve the task, while 0 means the model did not manage to get any extrinsic reward.
For all models we encourage goal diversity on the teacher with a high entropy coefﬁcient of .05 (as compared to .
0005 of the student).
As the table shows, removing the extrinsic reward or the environment change bonus severely hurts the model, making it unable to solve the harder environments.
The novelty bonus was minimally beneﬁcial in one of the environments (namely KChard ) but slightly ineffective on the others.
The more gradual reward forms considered are not robust to the learning dynamics and often result in the system going into rabbit holes where the algorithm learns to propose goals which provide sub-optimal rewards, thus not helping to solve the actual task.
Best results across all environments in our Full
Model were obtained using the simple threshold reward function along with entropy regularization and in combination with the extrinsic reward and changing bonuses, but without the novelty bonus.
E Q UALITATIVE A NALYSIS
To better understand the learning dynamics of AMIG O , Figure 5 shows the intrinsic reward throughout training received by the student (top panel) as well as the teacher (middle panel).
The bottom panel shows the difﬁculty of the proposed goals as measured by the target threshold t ∗ used by the teacher (described in Section 3).
The trajectories reﬂect interesting and complex learning dynamics.
For visualization purposes we divide this learning period into ﬁve phases: Phase 1 : The student slowly becomes able to reach intrinsic goals with minimal difﬁculty.
The teacher ﬁrst learns to propose easy nearby goals.
Phase 2 : Once the student learns how to reach nearby goals, the adversarial dynamics cause a drop in the teacher reward which is then forced to explore and propose harder goals.
Phase 3 : An equilibrium is found in which the student is forced to learn to reach more challenging goals.
[Figure]
Phase 4 : The student becomes too capable again and the teacher is forced to increase the difﬁculty of the proposed goals.
Phase 5 : The difﬁculty reaches a state where it induces a new equilibrium in which the student is unable to reach the goals and forced to improve its student.
AMIG O generates diverse and complex learning dynamics that lead to constant improvements of the agent’s policy.
In some phases, both components beneﬁt from learning in the environment (as is the case during the ﬁrst phase), while some phases are completely adversarial (fourth phase), and some phases require more exploration from both components (i.e. third and ﬁfth phases).
Figure 3, presented in Section 4.4, further exempliﬁes a typical curriculum in which the teacher learns to propose increasingly harder goals.
The examples show some of the typical goals proposed at different learning phases.
First, the teacher proposes nearby goals.
After some training, it learns to propose goals that involve traversing rooms and opening doors.
Eventually, the teacher proposes goals which involve interacting with different objects.
Despite the increasing capacity of the agent to interact with the environment, OMhard remains a challenging task and AMIG O learns to solve it in only one of the ﬁve runs.
F G OAL E XAMPLES
Figure 6 shows examples of goals proposed by the agent during different stages of learning.
Typically, in early stages the teacher learns to propose easy nearby goals.
As learning progresses it is incentivized to proposed farther away goals that often involve traversing rooms and opening doors.
Finally, in later stages the agent often learns to propose goals that involve removing obstacles and interacting with objects.
We often observe this before the policy achieves any extrinsic reward.