L EARNING A CHIEVEMENT S TRUCTURE FOR S TRUCTURED E XPLORATION IN D OMAINS WITH S PARSE R EWARD
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose Structured Exploration with Achievements (SEA), a multi-stage reinforcement learning algorithm that learns the environment structure with offline data and uses the learned structure to learn different skills and improve overall exploration with online environment interactions in a particular type of environment that has an internal achievement system.
SEA first uses a contrast-based loss function to learn the achievement representations and build an achievement classifier.
It then tries to recover the environment achievement structure with a heuristic algorithm.
Finally, SEA builds a meta-controller with the recovered structure to learn sub-policies and explore new tasks.
While exploration in a procedurally generated environment with high-dimensional input like images is extremely hard for reinforcement learning agents, we demonstrate that SEA is still able to recover the underlying structure and explore new tasks in different domains.
I NTRODUCTION
Exploration in complex environments with long horizon and high-dimensional input such as images has always been challenging in reinforcement learning.
In recent years, multiple works (Stadie et al., 2015; Bellemare et al., 2016; Tang et al., 2017; Pathak et al., 2017; Burda et al., 2019a;b; Badia et al., 2020) propose to use intrinsic motivation to encourage the agent to explore less-frequently visited states or transitions and gain success in some hard exploration tasks in RL.
Go-explore (Ecoffet et al., 2021) proposes to solve the hard exploration problem by building a state archive and training worker policies to reach the old states and explore new states.
The shared trait of these two streams of RL exploration algorithms is that they are both built on the idea of increasing the visiting frequency of unfamiliar states, either by providing intrinsic motivation or doing explicit planning.
However, This idea can be ineffective in procedurally generated environments since states in such environments can be infinite and scattered, to the extent that the same state may never appear twice.
This is why exploration in such environments can be extremely hard.
However, in most procedurally generated environments, there will be a finite underlying structure fixed in all environment instantiations.
This underlying structure is essentially what makes the environment “meaningful”.
Therefore, discovering this structure becomes vital in solving procedurally generated environments.
In this work, we set our sight on achievement-based environments .
Specifically, an achievementbased environment has a finite set of achievements.
The agent only gets rewarded when it completes one of the achievements for the first time in an episode.
Achievement-based environments are commonly seen in video games such as Minecraft.
We argue that this type of environment also models many real-life tasks.
For example, when we try to learn a new dish, the recipe usually decomposes the whole dish into multiple intermediate steps, where completing each step can be seen as an achievement.
A good example of achievement-based environments in reinforcement learning community is Crafter (Hafner, 2021), a 2d open-world game with procedurally generated world maps that includes 22 1 achievements.
We introduce this environment in more detail in Sec. 3.1.
Achievement-based environments bring both benefits and challenges to reinforcement learning exploration.
On the one hand, the achievement structure is a perfect invariant structure in a procedurally generated environment.
On the other hand, the reward signal in such an environment is sparse 1 21 in our modified version.
since the agent is only rewarded when an achievement is unlocked.
The reward signal can even be confusing since unlocking different achievements grants the same amount of reward.
We propose Structured Exploration with Achievements (SEA), a multi-stage algorithm that effectively solves exploration in achievement-based environments.
SEA first tries to recover the achievement structure of the environment from collected trajectories, either from experts or other bootstrapping RL algorithms.
With the recovered structure, SEA deploys a nonparametric meta-controller to learn a set of sub-policies that reach every discovered achievement and start from there to explore for new ones.
We empirically evaluate SEA in Crafter and an achievement-based environment TreeMaze that we designed based on MiniGrid (Chevalier-Boisvert et al., 2018).
In Crafter , our algorithm can complete the hard exploration achievements consistently and reach the hardest achievement ( collect diamond ) in the game at a non-negligible frequency.
None of our tested baselines can even complete any of the hard exploration achievements, and to the best of our knowledge, SEA is the first algorithm to complete this challenge.
We also show that SEA can recover the achievement dependency structure in both environments with high accuracy.
We are aware that achievement-based environments are not common in the reinforcement learning community.
However, we argue that this is in fact a good design choice.
Designing a set of achievements can be easier than crafting specific rewards system for the agent to learn.
We also show in
TreeMaze that the achievement choices do not even need to be entirely relevant for the targeted task solving.
R ELATED W ORK
Exploration in reinforcement learning.
Exploration in RL has long been an important topic, which can date back to ϵ -greedy and UCB algorithms in multi-armed-bandit problems.
In recent years, intrinsic motivation-based exploration algorithms (Stadie et al., 2015; Bellemare et al., 2016;
Tang et al., 2017; Pathak et al., 2017; Burda et al., 2019a;b; Badia et al., 2020) gained great success in this area.
These algorithms aim to find an estimation of the state or transition visiting frequency and use an intrinsic reward to encourage the agent to visit the less frequently visited states.
However, they can be ineffective when the environment is procedurally generated, where the number of states and transitions can be infinite and scattered.
Another line of work in this area relevant to ours is the go-explore (Ecoffet et al., 2021) series.
Go-explore builds an archive of explored states which are grouped into cells and learns a set of goalconditioned policies that reach each cell.
It then starts exploration from the archived cells to find new cells and updating the archive iteratively.
Go-explore shows success in the hard exploration tasks in
Atari games.
However, go-explore can also be ineffective in procedurally generated environments since it uses image down-sampling as the default state abstraction method.
Another point where our work differs from go-explore is that we don’t seek to group all states into cells, but instead only group the few import ones.
This is helpful since archiving all the states and learning all the goal-conditioned policies can take up huge amount of resources.
Hierarchical reinforcement learning.
Hierarchical reinforcement learning (HRL) explores the idea of dividing a large and complex task into smaller sub-tasks with reusable skills (Dayan &
Hinton, 1992; Sutton et al., 1999; Bacon et al., 2017; Vezhnevets et al., 2017; Nachum et al., 2018).
Eysenbach et al.
(2019) and Hartikainen et al.
(2020) propose to first learn a set of unsupervised skills then build a policy with the learned skills for faster learning and better generalization; Bacon et al.
(2017) builds a set of policy options to learn.
Goal-conditioned HRL (Vezhnevets et al., 2017; Nachum et al., 2018; 2019) train a manager policy that proposes parameterized goals and a goalconditioned worker policy to reach the proposed goals. Kulkarni et al. ; Lyu et al.
Rafati & Noelle (2019); Sohn et al.
Costales et al.
(2022) train a set of sub-policies to solve the subtasks provided by the environment and a meta-controller to maximize the reward. This line of work is similar to ours in that they assume the subtasks are provided by the environment. However, in our work, we don’t assume access to any additional subgoal information such as subgoal completion signals from the environment that are typically required in this line of work. Additionally, the problems that we are solving are more complex, namely including more tasks and being imagebased and procedurally generated.
[Figure]
Despite being conceptually similar to the subgoal/subtask in HRL, the achievement in our work bears a different meaning and serves a different purpose.
For an extended discussion on this topic, the reader can refer to Section D.1 in the appendix.
Building world models.
Watter et al.
; Karl et al.
; Ha & Schmidhuber (2018);
Hafner et al.
(2021) argue that learning a compact latent space world model if beneficial in environments with high-dimentional inputs as images.
L 3 P (Zhang et al., 2021) proposes to represent the world model as a graph by clustering states into nodes, much like the idea of ours.
However, L 3 P learns the state representation with the objective to recover reachabilility, which is not guaranteed to be useful in exploration.
Reward machines.
Reward machines (Icarte et al., 2018a;b) is a concept that uses linear temporal logic language to describe a non-Markovian reward function.
Recent work (Camacho et al., 2019;
Icarte et al., 2019; 2022) exploit this concept to do structured reinforcement learning and improve the performance.
However, they depend on the environment providing a labeling function to learn the reward machine instead of learning one from the environment.
Our definition of environment with achievements also features a non-Markovian reward function since completing one achievement only gets rewarded once in an episode, therefore it can be seen as a special instance of the reward machines.
P ROBLEM S ETUP 3. C RAFTER
To better explain the achievement-based environment, we introduce Crafter 2 (Hafner (2021),
Fig. 1a).
Crafter is a 2d open-world game featuring a procedurally generated world map andin-game achievements.
In this game, the agent needs to control the character to explore the world by collecting materials, crafting tools, and defeating monsters.
Whenever the agent completes a certain achievement for the first time in an episode, it will receive a reward of one.
For any subsequent completion, the agent will not be rewarded.
The goal of the game is to unlock as many achievements as possible.
In Crafter, some achievements may have dependencies on others.
For example, the make wood pickaxe achievement depends on the place table achievement since crafting a wood pickaxe requires the agent to be near a crafting table.
These achievements and their dependencies form a directed acyclic graph (DAG), demonstrated in Fig.
1b.
3. M ARKOV D ECISION P ROCESS WITH A CHIEVEMENTS
We formalise our definition of achievement-based environment as a Markov decision process with achievements (MDPA).
A Markov decision process with achievements is defined by a 5-tuple ( S , A , T, Γ , G ) , where S is the set of states, A is the set of actions, T : S×A → ∆ S is the stochastic state transition function, Γ is the set of achievements, and G : S
×
A × S → Γ ∪ {∅} is the achievement completion function.
For any state transition tuple ( s t , a t , s t +1 ) , the achievement completion function maps it to either one of the achievement in Γ , meaning that particular achievement is completed at this step, or ∅ , meaning none of the achievements is completed.
Achievement unlocking reward can be derived from a MDPA trajectory.
Specifically, for a trajectory τ = ( s 0 , a 0 , s 1 . . .
, s T ) , the reward at step t ∈ 0 , . . .
, T − 1 is:
[Equation]
where 1 [ · ] is the indicator function.
Note that directly adding the reward function into an MDPA breaks its Markovian property since the agent will only be rewarded for the first time it completes an achievement.
We can still derive a standard MDP from an MDPA by defining a set of achievementextended set of states S ∗ =
S × 2 Γ where we store the achievement completion information into the state.
We can also define the corresponding state transition function T ∗ : S ∗ ×
A → S ∗ and reward function R : S ∗ ×
A × S ∗ → R to complete the derivation.
M ETHOD
The advantage of learning in MDPA is that achievements, hinted by reward signals, are invariant across different environment instantiations.
These achievements can therefore be seen as checkpoints, which allows us to do a divide-and-conquer of the hard environment exploration task.
Following this argument, we propose an algorithm Structured Exploration with Achievements (SEA), which improves upon existing reinforcement learning algorithms or expert trajectories.
Specifically, we use the data collected from either trained agents or experts to recover the discovered achievements and their corresponding structures.
With the recovered achievement structure, we then train multiple sub-policies to reach every discovered achievement and start individual searches for new achievements from there.
4. A CHIEVEMENT L EARNING
The first step of our method is to identify the achievements in the environment by learning the representations of the achievements then cluster the achievements with their representations.
For any trajectory τ = ( s 0 , a 0 , . . .
, s T ) , we define U τ :=
{ t : r t = 1 } be the step set where an achievement is unlocked.
For a representation dimension W , we want to train a model parameterized by θ that maps the tuple ( s t , a t , s t +1 ) where t ∈ U τ , to a vector of size W .
We achieve this by using the uniqueness property of achievements, that every two steps in U τ correspond to different achievements.
In other words, we want to ensure the representations of each pair of steps in U τ are as distinct as possible.
Let ¯ s t :=
( s t , a t , s t +1 ) .
We define the achievement representation distance matrix of trajectory τ as
D τ :=
[ ∥ ϕ θ (¯ s t 1 ) , ϕ θ (¯ s t 2 ) ∥ 2 ] t 1 ,t 2 ∈ U τ and D τ :=
D τ / max D τ .
The normalization is used to prevent the representation from outward expanding, since expanding increases the pairwise distance but not the separability required for clustering.
To maximize the pairwise distance of the achievement representations, we use the following loss:
[Equation]
where det( · ) calculates the determinant of a matrix.
The loss reaches minimum of -1 when every pair of achievement representations in a trajectory are sufficiently different, so that the matrix we take determinant of is close to the identity matrix.
This formulation is inspired by Kulesza (2012) and Parker-Holder et al.
Compared to the more intuitive way of directly maximizing the average pairwise distances, the advantage of this formulation is that it punishes situations when the achievement representations in an episode form several clusters.
In such situations, the average pairwise distance is still high but the determinant loss is close to zero.
We then use the K-Means algorithm for clustering.
Suppose we cluster the achievements into N clusters with centroids denoted as c 1 , . . .
, c N .
We can now build an achievement classifier AC (¯ s ) :
= arg min i ∥
ϕ θ (¯ s ) − c i ∥ 2 .
However, to improve the generalization of this classifier, we would like it to gain the ability to recognize new achievements that are not in the dataset.
To do this, we set a threshold b = 0 .
· min i,j ∥ c i − c j ∥ 2 .
Let i ∗ =
arg min i ∥
ϕ θ (¯ s ) − c i ∥ 2 , the new classifier becomes:
AC (¯ s ) :
= (cid:26) i ∗ if ∥ ϕ θ (¯ s ) − c i ∗ ∥ 2 < b, −otherwise, where − 1 means ¯ s is a unrecognized new achievement.
4. A CHIEVEMENT D EPENDENCY G RAPH
With the achievements identified, we then use a heuristic algorithm to build a preliminary achievement dependency graph.
The algorithm comprises of two parts: 1) we build a complete dependency graph; 2) we trim down the complete graph into a minimal DAG.
We represent both graphs with adjacency matrix of N nodes.
We first calculate two heuristics of the collected trajectories.
[Equation]
We say i is a dependency of j if every time j happens, i happens before j .
Therefore, for the first part,
[Equation]
where ϵ is to account for identification error.
For a small ϵ , G is guaranteed to be a DAG.
[Figure] For the second part, we need to remove the unnecessary edges in G to get a minimal DAG while maintaining the topology of G .
In this case, an edge is considered unnecessary if removing it does not affect the topological order of the graph.
Since removing an edge i → j can only allow node j to appear earlier in the topological order, we only need to calculate the left-most position of a node, denoted as L EFT M OST ( G, i ) .
[Equation]
We point out that our recovered graph may include more dependency edges than the ground truth owing to the non-causal correlations from either agent preference or insufficient data.
We will address this issue in the next section.
4. S UB - POLICY L EARNING AND E XPLORATION
The last part of our algorithm is to learn a set of sub-policies that reach every achievement node and do exploration from them.
A sub-policy is defined as a parameterized achievement-conditioned policy σ ω ( a | s, g ) where ω is the parameter shared among all sub-policies and g ∈ Γ is the target achievement.
Each sub-policy only receives a reward of 1 upon reaching its target achievement, which can be done using the achievement classifier in Sec. 4.1.
To learn the sub-policies, we also need a meta-controller π ( g | s ) that proposes the target achievement to learn.
One option is to use a random controller that selects a not completed achievement uniformly randomly upon a completion of any achievement.
However, this can be ineffective when a target achievement is proposed before all its dependencies are completed.
This is because the sub-policy only has access to the reward of its target achievement, which makes the exploration much harder.
Therefore, we instead use a controller that topologically traverses the dependency graph.
Specifically, at the beginning of the episode or upon completion of any achievement, the controller randomly selects a unfinished achievement whose dependent achievements have all been completed.
In our implementation, we also bias the selection probability towards a direct child of the last completed achievement to provide continuity.
In the last section, we point out that our recovered graph may include unnecessary dependencies.
However, since we are only adding new restrictions, following our graph still guarantees that every proposed target achievements are solvable, barring any significant achievement identification errors.
With this setup, we can easily do the exploration while learning the sub-policies by adding a special exploration sub-policy.
The only reward it receives is any achievements that are not recognized by the cluster.
E XPERIMENTS 5. E XPERIMENT S ETUP
Our experiments are designed to evaluate the three main parts of our algorithm: the achievement learning, the graph recovery and the sub-policy learning and exploration.
To better investigate different aspects of our algorithm, we design a pedagogical environment based on MiniGrid (Chevalier- [Figure] Boisvert et al., 2018) called TreeMaze (Fig. 5).
TreeMaze contains 2 columns of N rooms.
Each room contains either a key that unlocks other rooms, or an object that can be interacted with, or nothing.
A key can only unlock doors with the same color as the key, while some of the rooms are not locked.
In TreeMaze , every interaction, including opening a door, picking up a key, interacting with other objects, is seen as an achievement.
The rooms are generated with the guarantee that it’s possible to unlock all achievements.
The location of the rooms are randomized but the dependency graph is fixed.
We use three versions of TreeMaze in our experiments, the TreeMaze-Easy which contains 4 rooms and 9 achievements, the TreeMaze-Normal which contains 10 rooms and 19 achievements and the TreeMaze-Hard that contains 18 rooms and 30 achievements.
For the Crafter environment, we also include a limited version ( Crafter-Limited ) which only contains 7 of the 21 achievements in the full environment.
We train a vanilla IMPALA (Espeholt et al., 2018) agent for the initial data gathering in each environment.
In the full
Crafter environment, the IMPALA agent discovers 17 achievements (with > 1% reaching rate).
In all the remaining environment, the IMPALA agents discovers all of the achievements.
We consider the 17 achievements reachable by a vanilla
IMPALA agent as the easy achievement set and the remaining 4 achievements ( make iron pickaxe , make iron sword , defeat zombie , collect diamond ) as the hard achievement set.
The reason why these achievements are particularly hard to complete is because crafting iron tools in Crafter has three prerequisites: the character near a crafting table, the character near a furnace, the character has at least one wood, one coal and one iron in the inventory.
Of the 4 hard achievements, collect diamond is objectively that hardest achievement to unlock as not only does it depends on the make iron pickaxe , but also that the diamond resource is extremely scarce in the game.
The complete achievement list can be found in App.
B.1.
5. A CHIEVEMENT LEARNING
[Table]
We first investigate the clustering accuracy of our algorithm.
As an evaluation metric, we calculate both cluster purity, which is defined as the proportion of the majority achievement in each cluster, and achievement purity, which is defined as the proportion of the majority cluster that the achievement is clustered into.
The mean purity across all clusters and achievements is regarded as the final clustering accuracy.
Baseline and result.
We design a latent space-based baseline for achievement learning.
Specifically, we train a neural network to predict the environmental reward and use the latent variable of the network as the achievement representation.
The idea behind this baseline is that to predict a certain achievement being completed, only the necessary information in the observation will be [Figure] preserved in the predictor network.
To test the effectiveness of our determinant-based loss function, we add an ablation experiment that replaces the determinant in Eqn.
1 with mean pairwise distances.
The clustering accuracy can be found in Tab. 1.
The latent-space baseline only manages to provide meaningful clustering in two of the easiest environments, while our algorithm does almost-perfect clustering in all tested environments.
The mean distance variant shows slightly worse clustering performance than SEA in the Crafter environment but completely fails in the TreeMaze-Hard environment, which has the largest amount of achievements.
We note here that even though the determinant loss massively outperforms the latent-space baseline in our tested environments, it has the advantage of not dependent on the achievement reward assumption.
In our preliminary testing, we find out that the latent-space baseline separates different sources of reward better in some environments where the achievement reward assumption does not hold.
Sample efficient.
We further investigate the sample efficiency of our algorithm.
We run three sets of experiments with 1 million, 200 thousand, 50 thousand environment steps of data, respectively.
The clustering accuracy in Crafter is shown in Fig.
Numerically, 1 million steps of data is suffice as it reaches > 99% accuracy in all clusters and achievements, while only using 200 thousand steps also reaches > 90% accuracy in 16 clusters/achievements out of all 17 ones.
5. S TRUCTURE RECOVERY
[Figure] graph In all three TreeMaze environments and Crafter-Limited , our algorithm successfully recovers the exact dependency graphs.
In full Crafter , the graph Hamming distance (defined as the number of different entries in the adjacency matrix) between our recovered graph and the ground truth graph is 7 out of 272 entries.
The detailed graph comparison can be found in Fig.
5. S UB -P OLICY L EARNING AND E XPLORATION
Meta-controller comparison.
We first show the effectiveness of the dependency graphbased meta-controller in training the sub-policies.
The mean easy achievement set unlocking rate compared to a random controller is shown in Fig.
As shown in the figure, a graph-based metacontroller learns the sub-policies much faster than a random controller.
Sub-policy learning and exploration.
For this part, we choose IMPALA (Espeholt et al., 2018) and PPO (Schulman et al., 2017) as widely-recognized model-free baselines, IMPALA + RND (Burda et al., 2019b) as an exploration baseline, DreamerV2 (Hafner et al., 2021) as the modelbased baseline since it achieves the highest reward in the original Crafter paper (Hafner, 2021), and HAL 3 (Costales et al., 2022) as a Hierarchical reinforcement learning baseline.
To evaluate SEA’s ability to master existing tasks and explore new tasks separately, we compare the mean achievement unlock rate 4 in the easy achievement set and the hard achievement set respectively, as shown in Fig.
From Fig.
4a, we notice that SEA learns slightly slower than vanilla IMPALA in the easy set.
This is due to the fact that SEA is not designed to maximize the total reward but to master each individual achievement-targeted sub-policy, as our meta-controller does not try to learn a optimal achievement completion route.
From Fig.
4a, we can see that SEA successfully discover the hard achievements with a significant reaching rate while vanilla IMPALA completely fails to reach any of them, which shows the exploration potential of our algorithm.
The complete unlocking rate curve can be found in Fig.
8 in the Appendix.
A detailed numerical comparison with all baselines is shown in Table 2.
We observe that SEA is the only algorithm that can consistently reach the hard set achievements and at the same time maintains close-to-optimum easy set achievements reaching rate.
SEA is also able to reach the hardest achievement in Crafter , collect diamond , with a non-negligible probability.
[Table]
Sample efficiency.
For SEA, we use 200 million environment steps to train the original IMPALA policy for the initial data collecting and 300 million steps for the sub-policy learning and exploration.
For both vanilla IMPALA and IMPALA + RND, we train the agent for 1 billion steps.
For PPO, we only manage to run the agent for 110 million steps due to time limit.
For DreamerV2, we run the agent for 2.5 million steps.
For HAL, we run the agent for 140 million steps.
We also note that for PPO, DreamerV2 and HAL, even though the environment steps does not match SEA, their training wall clock times are all longer than the 1 billion step IMPALA experiment and we don’t observe significant improvement for at least the last 1/3 of the training.
We refer the reader to Fig.
8 in the Appendix.
C ONCLUSION AND D ISCUSSION
Exploration in procedurally generated environments is particularly hard since state frequency-based exploration methods are often not effective in such domains.
We set our sight on achievementbased environments and propose Structured Exploration with Achievements (SEA), a multi-stage algorithm specialized in exploring in such domains.
Empirically, SEA successfully recovers the underlying structure of Crafter , a procedurally generated achievement-based environment and use the recovered structure to complete the hard exploration tasks in Crafter .
We notice that achievementbased environments are not common in the reinforcement learning community.
However, since achievement-based environments are easy to design and can help to learn hard exploration tasks even not with a particularly good choice of achievements, we call on the community to consider this design choice more often when developing new environments.
R EPRODUCIBILITY S TATEMENT
We provide the source code in the supplemental materials.
The code is also downloadable from here.
For detailed implementations and hyperparameters, we refer the readers to Appendix C.
A A DDITIONAL E XPERIMENT R ESULTS A. C RAFTER INDIVIDUAL ACHIEVEMENT LEARNING CURVE
[Figure]
[Figure]
B E NVIRONMENT D ETAILS B. C OMPLETE C RAFTER ACHIEVEMENT LIST
The complete list of Crafter achievements are { collect coal , collect diamond , collect drink , collect iron , collect sapling , collect stone , collect wood , defeat skeleton , defeat zombie , eat cow , make iron pickaxe , make iron sword , make stone pickaxe , make stone sword , make wood pickaxe , make wood sword , place furnace , place plant , place stone , place table , wake up } .
Of those the 17 achievements { collect coal , collect drink , collect iron , collect sapling , collect stone , collect wood , defeat skeleton , eat cow , eat plant , make stone pickaxe , make stone sword , make wood pickaxe , make wood sword , place furnace , place plant , place stone , place table , wake up } are the easy set since they are solvable by the vanilla IMPALA agent.
The remaining 4 tasks { collect diamond , defeat zombie , make iron pickaxe , make iron sword } are the hard set.
B. M ODIFICATIONS TO THE ORIGINAL C RAFTER ENVIRONMENT
We make some modifications to the original Crafter environment to make it better suits our setting.
The complete change list is:
• The health reward is removed since it does not fit into our achievement reward assumption.
Consequently, the character won’t lose health point in any way.
• The game ends if no achievement has been unlocked in the last 100 steps.
• When a creature (including a cow, a zombie and a skeleton) gets attacked by the character, it will not lose health point unless the attacking point of the character is greater than or equal to the health point of that creature.
With this modification, the eat cow , defeat skeleton and defeat zombie achievements can be connected to the main dependency graph instead of being independent nodes.
C R EPRODUCEBILITY C. A DDITIONAL ALGORITHM DETAILS C.1. M ETA - CONTROLLER DETAILS
Let the dependency graph be G . The meta-controller maintains a set of the completed achievements S that initialized to ∅ .
At the beginning of an episode and whenever an achievement has been unlocked, the meta-controller (re)generates a new target achievement.
To generate a new target achievement, if it is the beginning of an episode, the meta-controller randomly selects an achievement with no dependencies according to the dependency graph; Otherwise, suppose the just completed achievement is u .
Let V = { v / ∈ S | w ∈ S, ∀ w → v ∈ G } be the available achievements set.
Let CONT ( u ) = { v ∈ V | u → v ∈ G } be the subset of V that are direct descendants of u .
Let NGHB ( u ) = { v ∈ V | ∃ w ∈ V s.t. v → w ∈ G } be the subset that are neighbours of u (i.e. on which one or more u ’s direct descendants are depended).
Let
JUMP ( u ) = { v ∈ V | v / ∈ CONT ( u ) ∧ v / ∈ NGHB ( u ) } be the remaining achievements.
Notice these three subsets partitions V .
The meta-controller stochastically chooses the target achievement according to the following probabilities:
• With probability 40%, the meta-controller chooses the exploration task; • With probability 48%, the meta-controller uniformly randomly chooses an achievement from CONT ( u ) ; • With probability 6%, the meta-controller uniformly randomly chooses an achievement from NGHB ( u ) ; • With probability 6%, the meta-controller uniformly randomly chooses an achievement from JUMP ( u ) .
After selecting the next target, the meta-controller updates the completed achievements set S ←
S ∪ { u } .
C. I MPLEMENTATION
Our code is adapted from an open-source implementation of IMPALA, torchbeast 5 .
The complete code can be downloaded here 6 .
C. H YPERPARAMETERS
[Table]
C. I MPLEMENTATION OF A CHIEVEMENT R EPRESENTATION F UNCTION ϕ
ϕ uses the two networks to encode the states, one for the current state s t , one for the next state s t +1 .
The network structure is the same as the policy network.
The encoding of the two states are concatenated into the observation vector x obs .
x obs is then concatenated with the one-hot representation of the current action a t .
The concatenated vector is then fed into a fully-connected layer, resulting in x rep .
The final output of ϕ is x obs + x rep .
D A DDITIONAL D ISCUSSIONS D. C OMPARISON WITH HRL
Despite sharing similar concepts (achievements and subgoals/subtasks) with HRL, our work has a different focus.
Subgoals and subtasks in HRL normally refer to the intermediate steps that lead to an ultimate goal/task.
Achievements are all equal in nature, although some achievements can be the subgoals to other achievements.
Our algorithm can automatically determine which achievements can be useful (potentially as subgoals) to other achievements.
This lowers the standard for the achievement design as there is no need to carefully craft a series of intermediate steps that help the agent to learn.
Our achievement setting also allows for multiple ultimate tasks that our algorithm can solve for in one go.