A LIGN -RUDDER: L EARNING F ROM F EW D EMONSTRATIONS BY R EWARD R EDISTRIBUTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Reinforcement Learning algorithms require a large number of samples to solve complex tasks with sparse and delayed rewards.
Complex tasks are often hierarchically composed of sub-tasks.
Solving a sub-task increases the return expectation and leads to a step in the Q -function.
RUDDER identiﬁes these steps and then redistributes reward to them, thus immediately giving reward if sub-tasks are solved.
Since the delay of rewards is reduced, learning is considerably sped up.
However, for complex tasks, current exploration strategies struggle with discovering episodes with high rewards.
Therefore, we assume that episodes with high rewards are given as demonstrations and do not have to be discovered by exploration.
Unfortunately, the number of demonstrations is typically small and RUDDER’s
LSTM as a deep learning model does not learn well on these few training samples.
Hence, we introduce Align-RUDDER, which is RUDDER with two major modiﬁcations.
First,
Align-RUDDER assumes that episodes with high rewards are given as demonstrations, replacing RUDDER’s safe exploration and lessons replay buffer.
Second, we substitute RUDDER’s LSTM model by a proﬁle model that is obtained from multiple sequence alignment of demonstrations.
Proﬁle models can be constructed from as few as two demonstrations.
Align-RUDDER uses reward redistribution to speed up learning by reducing the delay of rewards.
Align-RUDDER outperforms competitors on complex artiﬁcial tasks with delayed rewards and few demonstrations.
On the MineCraft ObtainDiamond task, Align-RUDDER is able to mine a diamond, though not frequently.
I NTRODUCTION
Reinforcement learning algorithms struggle with learning complex tasks that have sparse and delayed rewards (Sutton & Barto, 2018; Rahmandad et al., 2009; Luoma et al., 2017).
For delayed rewards, temporal difference (TD) suffers from vanishing information (Arjona-Medina et al., 2019).
Monte
Carlo (MC) on the other hand has high variance, since it must average over all possible futures (Arjona-Medina et al., 2019).
Monte-Carlo Tree Search (MCTS) used for Go and chess can handle delayed and rare rewards since it has a perfect model of the environment (Silver et al., 2016; 2017).
RUDDER (Arjona-Medina et al., 2019; 2018) has shown to excel in model-free learning of policies when only sparse and delayed rewards are given.
RUDDER requires episodes with high rewards, to store them in its lessons replay buffer for learning a reward redistribution model like an LSTM network.
However, for complex tasks current exploration strategies ﬁnd episodes with high rewards only after an incommensurate long time.
Humans and animals obtain high reward episodes by teachers, role models, or prototypes.
Along this line, we assume that episodes with high rewards are given as demonstrations.
Since generating demonstrations is often tedious for humans and time-consuming for exploration strategies, typically only few demonstrations are available.
However,
RUDDER’s LSTM (Hochreiter, 1991; Hochreiter & Schmidhuber, 1997a) as a deep learning method requires many examples for learning.
Therefore, we introduce Align-RUDDER, which replaces
RUDDER’s LSTM by a proﬁle model obtained from multiple sequence alignment (MSA) of the demonstrations.
Proﬁle models are well known in bioinformatics, where they are used to score new sequences according to their sequence similarity to the aligned sequences.
Like RUDDER also Align-RUDDER performs reward redistribution —using an alignment model—, which considerably speeds up learning even if only few demonstrations are available.
[Figure]
Our main contributions are:
• We suggest a reinforcement algorithm that works well for sparse and delayed rewards, where standard exploration fails but few demonstrations with high rewards are available.
• We adopt multiple sequence alignment from bioinformatics to construct a reward redistribution technique that works with few demonstrations.
• We propose a method that uses alignment techniques and reward redistribution for identifying sub-goals and sub-tasks which in turn allow for hierarchical reinforcement learning.
R EVIEW OF RUDDER
Basic insight: Q -functions (action-value functions) are step functions.
Complex tasks are often hierarchically composed of sub-tasks (see Fig. 1).
Hence, the Q -function of an optimal policy resembles a step function.
A step is a change in return expectation, that is, the expected amount of the return or the probability to obtain the return changes.
Steps indicate achievements, failures, accomplishing sub-tasks, or changes of the environment.
Identifying large steps in the Q -function speeds up learning, since it allows (i) to increase the return and (ii) to sample more relevant episodes.
As these steps correspond to sub-tasks, identifying them also enables the construction of a hierarchical reinforcement method that solves sub-tasks sequentially.
A Q -function predicts the expected return from every state-action pair (see Fig. 1) and is prone to make a prediction error that hampers learning.
Since the Q -function is mostly constant, it is not necessary to predict the expected return for every state-action pair.
It is sufﬁcient to identify relevant state-actions across the whole episode and use them for predicting the expected return (see Fig. 1).
The LSTM network (Hochreiter, 1991; Hochreiter & Schmidhuber, 1995; 1997a;b) can store the relevant state-actions in its memory cells.
Subsequently, LSTM only updates its states and changes its return prediction, when a new relevant state-action pair is observed.
Thus, the LSTM return prediction is constant at most time points and does not have to be learned.
The basic insight that
Q -functions are step functions is the motivation for identifying these steps via return decomposition and speeding up learning via reward redistribution.
Reward Redistribution: Idea and return decomposition.
We consider reward redistributions that are obtained by return decomposition in episodic Markov decision processes (MDP).
The Q -function is assumed to be a step function (blue curve, row 1 of Fig.
1, right panel).
Return decomposition identiﬁes the steps of the Q -function (green arrows in Fig.
1, right panel).
A function (LSTM in
RUDDER, alignment model in Align-RUDDER) predicts the expected return (red arrow, row 1 of
Fig.
1, right panel) given the state-action sub-sequence.
The prediction is decomposed into single steps of the Q -function (green arrows in Fig. 1). The redistributed rewards (small red arrows in [Figure]
right panel where events are mapped to the leter code for amino acids.
Sequence alignment makes those conserved regions visible and enables redistribution of reward to important events. second and third row of right panel of Fig. 1) remove the steps.
Thus, the expected future reward is equal to zero (blue curve at zero in last row in right panel of Fig. 1).
Future rewards of zero means that learning the Q -values simpliﬁes to estimating the expected immediate rewards (small red arrows in right panel of Fig. 1), since delayed rewards are no longer present.
Learning methods based on reward redistribution.
The redistributed reward serves as reward for a subsequent learning method: (A) The Q -values can be directly estimated (Arjona-Medina et al., 2019), which is used in the experiments for the artiﬁcial tasks and BC pre-training for MineCraft.
(B)
Redistributed rewards can serve for learning with policy gradients like Proximal Policy Optimization (PPO)
(Schulman et al., 2018), which is used in the MineCraft experiments.
(C)
Redistributed rewards can serve for temporal difference learning like Q -learning Watkins, 1989).
Limitations of LSTM models for reward redistribution.
RUDDER uses an LSTM model for reward redistribution via return decomposition.
The reward redistribution is the difference of two subsequent predictions of the LSTM model.
If a state-action pair increases the prediction of the return, then it is immediately rewarded.
Using state-action sub-sequences ( s, a ) 0:
t = ( s 0 , a 0 , . . .
, s t , a t ) , the redistributed reward is R t
+1 =
g (( s, a ) 0: t ) − g (( s, a ) 0:
t − 1 ) , where g is the return decomposition function, which is an LSTM model that predicts the return of the episode.
The LSTM model ﬁrst learns to approximate the largest steps of the Q -function, since they reduce the prediction error the most.
Therefore the LSTM model extracts ﬁrst the relevant state-actions pairs (events).
However, RUDDER’s LSTM is severely limited when very few demonstrations are available. LSTM is a deep learning method and is known to require a large number of samples to generalise to new sequences. Another limitation is due to the problem setting of learning from expert demonstrations. The dataset in this setting is not balanced as only successful demonstrations are available.
As a result, it is difﬁcult for the LSTM to extract key events from very few sequences.
A LIGN -RUDDER: RUDDER WITH F EW D EMONSTRATIONS
We make use of techniques from sequence alignment to replace the LSTM model by a proﬁle model for return decomposition.
The proﬁle model is the result of a multiple sequence alignment of the demonstrations and allows aligning new sequences to it.
Both sub-sequences ( s, a ) 0: t − 1 and ( s, a ) 0: t are mapped to sequences of events and then are aligned to the proﬁle model.
Thus, both sequences receive an alignment score S , which is proportional to the function g .
Similar to the
LSTM model, we identify the largest steps in the Q -function via relevant events determined by the proﬁle model.
The redistributed reward is R t
+1 =
g (( s, a ) 0: t ) − g (( s, a ) 0:
t − 1 ) (see Eq.
Therefore, redistributing the reward by sequence alignment ﬁts into the RUDDER framework with all its theoretical guarantees.
RUDDER is valid and works if its LSTM is replaced by other recurrent networks, attention mechanisms, or, as in our case, sequence and proﬁle models. Advantages of sequence alignment compared to LSTM.
While LSTM models require a large amount of training data, sequence alignment can identify key events with as few as two examples. Another advantage of sequence alignment is its ability to perform well even when the dataset is imbalanced. For example, when only expert demonstrations are available an LSTM model for reward redistribution is difﬁcult or even impossible to train as both positive and negative examples III) Multiple Sequence Alignment dddddd
I) Deﬁning Events II) Scoring Matrix IV) PSSM and Proﬁle
V) Reward Redistribution ddτ t −τ t
ProﬁleModel
S ( τ t )
S ( τ t − 1 )
R t +1 =
( S ( τ t ) − S ( τ t − 1 ))
C are required.
In contrast, sequence alignment can extract the underlying strategy from expert demonstrations without the need for examples of negative sequences.
Reward Redistribution by Sequence Alignment.
In bioinformatics, sequence alignment identiﬁes similarities between biological sequences to determine their evolutionary relationship (Needleman & Wunsch, 1970; Smith & Waterman, 1981).
Align-RUDDER uses such alignment techniques to align two or more high return demonstrations.
We assume that the demonstrations follow the same underlying strategy, therefore they are similar to each other and can be aligned.
The alignment provides a proﬁle model (the strategy) given as a consensus sequence, a frequency matrix, or a
Position-Speciﬁc Scoring Matrix (PSSM) (Stormo et al., 1982).
The redistributed reward of a new sequence is the difference of the scores of consecutive sub-sequences when aligned to the proﬁle model, where the alignment score is the return-decomposition function g .
This difference indicates how much of the return is gained or lost by a sequence element, which is the main idea of return decomposition: g (( s, a ) 0: t ) − g (( s, a ) 0: t − 1 ) .
Align-RUDDER can be cast as following the underlying strategy, which has been extracted by the proﬁle model.
The new reward redistribution approach consists of ﬁve steps, see Fig.
3: (I) Deﬁne events to turn episodes of state-action sequences into sequences of events.
(II) Determine an alignment scoring scheme, so that relevant events are aligned to each other.
(III) Perform a multiple sequence alignment (MSA) of the demonstrations.
(IV) Compute the proﬁle model and the PSSM.
(V) Redistribute the reward: Each sub-sequence τ t of a new episode τ is aligned to the proﬁle.
The redistributed reward R t +1 is proportional to the difference of scores S based on the PSSM given in step (IV), i.e. R t +1 ∝ S ( τ t ) − S ( τ t − 1 ) .
(I) Deﬁning Events.
We consider differences of consecutive states, to detect a change caused by an important event. An event is deﬁned as a cluster of similar differences, since an event leads to similar changes in states like when achieving a sub-goal.
If states do not have a proper representation, we suggest to use the “successor representation” (Dayan, 1993) or “successor features” (Barreto et al., 2017).
To construct the successor representation, we use the demonstrations combined with state-action sequences generated by a random policy. The successor representation allows to deﬁne the similarity between two states, e.g. via the negative Euclidean distance or the cosine similarity.
To deﬁne events via clusters, we use similarity-based clustering like afﬁnity propagation (AP) (Frey & Dueck, 2007).
For AP the similarity does not have to be symmetric and the number of clusters does not have to be deﬁned a priori.
A sequence of events is obtained from a state-action sequence by mapping states s to their cluster identiﬁer e (the event) and ignoring the actions.
Alignment techniques from bioinformatics assume sequences composed of few events, e.g. 20 events.
If there are too many events, then for two demonstrations only few events can be matched and, in turn, good ﬁtting alignments cannot be distinguished from random alignments.
This effect is known in bioinformatics as “Inconsistency of Maximum Parsimony” (Felsenstein, 1978).
(II) Determining the Alignment Scoring System.
A scoring matrix S has entries s i,j that give the score for aligning event i with j .
[-The score S MSA of a MSA is the sum of all pairwise scores: S MSA = (cid:80) i,j,i< j (cid:80) Lt =
 s x i,t ,x j,t , where x i,t means-]A priori, we only know that a relevant event should be aligned to itself but not to other events.
Therefore we set s i,j to large values for every i
=
j and to low values for i (cid:54) =
j .
Concretely, we set s i,j = 1 /p
i for i =
j and s i,j = α for i (cid:54) =
j .
Here, p i is the probability of event i being observed in the demonstrations and α is a hyper-parameter, which is typically a small negative number. This scoring scheme encourages alignment of rare events, since for them p i is small.
Existing prior knowledge can be incorporated into s i,j , too.
For more details see Appendix Sec. A.3.
(III) Multiple sequence alignment (MSA).
An MSA algorithm maximizes the sum of all {+pairwise scores S MSA = (cid:80) i,j,i< j (cid:80) Lt =
 s i,j,t i ,t j ,t ,+} with s i,j,t i ,t j ,t as the score of aligning at alignment position t the event at position t i in sequence i to event at position t j in sequences j .
L ≥ T is the alignment length, since gaps make the alignment longer than the length of each sequence.
For MSA, ClustalW (Thompson et al., 1994) is very popular.
MSA ﬁrst constructs a guiding tree by agglomerative hierarchical clustering of pairwise alignments between all demonstrations.
This guiding tree allows to identify multiple strategies.
For more details see Appendix Sec. A.3.
(IV) Position-Speciﬁc Scoring Matrix (PSSM) and MSA proﬁle model.
From the alignment, we construct a) an MSA proﬁle model, with column-wise event probabilities, and b) a PSSM (Stormo et al., 1982), with column-wise scores to aligning new sequences to the proﬁle model.
More details are given in Appendix Sec. A.3.
(V) Reward Redistribution.
The reward redistribution is based on the proﬁle model.
A sequence τ = e 0: T ( e t is event at position t ) is aligned to the proﬁle, which gives the score S ( τ )
= (cid:80) {+Ll =
+}
s l,t .
Here, s l,t is the alignment score for event e t at position l in the alignment.
If τ t = e 0: t is the preﬁx sequence of τ of length t + 1 , then the reward redistribution R t +1 for 0 (cid:54) t (cid:54) T is
[Equation]
where C = E demo (cid:104) ˜ G 0 (cid:105) / E demo (cid:104)(cid:80)
Tt =0 S ( τ t ) − S ( τ t − 1 ) (cid:105) and ˜ G
= (cid:80) Tt =
˜ R t +1 is the original return of the sequence τ and S ( τ t − 1 ) = 0 .
E demo is the expectation of the return over demonstrations, and C scales R t +1 to the range of ˜ G 0 .
R T +2 is the correction of the redistributed reward (ArjonaMedina et al., 2019), with zero expectation for demonstrations: E demo [ R T +2 ]
= 0 .
Since τ t = e 0: t and e t = f
( s t , a t ) , we can set g (( s, a ) 0: t ) =
S ( τ t ) C .
We ensure strict return equivalence by
G 0 = (cid:80) T +1 t =
R t +1 =
˜ G 0 .
The redistributed reward depends only on the past: R t +1 =
h (( s, a ) 0: t ) .
Sub-tasks.
The reward redistribution identiﬁes sub-tasks as alignment positions with high redistributed reward.
It also determines the terminal states and assigns reward for solving the sub-tasks.
However, reward redistribution and Align-RUDDER cannot guarantee that the reward is Markov.
For redistributed Markov reward, options (Sutton et al., 1999), MAXQ (Dietterich, 2000), or recursive option composition (Silver & Ciosek, 2012) can be used for hierarchical reinforcement learning.
Higher Order Markov Redistribution. Align-RUDDER may lead to higher order Markov redistribution.
In the Appendix (see Corollary 1 and 2), we show that such higher order redistribution could lead to optimal reward redistribution, which ensures the same optimal policies (see Theorem 1).
E XPERIMENTS
Align-RUDDER is compared on three artiﬁcial tasks with sparse & delayed rewards and few demonstrations to Behavioral Cloning with Q -learning (BC+ Q ), Soft Q Imitation Learning (SQIL)
(Reddy et al., 2020), RUDDER (LSTM), and Deep Q -learning from Demonstrations (DQfD) (Hester et al., 2018).
GAIL (Ho & Ermon, 2016) failed to solve the two artiﬁcial tasks, as reported previously for similar tasks (Reddy et al., 2020).
Then, we test Align-RUDDER on the complex MineCraft
ObtainDiamond task with episodic rewards (Guss et al., 2019b).
All experiments use ﬁnite time
MDPs with γ = 1 and episodic reward.
More details are given in Appendix Sec.
A.6.
[Figure] Thresholding the Alignment vs LSTM in 1D key-chest environment. We use a 1D key-chest environment to show the effectiveness of sequence alignment in a low data regime compared to an LSTM model.
The agent has to collect the key and then open the chest, to get a positive reward at the last timestep.
See
Appendix Fig.
A.9 for a schematic representation of the environment.
As the key-events (important state-action pairs) in this environment are known we can compute the key-event detection rate of a reward redistribution model.
A key event is detected if the redistributed reward of an important state-action pair is larger than the average redistributed reward in the sequence.
We train the reward redistribution models with 2 , 5 and 10 training episodes and test on 1000 test episodes, averaged over 10 trials.
Align-RUDDER signiﬁcantly outperforms LSTM (RUDDER) for detecting these key events in all cases, with an average key-event detection rate of 0 .
96 for sequence alignment vs. 0 .for
the LSTM models over all dataset sizes.
See Appendix Fig.
A.10 for the detailed results.
Artiﬁcial tasks (I) and (II).
They are variations of the gridworld rooms example (Sutton et al., 1999), where cells are the MDP states.
In our setting, the states do not have to be time-aware for ensuring stationary optimal policies but the unobserved used-up time introduces a random effect.
The grid is divided into rooms.
The agent’s goal is to reach a target from an initial state with fewest steps.
It has to cross different rooms, which are connected by doors, except for the ﬁrst room, which is only connected to the second room by a teleportation portal .
The portal is introduced to avoid BC initialization alone solving the task.
It enforces that going to the portal entry cells is learned, when they are at positions not observed in demonstrations.
At every location, the agent can move up, down, left, right .
The state transitions are stochastic.
An episode ends after T = 200 time steps.
If the agent arrives at the target, then at the next step it goes into an absorbing state where it stays until T =without receiving further rewards.
Reward is only given at the end of the episode.
Demonstrations are generated by an optimal policy with 0.2 exploration rate.
The ﬁve steps of Align-RUDDER’s reward redistribution are: (1) Events are clusters of states obtained by Afﬁnity Propagation using as similarity the successor representation based on demonstrations.
The scoring matrix is obtained according to (II), using (cid:15) = 0 and setting all off-diagonal values of the scoring matrix to − 1 .
(3) ClustalW is used for the MSA of the demonstrations with zero gap penalties and no biological options.
The MSA supplies a proﬁle model and a PSSM as in (IV).
(5) Sequences generated by the agent are mapped to sequences of events according to (I).
Reward is redistributed via differences of proﬁle alignment scores of consecutive sub-sequences according to Eq.
(1) using the PSSM.
The reward redistribution determines sub-tasks like doors or portal arrival.
The sub-tasks partition the Q -table into sub-tables that represent a sub-agent.
However, we optimize a single Q -table in these experiments as in the tabular case, deﬁning sub-tasks has no effect on learning.
[Figure] p i s o d e s t [Figure] i m
[Figure] l r [Figure] t u r
[Figure] All compared methods learn a Q table and use (cid:15) greedy initialized behavioral cloning ( they visited in the an
Align-RUDDER learns the Q -table via RUDDER’s Q -value estimation (learning method (A) from
Sec.2).
For BC+ Q , RUDDER (LSTM), SQIL, and DQfD a Q -table is learned by Q -learning.
Hyperparameters are selected via grid search using the same amount of time for each method.
For different numbers of demonstrations, performance is measured by the number of episodes to achieve 80% of the average return of the demonstrations.
A Wilcoxon rank-sum test determines the signiﬁcance of performance differences between Align-RUDDER and the other methods.
Task (I) environment is a 12 × 12 gridworld with four rooms.
The target is in room #4 and the start is in room #1 with 20 portal entry locations.
The state contains the portal entry for each episode.
Fig.shows the number of episodes required for achieving 80% of the average reward of the demonstrations for different numbers of demonstrations.
Results are averaged over 100 trials.
Align-RUDDER signiﬁcantly outperforms all other methods, for (cid:54) 10 demonstrations ( p -values < 10 − 10 ).
Task (II) is a 12 × 24 gridworld with eight rooms: target in room #8, and start in room #1 with 20 portal entry locations.
Fig.
5 shows the results with settings as in Task (I).
Align-RUDDER signiﬁcantly outperforms all other methods, for (cid:54) 10 demonstrations ( p -values < 10 − 19 ).
We also conduct an ablation study to study performance of Align-RUDDER, while changing various parameters, like environment stochasticity (See Sec.
A.6.4) and number of clusters (See Sec.
A.6.5).
MineCraft.
We further test Align-RUDDER on MineCraft ObtainDiamond task from the
MineRL dataset (Guss et al., 2019b).
We do not use intermediate rewards given by achieving subgoals from the challenge, since Align-RUDDER is supposed to discover such sub-goals automatically via reward redistribution.
We only give reward for mining the diamond.
This requires resource gathering and tool building in a hierarchical way.
To the best of our knowledge, no pure learning method (sub-goals are also learned) has mined a diamond yet (Scheller et al., 2020).
The dataset contains demonstrations which are insufﬁcient to directly learn a single policy (117 demonstrations, 67 mined a diamond).
Implementation: (1) A state consists of a visual input and an inventory (incl.
equip state).
Both inputs are normalized to the same information, that is, the same number of components and the same variance.
We cluster the differences of consecutive states (Arjona-Medina et al., 2019).
Very large clusters are removed and small merged giving 19 clusters corresponding to events, which are characterized by inventory changes.
Finally, demonstrations are mapped to sequences of events.
The scoring matrix is computed according to (II).
(3) The 10 shortest demonstrations that obtained a diamond are aligned by ClustalW with zero gap penalties and no biological options.
The multiple alignment gives a proﬁle model and a PSSM.
(5) Reward is redistributed via differences of proﬁle alignment scores of consecutive sub-sequences according to Eq.
(1) using the PSSM.
Based on the reward redistribution we deﬁne sub-goals.
Sub-goals are identiﬁed as proﬁle model positions which obtain an average redistributed reward above a threshold for the demonstrations.
Demonstration sub-sequences between sub-goals are considered as demonstrations for the sub-tasks.
New sub-sequences generated by the agent are aligned to the proﬁle model to determine whether a sub-goal is achieved.
The redistributed reward between two sub-goals is given at the end of the sub-sequence, therefore, the sub-tasks have also episodic reward.
Fig.
4 shows how sub-goals are [Table] identiﬁed.
Sub-agents are pre-trained on the demonstrations for the sub-tasks using BC, and further trained in the environment using Proximal Policy Optimization (PPO) (Schulman et al., 2018).
BC pre-training corresponds to RUDDER’s Q -value estimation (learning method (A) from above), while
PPO corresponds to RUDDER’s PPO training (learning method (B) from above) [Table]
Our main agent can perform all actions but additionally can execute sub-agents and learns via the redistributed reward.
The main agent corresponds to and is treated like a Manager module (Vezhnevets et al., 2017).
The main agent is initialized by executing sub-agents according to the alignment but can deviate from this strategy.
When a sub-agent successfully completes its task, the main agent executes the next sub-agent according to the alignment.
More details can be found in Appendix
Sec. A.7.1.
Using only 10 demonstrations, Align-RUDDER is able to learn to mine a diamond.
A diamond is obtained in 0.1% of the cases.
With 0.5 success probability for each of the 31 extracted sub-tasks (skilled agents not random agents), the resulting success rate for mining the diamond would be 4 .
× 10 − 10 .
Tab. 1 shows a comparison of methods on the MineCraft MineRL dataset by the maximum item score (Milani et al., 2020).
Results are taken from (Milani et al., 2020), in particular from Figure 2, and completed by (Skrynnik et al., 2019; Kanervisto et al., 2020; Scheller et al., 2020).
Align-RUDDER was not evaluated during the MineCraft MineRL challenge, but it follows the timesteps limit (8 million) imposed by the challenge.
Align-RUDDER did not receive the intermediate rewards provided by the challenge that hint at sub-tasks, thus tries to solve a more difﬁcult task.
Recently, ForgER++ (Skrynnik et al., 2020) was able to mine a diamond in 0.0667% of the cases.
We do not include it in Table 1 as it did not have any limitations on the number of timesteps.
Also, ForgER++ generates sub-goals for MineCraft using a heuristic, while Align-RUDDER uses redistributed reward to automatically obtain sub-goals.
Analysis of MineCraft Agent Behaviour.
For each agent and its sub-task, we estimate the success rate and its improvement during ﬁne-tuning by averaging over return of multiple runs (see Fig. 6).
For earlier sub-tasks, the agent has a relatively higher sub-task success rate.
This also corresponds to the agent having access to much more data for earlier sub-tasks.
During learning from demonstrations, much less data is available for training for later sub-tasks, as not all expert demonstrations achieve the later tasks.
During online training using reinforcement learning, an agent has to successfully complete all earlier sub-tasks to generate trajectories for later sub-tasks.
This is exponentially difﬁcult. Lack of demonstrations and difﬁculty of the learned agent to generate data for later sub-tasks leads to degradation of the success rate in MineCraft.  R ELATED WORK Learning from demonstrations has been widely studied over the last 50 years (Billard et al., 2008).
An example is imitation learning, which uses supervised techniques when the number of demonstrations is large enough (Michie et al., 1990; Pomerleau, 1991;
Michie & Camacho, 1994; Schaal, 1996;
Kakade & Langford, 2002).
However, policies trained with imitation learning tend to drift away
[Figure]
P P P
P
P L V
N L A
A
A A
Y L Q
Q
Q F K
K
K E
D exact consensus pathRL fine tuninghuman demonstrationsBC consensus S S
[Figure]
from
(Ross
Bagnell, mitigated (III et al., 2009; Ross & Bagnell, 2010; Ross et al., 2011; Judah et al., 2014; Sun et al., 2017; 2018).
Many approaches use demonstrations for initialization, e.g. of policy networks (Taylor et al., 2011; Silver et al., 2016), value function networks (Hester et al., 2017; 2018), both networks (Zhang & Ma, 2018; Nair et al., 2018), or an experience replay buffer (Hosu & Rebedea, 2016).
Beyond initialization, demonstrations are used to deﬁne constraints (Kim et al., 2013), generate sub-goals (Eysenbach et al., 2019), enforce regularization (Reddy et al., 2020), guide exploration (Subramanian et al., 2016;
Jing et al., 2019), or shape rewards (Judah et al., 2014;
Brys et al., 2015; Suay et al., 2016).
Demonstrations may serve for inverse reinforcement learning (Ng & Russell, 2000; Abbeel & Ng, 2004; Ho & Ermon, 2016), which aims at learning a (non-sparse) reward function that best explains the demonstrations.
Learning reward functions requires a large number of demonstrations (Syed & Schapire, 2007;
Ziebart et al., 2008; Silva et al., 2019).
Some approaches rely on few-shot or/and meta learning (Duan et al., 2017; Finn et al., 2017; Zhou et al., 2020).
However, few-shot and meta learning demand a large set of auxiliary tasks or prerecorded data.
Concluding, most methods that learn from demonstrations rely on the availability of many demonstrations (Khardon, 1999; Lopes et al., 2009), in particular, if using deep learning methods (Bengio & Lecun, 2007; Lakshminarayanan et al., 2016).
Some methods can learn on few demonstrations like Soft Q Imitation Learning (SQIL) (Reddy et al., 2020), Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016), and
Deep Q -learning from Demonstrations (DQfD) (Hester et al., 2018).
D ISCUSSION AND C ONCLUSION
Discussion.
Firstly, reward redistributions do not change the optimal policies (see Theorem 1 in
Appendix).
Thus, suboptimal reward redistributions due to alignment errors or choosing events that are non-essential for reaching the goal might not speed up learning, but also do not change the optimal policies.
Secondly, while Align-RUDDER can speed up learning even in complex environments, the resulting performance depends on the quality of the alignment model.
A low quality alignment model can arise from multiple factors, one of which is having large number ( (cid:29) 20) of distinct events.
Clustering can be used to reduce the number of events, which could also lead to a low quality alignment model if too many relevant events are clustered together.
While the optimal policy is not changed by poor demonstration alignment, the beneﬁt of employing reward redistribution based on it diminishes.
Thirdly, the alignment could fail if the demonstrations have different underlying strategies
i.e no events are common in the demonstrations.
We assume that the demonstrations follow the same underlying strategy, therefore they are similar to each other and can be aligned.
However, if no underlying strategy exists, then identifying those relevant events via alignment, which should receive high redistributed rewards, may fail.
In this case, reward is given at sequence end, when the redistributed reward is corrected, which leads to an episodic reward without reducing the delay of the rewards and speeding up learning.
Conclusions.
We have introduced Align-RUDDER to solve highly complex tasks with delayed and sparse reward from few demonstrations.
We have shown experimentally that Align-RUDDER outperforms state of the art methods designed for learning from demonstrations in the regime of few demonstrations.
On the MineCraft ObtainDiamond task, Align-RUDDER is, to the best of our knowledge, the ﬁrst pure learning method to mine a diamond.
E THICS S TATEMENT
Impact on ML and related scientiﬁc ﬁelds.
Our research has the potential to positively impact a wide variety of ﬁelds of life due to its general applicability.
Most importantly, it has the potential to reduce the cost for training and deploying agents in real world applications and therefore enable systems that have not been possible until now.
However, any new development in machine learning can be applied for good or for bad.
Our system can be used for medical applications where it can save life but it could be used for malevolent systems.
It is the society that decides how new technology is employed.
However, we as scientist have to inform the society and the decision makers about our technologies.
We have to show the limits of our technology, to give ideas of possible applications, to point out possible misuse or erroneous operation of our new technology.
Impact on society.
A big danger is that users rely too much on our new approach and use it without reﬂecting on the outcomes.
For example, in medical treatment decisions doctors may rely on the technical system and push are the responsibility toward the machine: “The machine suggested this treatment, therefore it is not my fault”.
Another example is self-driving cars where we see that drivers become more careless even if they are supposed to pay attention and keep the hands on the steering wheel.
They trust too much in the technology, even if the technology does not justify this trust or is not mature.
Finally, our method can be deployed in companies for job automation.
Therefore there is the danger that some people lose their jobs, particularly those whose work is to perform predictable and repetitive tasks.
An often used example is the taxi driver who would lose their job because of self-driving cars.
The same holds for many jobs in production industry where automation can replace jobs.
However all industrialization led to loss of jobs but new jobs have been created.
Consequences of failures of the method.
Depending on the application area, a failure of this method might be of lesser concern, such as a failed execution of a computer program.
If our method is employed within a larger automation system, a failure can result in damages such as a car accident.
However, this holds for almost all reinforcement learning methods, and usage and testing falls within the responsibility of the application area.
We note that in this work, the method was only used in computer game environments.
Leveraging of biases in the data and potential discrimination.
Our proposed method relies on human demonstrations and thereby human decisions, which are usually strongly biased.
As almost all machine learning methods trained on human-inﬂuenced data, our method could learn to use and exploit those biases and make similar decisions (Solaiman et al., 2019).
Therefore, the responsible use of our method depends on a careful selection of the training data and awareness of the potential biases within those.
R EPRODUCIBILITY S TATEMENT
Code for experiments on the FourRooms and EightRooms environment is included as supplementary material.
The README contains step-by-step instructions to set up an environment and run the experiments.
We have speciﬁed all the training details ex. hyperparameters and how they were chosen in the Appendix (See Section A.6).
We trained 100 replicates for each datapoint of the ﬁrst set of experiments and are shown in Fig.
Using the code in the supplementary material, it is quite easy to reproduce our results for these experiments.
We also include code for the experiments done for MineCraft in the supplementary materials.
All the preprocessing steps, hyperparameters and other implementation details are given in the Appendix (See Section A.7).
We also provide a deeper overview of the RUDDER (Arjona-Medina et al., 2019) theory in the
Appendix (See Section A.2) as it is important for many design choices in Align-RUDDER.
Finally, a video showcasing the MineCraft agent is also provided as supplementary material.
PPENDIX C ONTENTS OF THE APPENDIX
A.1 Introduction to the Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Review Reward Redistribution . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3 The Five Steps of Align-RUDDER’s Reward Redistribution . . . . . . . . . . . . .
A.4 Sequence Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.5 Extended Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.6 Artiﬁcial Task Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.6.1 Hyperparameter Selection . . . . . . . . . . . . . . . . . . . . . . . . . .
A.6.2 Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.6.3 Artiﬁcial Task p-values . .
A.6.4 Stochastic Environments . . . . . . . . . . . . . . . . . . . . . . . . . . .
[-A.5.
-]{+A.6.
+}
Changing number of Clusters . . . . . . . . . . . . . . . . . . . . . . . .
A.6.6 Key-Event Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.7 Minecraft Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A. 7.1 MineCraft A.
2 Related Work and Steps Towards a General Agent . . . . . . . . . . . . .
A.7.
[Table]
Implementation of our Algorithm for Minecraft
A.7.
[Table]
and Value Network Architecture . . . . . . . . . . . . . . . . . . .
A.7.
[Table]
Learning of Sub- [Table] Agents . . .
A.7. [Table] Reinforcement Learning on Sub
-Task Agents . . . A.8 Reproducing the Artiﬁcial
Task Results . . . . . . . . . . . . . . . . . . . . . . .
A.9 Software Libraries . .
A.10 Compute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
L IST OF FIGURES
A.2 Clusters formed in the FourRooms and EightRooms environment . .
A.3 Clusters formed in the FourRooms and EightRooms environment . .
A.4 Clusters formed in the FourRooms and EightRooms environment . .
A.5 FourRooms and EightRooms environments . .
A.6 Reward redistribution for the FourRooms and EightRooms environments . . . . . .
A.11 Conceptual overview of our MineRL agent . . . . . . . . . . . . . . . . . . . . . .
A.12 Mapping of clusters to letters
A.14 Trajectory replay given by an exemplary consensus
A.15 Conceptual architecture of Align-RUDDER MineRL policy and value networks
A.16 Discretization and interpolation of camera angles
A. I NTRODUCTION TO THE A PPENDIX
This is the appendix to the paper “Align-RUDDER:
Learning from few Demonstrations by Reward
Redistribution”.
The appendix aims at supporting the main document and provides more detailed information about the implementation of our method for different tasks.
The content of this document is summarized as follows:
• Section A.3 describes the ﬁve steps of Align-RUDDER’s reward redistribution in more detail.
In particular, the scoring systems are described in more detail.
• Section
A.4 provides a brief overview of sequence alignment methods and the hyperparameters used in our experiments.
• Section A. provides ﬁgures and tables to support the results of the experiments in Artiﬁcial Tasks (I) and (II).
• Section A
explains in detail the experiments conducted in the Minecraft ObtainDiamond task.
A. R EVIEW R EWARD R EDISTRIBUTION
Reward redistribution and return decomposition are concepts introduced in RUDDER but also apply to
Align-RUDDER as it is a variant of RUDDER.
Reward redistribution based on return decomposition eliminates – or at least mitigates – delays of rewards while preserving the same optimal policies.
Align-RUDDER is justiﬁed by the theory of return decomposition and reward redistribution when using multiple sequence alignment for constructing a reward redistribution model.
In this section, we review the concepts of return decomposition and reward redistribution.
Preliminaries.
We consider a ﬁnite MDP deﬁned by the 5-tuple P =
( S , A , R , p, γ ) where the state space S and the action space
A are sets of ﬁnite states s and actions a and R the set of bounded rewards r .
For a given time step t , the corresponding random variables are S t , A t and R t +1 .
Furthermore, P has transition-reward distributions p ( S t
+1 =
s (cid:48) , R t +1 =
r | S t = s, A t = a ) , and a discount factor γ ∈ (0 , 1] , which we keep at γ = 1 .
A Markov policy π ( a | s ) is a probability of an action a
given a state s .
We consider MDPs with ﬁnite time horizon or with an absorbing state.
The discounted return of a sequence of length T at time t is G t = (cid:80) T − t k =
γ k R t + k +1 .
As usual, the Q -function for a given policy π is q π ( s, a ) =
E π
[ G t | S t = s, A t = a ] .
E π
[ x | s, a ] is the expectation of x , where the random variable is a sequence of states, actions, and rewards that is generated with transition-reward distribution p , policy π , and starting at ( s, a ) .
The goal is to ﬁnd an optimal policy π ∗ =
argmax π E π [ G 0 ] maximizing the expected return at t = 0 .
We assume that the states s are time-aware (time t can be extracted from each state) in order to assure stationary optimal policies.
According to Proposition 4.4.3 in (Puterman, 2005), a deterministic optimal policy π ∗ exists.
Deﬁnitions.
A sequence-Markov decision process (SDP) is deﬁned as a decision process that has
Markov transition probabilities but a reward probability that is not required to be Markov.
Two SDPs ˜ P and P with different reward probabilities are return-equivalent if they have the same expected return at t = 0 for each policy π , and strictly return-equivalent if they additionally have the same expected return for every episode.
Since for every π the expected return at t = 0 is the same, return-equivalent SDPs have the same optimal policies.
A reward redistribution is a procedure that —for a given sequence of a delayed reward SDP ˜ P — redistributes the realization or expectation of its return ˜ G 0 along the sequence.
This yields a new SDP P with R as random variable for the redistributed reward and the same optimal policies as ˜ P :
Theorem 1 (Arjona-Medina et al.
Both the SDP ˜ P with delayed reward ˜ R t +1 and the SDP
P with redistributed reward R t +1 have the same optimal policies.
Proof.
The proof can be found in (Arjona-Medina et al., 2019).
The delay of rewards is captured by the expected future rewards κ ( m, t − 1) at time ( t − 1) .
κ is deﬁned as κ ( m, t − 1) :
= E π [ (cid:80) mτ
=
R t +1+ τ | s t − 1 , a t − 1 ] , that is, at time ( t − 1) the expected sum of future rewards from R t +1 to R t +
+ m but not the immediate reward R t .
A reward redistribution is deﬁned to be optimal , if κ ( T − t − 1 , t ) = 0 for 0 (cid:54) t (cid:54)
T − 1 , which is equivalent to
E π
[ R t +1 | s t − 1 , a t − 1 , s t , a t ] =
˜ q π ( s t , a t ) − ˜ q π ( s t − 1 , a t − 1 ) :
Theorem 2 (Arjona-Medina et al.
We assume a delayed reward MDP ˜ P , with episodic reward.
A new SDP P is obtained by a second order Markov reward redistribution, which ensures that P is return-equivalent to ˜ P .
For a speciﬁc π , the following two statements are equivalent:
[Equation]
e
[Equation]
the reward redistribution is optimal
[Equation]
An optimal reward redistribution fulﬁlls for 1 (cid:54) t (cid:54) T and 0 (cid:54) m (cid:54) T − t : κ ( m, t − 1) = 0 .
Proof.
The proof can be found in (Arjona-Medina et al., 2019).
This theorem shows that an optimal reward redistribution relies on steps ˜ q π
[Equation]
[Equation]
− 1 ) of the Q -function.
Identifying the largest steps in the Q -function detects the largest rewards that have to be redistributed, which makes the largest progress towards obtaining an optimal reward redistribution.
Corollary 1 (Higher order Markov reward redistribution optimality conditions) .
We assume a delayed reward MDP ˜ P , with episodic reward.
A new SDP P is obtained by a higher order Markov reward redistribution.
The reward redistribution ensures that P is return-equivalent to ˜ P .
If for a speciﬁc π
[Equation]
holds, then the higher order reward redistribution R t +1 is optimal, that is, κ ( T − t − 1 , t ) = 0 .
Proof.
The proof is just PART (II) of the proof of Theorem 2 in (Arjona-Medina et al., 2019).
We repeat it here for completeness.
We assume that
[Equation] [Equation]
where we abbreviate the expected R t +1 by h t :
[Equation]
The expectations E π [ .
| s t − 1 , a t − 1 ] like E π (cid:104)
˜ R T +1 | s t − 1 , a t − 1 (cid:105) are expectations over all episodes that contain the state-action pair ( s t − 1 , a t − 1 ) at time t − 1 .
The expectations E π [ .
| s t − 1 , a t − 1 , s t , a t ] like E π (cid:104)
˜ R T +1 | s t − 1 , a t − 1 , s t , a t (cid:105) are expectations over all episodes that contain the state-action pairs ( s t − 1 , a t − 1 ) at time t − 1 and ( s t , a t ) at time t .
The Q -values are deﬁned as
[Equation]
which are expectations over all trajectories that contain ( s t , a t ) at time t .
Since ˜ P is Markov, for ˜ q π only the sufﬁx trajectories beginning at ( s t , a t ) enter the expectation.
The deﬁnition of κ ( m, t − 1) for 1 (cid:54) t (cid:54) T and 0 (cid:54) m (cid:54) T − t was κ ( m, t − 1) =
E π [ (cid:80) mτ
=
R t +1+ τ | s t − 1 , a t − 1 ] .
We have to proof κ ( T − t − 1 , t ) = 0 .
First, we consider m =
and 1 (cid:54) t (cid:54) T , therefore κ (0 , t − 1) =
E π
[ R t
[Equation]
| s t − 1 , a t − 1 ] .
Since the original MDP ˜ P has episodic reward, we have ˜ r
( s t − 1 , a t − 1 )
=
E (cid:104)
˜
R t | s t − 1 , a t − 1 (cid:105) = 0 for 1 (cid:54) t (cid:54) T .
Therefore, we obtain:
[Equation]
Using this equation we obtain for 1 (cid:54) t (cid:54) T :
[Equation]
Next, we consider the expectation of (cid:80) mτ =
R t +1+ τ for 1 (cid:54) t (cid:54) T and 1 (cid:54) m
(cid:54) T − t
[Equation]
for m > 0 )
[Equation]
=
[Equation]
We used that ˜ R t +1 = 0 for t < T .
For the particualr cases t = τ + 1 and m = T − t = T − τ − 1 we have
[Equation]
That is exactly what we wanted to proof.
Corollary 1 explicitly states that the optimality criterion ensures an optimal reward redistribution even if the reward redistribution is higher order Markov.
For Align-RUDDER we may obtain a higher order Markov reward redistribution due to the proﬁle alignment of the sub-sequences.
Corollary 2 (Higher order Markov reward redistribution optimality representation) .
We assume a delayed reward MDP ˜ P , with episodic reward and that a new SDP P is obtained by a higher order
Markov reward redistribution.
The reward redistribution ensures that P is strictly return-equivalent to ˜ P .
We assume that the reward redistribuition is optimal, that is, κ ( T − t − 1 , t ) = 0 .
If the condition
[Equation]
holds, then
[Equation]
Proof.
By and large, the proof is PART (I) of the proof of Theorem 2 in (Arjona-Medina et al., 2019).
We repeat it here for completeness.
We assume that the reward redistribution is optimal, that is,
[Equation]
We abbreviate the expected R t +1 by h t :
[Equation]
In (Arjona-Medina et al., 2019)
Lemma A4 is as follows.
Lemma 1.
Two strictly return-equivalent SDPs ˜ P and P have the same expected return for each start state-action sub-sequence ( s 0 , a 0 , . . .
, s t , a t ) , 0 (cid:54) t (cid:54) T
[Equation]
The assumptions of Lemma 1 hold for for the delayed reward MDP ˜ P and the redistributed reward
SDP P , since a reward redistribution ensures strictly return-equivalent SDPs.
Therefore for a given state-action sub-sequence ( s 0 , a 0 , . . .
, s t , a t ) , 0 (cid:54) t (cid:54)
[Equation]
[Equation]
[Equation]
(17) with G 0 = (cid:80) T τ =
R τ +
and ˜ G
= ˜ R T +1 .
The Markov property of the MDP ˜ P ensures that the future reward from t + 1 on is independent of the past sub-sequence s 0 , a 0 , . . .
, s t − 1 , a t − 1 :
E π (cid:34) T − t (cid:88) τ
= ˜
R t +1+ τ
| s t , a t (cid:35) =
E π (cid:34) T
[Equation]
According to Eq.
(12), the future reward from t + 2 on is independent of the past sub-sequence s 0 , a 0 , . . .
, s t − 1 , a t − 1 :
[Equation]
Using these properties we obtain
[Equation]
We used the optimality condition
[Equation]
It follows that
[Equation]
This is exactly what we wanted to proof.
This corollary shows that optimal reward redistributions can be expressed as difference of Q -values if
Eq.
(12) holds.
Eq.
(12) states that the past can be averaged out.
However, there may exist optimal reward redistributions for which Eq.
(12) does not hold.
If the reward redistribution is optimal, the Q -values of P are given by q π ( s t , a t ) =
˜ q π ( s t , a t ) − ψ π ( s t ) and therefore ˜ P and P have the same advantage function:
Theorem 3 (Arjona-Medina et al.
If the reward redistribution is optimal, then the Q -values of the SDP P are q π ( s t , a t )
=
r ( s t , a t ) and
[Equation]
The SDP P and the original MDP ˜ P have the same advantage function.
Proof.
The proof can be found in (Arjona-Medina et al., 2019).
III) Multiple Sequence Alignment dddddd
I) Deﬁning Events II) Scoring Matrix IV) PSSM and Proﬁle
V) Reward Redistribution ddτ t −τ t
ProﬁleModel
S ( τ t )
S ( τ t − 1 )
R t +1 =
( S ( τ t ) − S ( τ t − 1 ))
C
For an optimal reward redistribution only the expectation of the immediate reward r ( s t , a t ) =
E π
[ R t +1 | s t , a t ] must be estimated.
This considerably simpliﬁes learning.
Learning methods according to Arjona-Medina et al.
The redistributed reward serves as reward for a subsequent learning method, which can be Type A, B, and C as described in ArjonaMedina et al.
Type A methods estimate the Q -values.
They can be estimated directly according to Eq.
(23) assuming an optimal redistribution (Type A variant i).
Q -values can be corrected for a non-optimal reward redistribution by additionally estimating κ (Type A variant ii).
Q -value estimation can use eligibility traces (Type A variant iii).
Type B methods use the redistributed rewards for policy gradients like Proximal Policy Optimization (PPO)
Schulman et al.
Type
C methods use TD learning like Q -learning Watkins (1989), where immediate and future reward must be drawn together as typically done.
For all these learning methods, demonstrations can be used for initialization (e.g. experience replay buffer) or pre-training (e.g. policy network with behavioral cloning).
Recently, the convergence of RUDDER learning methods has been proven under commonly used assumptions (Holzleitner et al., 2020).
Non-optimal reward redistribution and Align-RUDDER.
According to Theorem 1, non-optimal reward redistributions do not change the optimal policies.
The value κ ( T − t − 1 , t ) measures the remaining delayed reward.
The smaller κ is, the faster is the learning process.
For Monte Carlo (MC) estimates, smaller κ reduces the variance of the future rewards, and, therefore the variance of the estimation.
For temporal difference (TD) estimates, smaller κ reduces the amount of information that has to ﬂow back.
Align-RUDDER dramatically reduces the amount of delayed rewards by identifying key events via multiple sequence alignment, to which reward is redistributed.
For an episodic MDP, a reward that is redistributed to time t reduces all κ ( m, τ ) with t (cid:54)
τ < T by the expectation of the reward.
Therefore, in most cases Align-RUDDER makes κ -values much smaller.
A. T HE F IVE S TEPS OF A LIGN -RUDDER’ S R EWARD R EDISTRIBUTION
The new reward redistribution approach consists of ﬁve steps, see Fig.
A.1: (I) Deﬁne events to turn episodes of state-action sequences into sequences of events.
(II) Determine an alignment scoring scheme, so that relevant events are aligned to each other.
(III) Perform a multiple sequence alignment (MSA) of the demonstrations.
(IV) Compute the proﬁle model and the PSSM.
(V) Redistribute the reward: Each sub-sequence τ t of a new episode τ is aligned to the proﬁle.
The redistributed reward R t +1 is proportional to the difference of scores S based on the PSSM given in step (IV), i.e.
R t +1 ∝ S ( τ t ) − S ( τ t − 1 ) .
(I) Deﬁning Events.
Alignment techniques assume that sequences consist of few symbols, e.g. about 20 symbols, the events.
It is crucial to keep the number of events small in order to increase the difference between a random alignment and an alignment of demonstrations.
If there are many events, then two demonstrations might have few events that can be matched, which cannot be well distinguished from random alignments.
This effect is known in bioinformatics as “Inconsistency of
Maximum Parsimony” (Felsenstein, 1978).
The events can be the original state-action pairs, clusters thereof, or other representations of state-action pairs, e.g. indicating changes of inventory, health, energy, skills etc.
In general, we deﬁne events as a cluster of states or state-actions.
A sequence of events is obtained from a state-action sequence by substituting states or state-actions by their cluster identiﬁer.
In order to cluster states, a similarity measure between them is required.
We suggest to use the “successor representation” (Dayan, 1993) of the states, which gives a similarity matrix based on how connected two states are given a policy.
Successor representation have been used before (Machado et al., 2017; Ramesh et al., 2019) to obtain important events, for option learning.
For computing the successor representation, we use the demonstrations combined with state-action sequences generated by a random policy.
For high dimensional state spaces “successor features” (Barreto et al., 2017) can be used.
We use similarity-based clustering methods like afﬁnity propagation (AP) (Frey & Dueck, 2007).
For AP the similarity matrix does not have to be symmetric and the number of clusters need not be known.
State action pairs ( s, a ) are mapped to events e .
(II) Determining the Alignment Scoring System.
Alignment algorithms distinguish similar sequences from dissimilar sequences using a scoring system.
A scoring matrix S has entries s i,j that give the score for aligning event i with j .
The MSA score S MSA of a multiple sequence alignment is the sum of all pairwise scores: S MSA =
(cid:80) i,j,i<
j (cid:80) Lt =
s x i,t ,x j,t , where x i,t means that event x
i,t is at position t for sequence
τ i =
e i, 0: T in the alignment, analog for x j,t and the sequence τ j = e j, 0: T , and L is the alignment length.
Note that L ≥ T and x i,t (cid:54) =
e i,t , since gaps are present in the alignment.
In the alignment, events should have the same probability of being aligned as they would have if we know the strategy and align demonstrations accordingly.
The theory of high scoring segments gives a scoring scheme with these alignment probabilities (Karlin & Altschul, 1990; Karlin et al., 1990; Altschul et al., 1990).
Event i is observed with probability p
i in the demonstrations, therefore a random alignment aligns event i with j with probability p
i
p
j .
An alignment algorithm maximizes the
MSA score S MSA and, thereby, aligns events i and j with probability q ij for demonstrations.
High values of q ij means that the MSA often aligns events i and j in the demonstrations using the scoring matrix S with entries s i,j .
According to Theorem 2 and Equation [3] in Karlin & Altschul (1990), asymptotically with the sequence length, we have s i,j = ln( q ij / ( p i p j ))
/λ ∗ , where λ ∗ is the unique positive root of (cid:80) n,ni =1 ,j =1 p i p
j exp( λ s i,j ) =
1 (Equation [4] in Karlin & Altschul (1990)).
We can now choose a desired probability q ij and then compute the scoring matrix S with entries s i,j .
High values of q ij should indicate relevant events for the strategy.
A priori, we only know that a relevant event should be aligned to itself, while we do not know which events are relevant.
Therefore we set q ij to large values for every i
=
j and to low values for i (cid:54) =
j .
Concretely, we set q ij = p i − (cid:15) for i = j and q ij = (cid:15) / ( n − 1) for i (cid:54) =
j , where n is the number of different possible events.
Events with smaller p i receive a higher score
s i,i when aligned to themselves since this self-match is less often observed when randomly matching events (
p i p i is the probability of a random self-match).
Any prior knowledge about events should be incorporated into q ij .
(III) Multiple sequence alignment (MSA).
MSA ﬁrst produces pairwise alignments between all demonstrations.
Then, a guiding tree (agglomerative hierarchical clustering) is produced via hierarchical clustering sequences, according to their pairwise alignment scores.
Demonstrations which follow the same strategy appear in the same cluster in the guiding tree.
Each cluster is aligned separately via MSA to address different strategies.
However, if there is not a cluster of demonstrations, then the alignment will fail.
MSA methods like ClustalW (Thompson et al., 1994) or MUSCLE (Edgar, 2004) can be used.
(IV) Position-Speciﬁc Scoring Matrix (PSSM) and Proﬁle.
From the ﬁnal alignment, we construct a) an MSA proﬁle (column-wise event frequencies q i,j ) and b) a PSSM (Stormo et al., 1982) which is used for aligning new sequences to the proﬁle of the MSA.
To compute the PSSM (column-wise scores s i,t ), we apply Theorem 2 and Equation [3] in Karlin & Altschul (1990).
Event i is observed with probability p
i in the data.
For each position t in the alignment, we compute q
i,t , which indicates the frequency of event i at position t .
The PSSM is s i,t
= ln
( q i,t /p i )
/λ ∗ t , where λ ∗ t is the single unique positive root of (cid:80) ni
=1 p i exp( λ s i,t ) =
1 (Equation [1] in Karlin & Altschul (1990)).
If we align a new sequence that follows the underlying strategy (a new demonstration) to the proﬁle model, we would see that event i is aligned to position t in the proﬁle with probability q i,t .
(V) Reward Redistribution.
The reward redistribution is based on the proﬁle model.
A sequence τ = e 0: T ( e t is event at position t ) is aligned to the proﬁle, which gives the score S ( τ )
= (cid:80)
Lt =0 s x t ,t .
Here, s i,t is the alignment score for event i and x t is the event of τ at position t in the alignment.
L is the proﬁle length, where L ≥ T and x t (cid:54) =
e t , because of gaps in the alignment.
If τ t = e 0: t is the preﬁx sequence of τ of length t + 1 , then the reward redistribution R t +1 for 0 (cid:54) t (cid:54) T is
R t +1 =
( S ( τ t ) − S ( τ t − 1 ))
C =
g
(( s, a ) 0: t ) − g (( s, a ) 0: t − 1 ) , R T +2 =
˜ G 0 −
T (cid:88) t =
R t +1 , (24) where
C =
E demo (cid:104) ˜ G 0 (cid:105) / E demo (cid:104)(cid:80)
Tt =0 S ( τ t ) − S ( τ t − 1 ) (cid:105) and ˜ G
= (cid:80) Tt =
˜ R t +1 is the original return of the sequence τ and S ( τ t − 1 ) = 0 .
E demo is the expectation over demonstrations, and C scales R t +1 to the range of ˜ G 0 .
R T +2 is the correction of the redistributed reward (Arjona-Medina et al., 2019), with zero expectation for demonstrations: E demo [ R T +2 ]
= 0 .
Since τ t = e 0: t and e t = f
( s t , a t ) , we can set g (( s, a ) 0: t ) =
S ( τ t ) C .
We ensure strict return equivalence, since G 0 = (cid:80) T + t =
R t +1 =
˜ G 0 .
The redistributed reward depends only on the past, that is,
R t +1 =
h (( s, a ) 0: t ) .
For computational efﬁciency, the alignment of τ t − 1 can be extended to one for τ t , like exact matches are extended to high-scoring sequence pairs with the BLAST algorithm
(Altschul et al., 1990; 1997).
Sub-tasks.
The reward redistribution identiﬁes sub-tasks, which are alignment positions with high redistributed reward.
It also determines the terminal states and automatically assigns reward for solving the sub-tasks.
However, reward redistribution and Align-RUDDER cannot guarantee that the reward is Markov.
For redistributed reward that is Markov, the option framework (Sutton et al., 1999), the MAXQ framework (Dietterich, 2000), or recursive composition of option models (Silver & Ciosek, 2012) can be used as subsequent approaches to hierarchical reinforcement learning.
A. S EQUENCE A LIGNMENT
In bioinformatics, sequence alignment identiﬁes regions of signiﬁcant similarity among different biological sequences to establish evolutionary relationships between those sequences.
In 1970,
Needleman and Wunsch proposed a global alignment method based on dynamic programming (Needleman & Wunsch, 1970).
This approach ensures the best possible alignment given a substitution matrix, such as PAM (Dayhoff, 1978) or BLOSUM(Henikoff & Henikoff, 1992), and other parameters to penalize gaps in the alignment.
The method of Needlemann and Wunsch is of O ( mn ) complexity both in memory and time, which could be prohibitive in long sequences like genomes.
An optimization of this method by Hirschberg (1975), reduces memory to O ( m + n ) , but still requires O ( mn ) time.
Later, Smith and Waterman developed a local alignment method for sequences (Smith & Waterman, 1981).
It is a variation of Needleman and Wunsch’s method, keeping the substitution matrix and the gap-scoring scheme but setting cells in the similarity matrix with negative scores to zero.
The complexity for this algorithm is of O ( n 2 M ) .
Osamu Gotoh published an optimization of this method, running in O ( mn ) runtime (Gotoh, 1982).
The main difference between both methods is the following:
• The global alignment method by Needleman and Wunsch aligns the sequences ﬁxing the ﬁrst and the last position of both sequences.
It attempts to align every symbol in the sequence, allowing some gaps, but the main purpose is to get a global alignment.
This is especially useful when the two sequences are highly similar.
For instance:
[Equation]
• As an alternative to global methods, the local method of Smith and Waterman aligns the sequences with a higher degree of freedom, allowing the alignment to start or end with gaps.
This is extremely useful when the two sequences are substantially dissimilar in general but suspected of having a highly related sub region.
ATCAAGGAGATCATCGCTGGACTGAGTGGCT----ACGTGGTATGT
ATC----CGATCATCGCTGG-CTGATCGACCTTCTACGT------- *** ************ **** * * ****
A.4.0.1 Multiple Sequence Alignment algorithms.
The sequence alignment algorithms by
Needleman and Wunsch and Smith and Waterman are limited to aligning two sequences.
The approaches for generalizing these algorithms to multiple sequences can be classiﬁed into four categories:
• Exact methods (Wang & Jiang, 1994).
• Progressive methods: ClustalW (Thompson et al., 1994), Clustal Omega (Sievers et al., 2014), T-Coffee (Notredame et al., 2000).
• Iterative and search algorithms: DIALIGN (Morgenstern, 2004), MultiAlign (Corpet, 1988).
• Local methods: eMOTIF (Mccammon & Wolynes, 1998), PROSITE (Bairoch & Bucher, 1994).
For more details, visit Sequence Comparison: Theory and methods (Chao & Zhang, 2009).
In our experiments, we use ClustalW from Biopython (Cock et al., 2009) with the following parameters:
clustalw2 -ALIGN
-CLUSTERING=UPGMA -NEGATIVE " \ "-INFILE={infile} -OUTFILE={outfile} " \ "-PWMATRIX={scores} -PWGAPOPEN=0 -PWGAPEXT=0 " \ "-MATRIX={scores} -GAPOPEN=
-GAPEXT=
-CASE
=UPPER " \ "-NOPGAP
-NOHGAP -MAXDIV=0 -ENDGAPS -NOVGAP " \ "-NEWTREE={outputtree} -TYPE
=PROTEIN -OUTPUT=
GDE
where the PWMATRIX and MATRIX are computed according to step (II) in Sec.
3 of the main paper.
A. E XTENDED R ELATED W ORK
Align-RUDDER allows to identify sub-goals and sub-tasks, therefore it is related to hierarchical reinforcement learning (HRL) approaches like the option framework (Sutton et al., 1999), the MAXQ framework (Dietterich, 2000), or the recursive composition of option models (Silver & Ciosek, 2012).
However, these methods do not address the problem of ﬁnding good options, good sub-goals, or good sub-tasks.
Methods to learn good options have been proposed.
Frequently observed states in solutions are chosen as targets (Stolle & Precup, 2002).
Gradient-based approaches improving the termination function for options (Comanici & Precup, 2010; Mankowitz et al., 2016).
Policy gradient optimized a uniﬁed policy consisting of intra-option policies, option termination conditions, and an option selection policy (Levy & Shimkin, 2012).
Parametrized options are learned by treating the termination functions as hidden variables and using expectation maximization (Daniel et al., 2016).
Intrinsic rewards are used to learn the policies within options, and extrinsic rewards to learn the policy over options (Kulkarni et al., 2016).
Options have been jointly learned with an associated policy using the policy gradient theorem for options (Bacon et al., 2017).
A slow time-scale manager module learns sub-goals that are achieved by fast time-scale worker (Vezhnevets et al., 2017).
Next, we relate Align-RUDDER to imitation learning and trajectory matching.
Imitation learning aims at learning a behavior close to the data generating policy by matching the trajectories of single demonstrations.
In contrast, Align-RUDDER does not try to match single trajectories but identiﬁes relevant events that are shared among successful demonstrations.
In complex tasks like MineCraft trajectory matching fails, since large state spaces do not allow to match one of the few demonstrations.
However, relevant events can still be matched as they appear in most demonstrations, therefore
Align-RUDDER excels in such complex tasks.
A. A RTIFICIAL T ASK E XPERIMENTS
This section provides additional information that supports the results reported in the main paper for
Artiﬁcial Tasks (I) and (II).
A.6. H YPERPARAMETER S ELECTION
For (BC)+ Q -Learning and Align-RUDDER, we performed a grid search to select the learning rate from the following values [0 . 1 , 0 . 05 , 0 . 01] .
We used 20 different seeds for each value and each number of demonstrations and then selected the setting with the highest success for all number of demonstrations.
The ﬁnal learning rate for (BC)+ Q -Learning and DQfD is 0 .
01 and for AlignRUDDER it is 0 .
For DQfD, we set the experience buffer size to 30 , 000 and the number of experiences sampled at every timestep to 10 .
The DQfD loss weights are set to 0 .
01 , 0 . 01 and 1 . 0 for the Q -learning loss term, n -step loss term and the expert loss respectively during pre-training.
During online learning, we change the loss terms to 1 . 0 , 1 . 0 and 0 .
01 for the Q -learning loss term, n -step loss term and the expert loss term.
This was necessary to enable faster learning for DQfD.
The expert action margin is 0 . 8 .
For successor representation, we use a learning rate of 0 . 1 and a gamma of 0 . 99 .
We update the successor table multiple times using the same transition (state, action, next state) from the demonstration.
For afﬁnity propagation, we use a damping factor of 0 . 5 and set the maximum number of iterations to 1000 .
Furthermore, if we obtain more than 15 clusters, then we combine clusters based on the similarity of the cluster centers.
A.6. F IGURES
Figure A.5 shows sample trajectories in the FourRooms and EightRooms environment, with the initial and target positions marked in red and green respectively.
Figure A.2 shows the clusters after performing clustering with Afﬁnity Propagation using the successor representation withdemonstrations and an environment with 1% stochasticity on the transitions.
Different colors indicate different clusters.
Figures A.3 and A.4 show clusters for different environment settings.
Figure A. shows clusters when using 10 demonstrations and for Figure A.4 environments with 5% stochastictiy on transitions was used.
Figure A.6 shows the reward redistribution for the given example trajectories in the FourRooms and EightRooms environments.
A RTIFICIAL T ASK P - VALUES
Tables A.1 and A.2 show the p -values obtained by performing a Mann-Whitney-U test between
Align-RUDDER and BC+ Q -Learning and DQfD respectively.
Align-RUDDER vs. BC+ Q -Learn.
8.8e2.8e1.1e3.5e1.6eAlign-RUDDER vs. SQIL 3.6e5.2e3.1e8.6e1.9eAlign-RUDDER vs. DQfD 2.7e4.3e1.3e1.0e+ 1.0e+
Align-RUDDER vs. RUDDER (LSTM)
1.9e1.9e3.7e1.7e7.3e-
A.6. S TOCHASTIC E NVIRONMENTS
Figure A.7 shows results for the FourRooms environment with different levels of stochasticity (5%, 10%, 15%, 25% and 40%) on the transitions.
Figure A.8 shows results for the EightRooms environment with different levels of stochasticity (5% and 10%) on the transitions.
Align-RUDDER vs. BC+ Q -Learn.
4.5e1.3e4.9e3.7e6.1eAlign-RUDDER vs. SQIL 1.8e2.8e1.9e1.7e1.9eAlign-RUDDER vs. DQfD 1.2e8.9e5.6e1.0e+ 1.0e+
[Table]
LSTM)
1.2e1.3e3.9e8.7e1.0e-
[Figure]
E p
i s o d e s t
o
[Figure]
o p t i m a l r e t u r
[Figure]
i s
[Figure]
m
[Figure]
l r
[Figure]
t u r n
[Figure] [Figure]
i s o d e s t
[Figure]
p t i m a l r e t u r
[Figure]
A.6. C HANGING NUMBER OF C LUSTERS
We use Afﬁnity Propagation for clustering, and do not set the number of clusters.
Although, we set the max number of clusters allowed.
If Afﬁnity propagation results in more clusters, they are combined and reduced to the maximum clusters allowed.
This is necessary due to the limitations of the underlying alignment library we are using.
For the experiments on FourRooms and EightRooms in the main paper
[Figure]
we ﬁx the max number of clusters to 15.
We conduct an experimental study on how changing the max number of clusters changes the performance of Align-RUDDER on the FourRooms environment.
The results are in table A.
[Figure]
[Table]
EY
D ETECTION
1D key-chest environment.
We use a 1D key-chest environment to show the effectiveness of sequence alignment in a low data regime compared to an LSTM model.
The agent has to collect the key and then open the chest, to get a positive reward at the last timestep.
See Appendix Fig.
A.9 for a schematic representation of the environment.
As the key-events (important state-action pairs) in this environment are known we can compute the key-event detection rate of a reward redistribution model.
A key event is detected if the redistributed reward of an important state-action pair is larger than the average redistributed reward in the sequence.
We train the reward redistribution models with 2 ,and 10 training episodes and test on 1000 test episodes, averaged over 10 trials.
Align-RUDDER signiﬁcantly outperforms LSTM (RUDDER) for detecting these key events in all cases, with an average key-event detection rate of 0 .
96 for sequence alignment vs. 0 .
46 for the LSTM models over all dataset sizes.
See Appendix Fig.
A.10 for the detailed results.
[Figure]
v e n t
[Figure]
t
[Figure]
t i o n
R
[Figure]
t
[Figure]
A. M INECRAFT E XPERIMENTS
In this section we explain in detail the implementation of Align-RUDDER for solving the task
ObtainDiamond .
A.7. M INE C RAFT
We show that our approach can be applied to complex tasks by evaluating it on the MineRL Minecraft dataset (Guss et al., 2019b).
This dataset provides a large collection of demonstrations from human players solving six different tasks in the sandbox game MineCraft.
In addition to the human demonstrations the MineRL dataset also provides an OpenAI-Gym wrapper for MineCraft.
The dataset includes demonstrations for the following tasks:
• navigating to target location following a compass, • collecting wood by chopping trees, • obtaining an item by collecting resources and crafting, and • free play "survival" where the player is free to choose his own goal.
The demonstrations include the video showing the players’ view (without user interface), the players’ inventory at every time step and the actions performed by the player.
We focus on the third task of obtaining a target item, namely a diamond.
This task is very challenging as it is necessary to obtain several different resources and tools and has been the focus of a challenge (Guss et al., 2019a) at
NeurIPS’19.
By the end of this challenge no entry was able to obtain the diamond.
We show that our method is well suited for solving the task of obtaining the diamond, which can be decomposed into sub-tasks by reward redistribution after aligning successful demonstrations.
A.7. R ELATED W ORK AND S TEPS T OWARDS A G ENERAL A GENT
In the following, we review two approaches Skrynnik et al.
; Scheller et al.
(2020) where more details are available and compare them with our approach.
Skrynnik et al.
(2019) address the problem with a TD based hierarchical Deep Q -Network (DQN) and by utilizing the hierarchical structure of expert trajectories by extracting sequences of meta-actions and sub-goals.
This approach allowed them to achieve the 1st place in the ofﬁcial NeurIPS’
MineRL challenge (Skrynnik et al., 2019).
In terms of pre-processing, our approaches have in common that both rely on frame skipping and action space discretization.
However, they reduce the action space to ten distinct joint environment actions (e.g. move camera & attack ) and treat inventory actions separately by executing a sequence of semantic actions.
We aim at taking a next step towards a more general agent by introducing an action space preserving the agent’s full freedom of action in the environment (more details are provided below).
This allows us to avoid the distinction between item (environment) and semantic (inventory) agents and to train identically structured agents in the same fashion regardless of facing a mining, crafting, placing or smelting sub-task.
Skrynnik et al.
(2019) extract a sub-task chain by separately examining each expert trajectory and by considering the time of appearance of items in the inventory in chronological order.
For agent training their approach follows a heuristic where they distinguish between collecting the item log and all remaining items.
The log -agent is trained by starting with the TreeChop expert trajectories and then gradually injecting trajectories collected from interactions with the environment into the DQN’s replay buffer.
For the remaining items they rely on the expert data of ObtainIronPickaxeDense and imitation learning.
Given our proposed sequence alignment and reward redistribution methodology we are able to avoid this shift in training paradigm and to leverage all available training data ( ObtainDiamond ,
ObtainIronPickaxe and TreeChop ) at the same time.
In short, we collect all expert trajectories in one pool, perform sequence alignment yielding a common diamond consensus along with the corresponding reward redistribution and the respective sub-task sequences.
Given this restructuring of the problem into local sub-problems with redistributed reward all sub-task agents are then trained in the same fashion (e.g. imitation learning followed by RL-based ﬁne-tuning).
Reward redistribution guarantees that the optimal policies are preserved (Arjona-Medina et al., 2019).
Scheller et al.
(2020) achieved the 3rd place on the ofﬁcial leader board following a different line of research and addressed the problem with a single end-to-end off-policy IMPALA (Espeholt et al., 2018) actor-critic agent, again utilizing experience replay to incorporate the expert data (Scheller et al., 2020).
To prevent catastrophic forgetting of the behavior for later, less frequent sub-tasks they introduce value clipping and apply CLEAR
(Rolnick et al., 2019) to both, policy and value networks.
Treating the entire problem as a whole is already the main distinguishable feature compared to our method.
To deal with long trajectories they rely on a trainable special form of frame skipping where the agent also has to predict how many frames to skip in each situation.
This helps to reduce the effective length (step count) of the respective expert trajectories.
In contrast to the approach of (Scheller et al., 2020) we rely on a constant frame skip irrespective of the states and actions we are facing.
Finally, there are also several common features including:
a supervised BC pre-training stage prior to RL ﬁne-tuning, 2.
separate networks for policy and value function, 3.
independent action heads on top of a sub-sequence LSTM, 4.
presenting the inventory state in a certain form to the agent and 5.
applying a value-function-burn-in prior to RL ﬁne-tuning.
A.7. I MPLEMENTATION OF OUR A LGORITHM FOR M INECRAFT
The architecture of the training pipeline incorporates three learning stages:
• sequence alignment and reward redistribution • learning from demonstrations via behavioral cloning (pre-training) and • model ﬁne-tuning with reinforcement learning.
Figure A.11 shows a conceptual overview of all components.
Sequence alignment and reward redistribution.
First, we extract the sequence of states from human demonstrations, transform images into feature vectors using a standard pre-trained network and transform them into a sequence of consecutive state deltas (concatenating image feature vectors and inventory states). A pre-trained network can be model trained for image classiﬁcation or an auto-encoder model trained on images. In our case, we used an auto-encoder model trained on the MineRL obtainDiamond dataset.
We cluster the resulting state deltas and remove clusters with a large number of members and merged smaller clusters.
This results in 19 events and we map the demonstrations to sequences of events.
These events correspond to inventory changes.
For each human demonstration we get a sequence of events which we map to letters from the amino acid code, resulting in a sequence of letters.
In Fig.
A.12 we show all events with their assigned letter encoding that we deﬁned for the Minecraft environment.
[Figure] [Figure] [Figure] We then calculate a scoring matrix according to step (II) in Sec. 3 in the main document.
Then, we perform multiple sequence alignment to align sequences of events of the top N demonstrations, where shorter demonstrations are ranked higher.
This results in a sequence of common events which we denote as the consensus.
In order to redistribute the reward, we use the PSSM model and assign the respective reward.
Reward redistribution allows the sub-goal deﬁnition i.e. positions where the reward redistribution is larger than a threshold or positions where the reward redistribution has a certain value.
In our implementation sub-goals are obtained by applying a threshold to the reward redistribution.
The main agent is initialized by executing sub-agents according to the alignment.
Figure 4 shows how sub-goals are identiﬁed using reward redistribution.
Learning from demonstrations via behavioral cloning.
We extract demonstrations for each individual sub-task in the form of sub-sequences taken from all demonstrations.
For each sub-task we train an individual sub-agent via behavioral cloning.
Model ﬁne-tuning with reinforcement learning.
We ﬁne-tune the agent in the environment using
PPO (Schulman et al., 2018).
During ﬁne-tuning with PPO, an agent receives reward if it manages to reach its sub-goal.
To evaluate the performance of an agent for its current sub-goal, we average the return over multiple roll-outs.
This gives us a good estimate of the success rate and if trained models have improved during ﬁne tuning or not.
In Fig.
6, we plot the overall success rate of all models evaluated sequentially from start to end.
[Figure]
demo 5 demohuman demonstrations alignment log dirt planks crafting table stick wooden pickaxe cobblestone stone pickaxe iron ore iron ingot iron pickaxe diamond ore furnace events
[Figure]
[Figure]
consensus
S S S S S
S
S ...
P P P
P
L
V
N
P P P
P ...
L
V
N S S S S S
S
S replay until L
Agent replay until P ...
L
V
N S S S S S
S
S
P P P
P
In order to shorten the training time of our agent, we use trajectory replay and state resetting, similar to the idea proposed in (Ecoffet et al., 2019), allowing us to train sub-task agents in parallel. This is not necessary for the behavioral cloning stage, since here we can independently train agents according to the extracted sub-goals. However, ﬁne-tuning a sub-task agent with reinforcement learning requires agents for all previous sub-tasks. To ﬁne-tune agents for all sub-tasks, we record successful experiences (states, actions, rewards) for earlier goals and use them to reset the environment where a subsequent agent can start its training.
In Fig.
A.14, we illustrate a trajectory replay given by an exemplary consensus.
P OLICY AND V ALUE N ETWORK A RCHITECTURE
Figure A.15 shows a conceptual overview of the policy and value networks used in our MineRL experiments.
The networks are structured as two separate convolutional encoders with an LSTM layer before the respective output layer, without sharing any model parameters.
The input to the model is the sequence of the 32 most recent frames, which are pre-processed in the following way: ﬁrst, we add the currently equipped item as a color-coded border around each RGB frame.
Next, the frames are augmented with an inventory status bar representing all 18 available inventory items (each inventory item is drawn as an item-square consisting of 3 × 3 pixels to the frame).
Depending on the item count the respective square is drawn with a linearly interpolated gray-scale ranging from white (no item at all) to black (item count > 95 ).
The count of 95 is the 75 -quantile of the total amount of collected cobblestones and dirt derived from the inventory of all expert trajectories.
Intuitively, this count should be related to the current depth (level) where an agent currently is or at least has been throughout the episode.
In the last pre-processing step the frames are resized from 64 × 64 to 48 × 48 pixels and divided by 255 resulting in an input value range between zero and one.
The ﬁrst network stage consists of four batch-normalized convolution layers with ReLU activation functions.
The layers are structured as follows:
Conv-Layer-1 (16 feature maps, kernel size 4, stride 2, zero padding 1), Conv-Layer-2 (32 feature maps, kernel size 4, stride 2, zero padding 1),
Conv-Layer-3 (64 feature maps, kernel size 3, stride 2), and Conv-Layer-4 (32 feature maps, kernel size 3, stride 2).
The ﬂattened latent representation ( ∈ R 32 × 288 ) of the convolution stage is further processed with an LSTM layer with 256 units.
Given this recurrent representation we only keep the last time step (e.g. the prediction for the most recent frame).
The value head is a single linear layer without non-linearity predicting the state-value for the most recent state.
For action prediction, two types of output heads are used depending on the underlying action distribution.
The binary action head represents the actions attack , back , forward , jump , left , right , sneak and sprint which can be executed concurrently and are therefore modeled based on a
Bernoulli distribution.
Since only one item can be equipped, placed, or crafted at a time these actions are modeled with a categorical distribution.
The equip head selects from none , air , wooden-axe , wooden-pickaxe , stone-axe , stone-pickaxe , iron-axe and iron-pickaxe .
The place head selects from none , dirt , stone , cobblestone , crafting-table , furnace and torch .
The craft head selects from none , torch , stick , planks and crafting-table .
Items which have to be crafted nearby are none , wooden-axe , wooden-pickaxe , stone-axe , stone-pickaxe , iron-axe , iron-pickaxe and furnace .
Finally, items which are smelted nearby are none , iron-ingot and coal .
For predicting the camera angles ( up/down as well as left/right ) we introduce a custom action space outlined in Figure A.16.
This space discretizes the possible camera angles into 11 distinct bins for both orientations leading to the 22 output neurons of the camera action head.
Each bin holds the probability for sampling the corresponding angle as a camera action, since in most of the cases the true camera angle lies in between two such bins.
We share the bin selection probability by linear interpolation with respect to the distance of the neighboring bin centers to the true camera angle.
This way we are able to train the model with
[Figure] [Figure] [Figure] standard categorical cross-entropy during behavioral cloning and sample actions from this categorical distribution during exploration and agent deployment.
A.7. I MITATION L EARNING OF S UB -T ASK A GENTS
Given the sub-sequences of expert data separated by task and the network architectures described above we perform imitation learning via behavioral cloning (BC) on the expert demonstrations.
All sub-task policy networks are trained with a cross-entropy loss on the respective action distributions using stochastic gradient decent with a learning rate of 0 .
01 and a momentum of 0 .
Mini-batches of size 256 are sampled uniformly from the set of sub-task sequences.
As we have the MineRL simulator available during training we are able to include all sub-sequences in the training set and evaluate the performance of the model by deploying it in the environment every 10 training epochs.
Once training over 300 epochs is complete we select the model checkpoint based on the total count of collected target items over 12 evaluation trials per checkpoint.
Due to presence of only successful sequences, the separate value network is not pre-trained with BC.
A.7. R EINFORCEMENT L EARNING ON S UB -T ASK A GENTS
After the pretraining of the Sub-Task agents, we further ﬁne tune the agents using PPO in the MineRL environment.
The reward is the redistributed reward given by Align-RUDDER.
The value function is initialized in a burn-in stage prior to policy improvement where the agent interacts with the environment for 50k timesteps and only updates the value function.
Finally, both policy and the value function are trained jointly for all sub-tasks.
All agents are trained between 50k timesteps and 500k timesteps.
We evaluate each agent periodically during training and in the end select the best performing agent per sub- [Table] [Table]
[Table] present evaluation curves of some sub task agents during learning from demonstrations using behavioral cloning and learning online using PPO [Figure] [Figure]
[Figure]
A. R EPRODUCING THE A RTIFICIAL T ASK R ESULTS
The code to reproduce the results and ﬁgures of both artiﬁcial tasks is provided as supplementary material.
The README contains step-by-step instructions to set up an environment and run the experiments.
By default, instead of using 100 seeds per experiment only 10 are used in the demonstration code.
Finally
[Figure]
a video showcasing the MineCraft agent is also provided as supplementary material
[Figure] [Figure]
A. S OFTWARE L IBRARIES
We are thankful towards the developers of Mazelab Zuo (2018), PyTorch Paszke et al.
, OpenAI
Gym Brockman et al.
Numpy Harris et al.
(2020), Matplotlib Hunter (2007) and Minecraft
Guss et al.
(2019b).
A. C OMPUTE
Artiﬁcial task (I) and (II) experiments were performed using CPU only as GPU speed-up was negligible.
The ﬁnal results for all methods were created on an internal CPU cluster with 128 CPU cores with a measured wall-clock time of 10,360 hours.
The majority of compute is spent on baseline methods.
For minecraft, during development 6 to 8 nodes each with 4 GPUs of an internal GPU cluster were used for roughly six months of GPU compute time (Nvidia Titan V and 2080 TI).
The compute required for training the ﬁnal agent was well within the challenge parameters (4 days on a single node with one GPU).