G ROUNDING A LEATORIC U NCERTAINTY IN U NSUPERVISED E NVIRONMENT D ESIGN
Anonymous authors
Paper under double-blind review
A BSTRACT
In reinforcement learning (RL), adaptive curricula have proven highly effective for learning policies that generalize well under a wide variety of changes to the environment.
Recently, the framework of Unsupervised Environment Design (UED) generalized notions of curricula for RL in terms of generating entire environments, leading to the development of new methods with robust minimaxregret properties.
However, in partially-observable or stochastic settings (those featuring aleatoric uncertainty ), optimal policies may depend on the groundtruth distribution over the aleatoric features of the environment.
Such settings are potentially problematic for curriculum learning, which necessarily shifts the environment distribution used during training with respect to the ﬁxed groundtruth distribution in the intended deployment environment.
We formalize this phenomenon as curriculum-induced covariate shift , and describe how, when the distribution shift occurs over such aleatoric environment parameters, it can lead to learning suboptimal policies.
We then propose a method which, given black-box access to a simulator, corrects this resultant bias by aligning the advantage estimates to the ground-truth distribution over aleatoric parameters.
This approach leads to a minimax-regret UED method, SAMPLR, with Bayes-optimal guarantees.
I NTRODUCTION
Adaptive curricula, which dynamically adjust the distribution of training environments to optimize the performance of the resulting policy, have played a key role in many recent achievements in deep reinforcement learning (RL).
Applications have spanned both single-agent RL (Portelas et al., 2020; Wang et al., 2019; Zhong et al., 2020; Justesen et al., 2018), where adaptation occurs over environment variations, and multi-agent RL (MARL), where adaptation can additionally occur over co-players (Silver et al., 2016;
Vinyals et al., 2019; Stooke et al., 2021).
By presenting the agent with challenges at the threshold of its abilities, such methods demonstrably improve the sample efﬁciency and the generality of the ﬁnal policy (Matiisen et al., 2017; Dennis et al., 2020; Jiang et al., 2021b;a).
This work introduces a fundamental problem relevant to adaptive curriculum learning methods for RL, which we call curriculum-induced covariate shift (CICS).
Analogous to the covariate shift that occurs in supervised learning (SL), CICS refers to a mismatch between the input distribution at training and test time, and in this case, speciﬁcally when the distribution shift is caused by the selective sampling performed by an adaptive curriculum.
While there may be cases in which CICS impacts model performance in SL, adaptive curricula for SL have generally not been found to be as impactful as in
RL (Wu et al., 2021).
Therefore, here we focus on addressing this problem speciﬁcally as it arises in the RL setting, and leave investigation of its potential impact in SL to future work.
To establish precise language around adaptive curricula, we cast our discussion under the lens of
Unsupervised Environment Design (UED, Dennis et al., 2020).
UED provides a formal problem description for which curriculum learning is the solution, by deﬁning the Underspeciﬁed POMDP (UPOMDP; see Section 2), which expands the classic POMDP with a set of free parameters Θ , representing the dimensions along which the environment may vary across episodes.
The goal of
UED is then to adapt distributions over Θ , so to maximize some objective, which could be tied to an
RL agent’s performance over this distribution.
This allows us to view adaptive curricula as emerging via a multi-agent game between a teacher that proposes environments with parameters θ ∼ Θ and a student that learns to solve them.
In addition to the notational clarity it provides, this formalism lends
[Figure]
the analysis of adaptive curricula to useful game theoretic constructs, such as Nash equilibria (NE,
Nash et al., 1950).
This game-theoretic view has led to the development of curriculum methods with principled robustness guarantees, such as PAIRED (Dennis et al., 2020) and Prioritized Level Replay (PLR, Jiang et al., 2021a), which showed that curricula optimizing for the student’s regret lead to minimax regret (Savage, 1951) policies at Nash equilibria, implying the agent can solve all solvable environments within the training domain.
While other methods can also be cast in this framework, they do not hold the same desirable property at equilibrium.
For this reason, we will focus on addressing CICS for regret-maximizing UED, but note that our solution can be used with other UED methods.
To see how the CICS can be problematic, consider the simpliﬁed case of training a self-driving car in simulation, so it learns to take the fastest route from home to ofﬁce.
Suppose trafﬁc data shows on 70% of the days, Route 1 is faster than Route 2.
Moreover, on any given day the self-driving car cannot infer which route is faster ahead of time, so always picking Route 1 is faster in expectation.
To support training a policy using an adaptive curriculum, one could build a simulator which sets road conditions per episode based on a random day sampled from the trafﬁc data.
However, adaptive curriculum over the trafﬁc settings may oversample days when Route 1 is closed—perhaps because it ﬁnds the agent needs more practice on Route 2—shifting the best choice to Route 2 in training.
In fact, methods for minimax regret UED like PLR would keep shifting the distribution of fastest route, to maximize the agent’s regret.
Their curriculum dynamics would map to the zero-sum game of matching pennies, in which one player wins for guessing whether the other chose heads or tails; the NE corresponds to each player randomly playing each option half the time.
Randomly picking the route in this way is suboptimal, because in reality Route 1 is faster in expectation.
This example is depicted in Figure 1, where the two routes are replaced by an apple and banana.
If, on a given day, the faster route could be identiﬁed before having to pick one, the agent could choose optimally.
Instead, it is an aleatoric parameter, inducing irreducible uncertainty in the limit of inﬁnite experiential data (Der Kiureghian & Ditlevsen, 2009).
When CICS occurs over such parameters, with respect to a ground-truth distribution of environments P ( θ ) , the learned policy can be suboptimal with respect to P .
It can therefore be useful to ground —that is, to constrain—the aleatoric parameters Θ (cid:48) ⊂
Θ to P ( θ (cid:48) ) when it is known or can be learned, as in simulation or from real-world data.
However, grounding all θ is undesirable, preventing the curriculum from sampling enough opportunities to learn from useful scenarios with low support under P ( θ ) .
In this work, we formalize the problem of CICS in RL, and provide a solution by proposing a UED method to ﬁnd robustly Bayes-optimal policies where θ (cid:48) is grounded to P ( θ (cid:48) ) .
Our solution called
Sample-Matched PLR (SAMPLR) extends PLR, a state-of-the-art UED algorithm, by constraining the advantage estimates to match those observed if training under P ( θ (cid:48) ) .
This advantage correction adapts
Off-Belief Learning
(Hu et al., 2021) from cooperative MARL, revealing an intriguing connection between curriculum biases observed in single and multi-agent RL.
Our experiments in challenging environments based on the NetHack Learning Environment (NLE, Küttler et al., 2020) demonstrate that SAMPLR learns near-optimal policies under CICS, in cases where standard PLR fails.
B ACKGROUND 2. U NSUPERVISED E NVIRONMENT D ESIGN
The problem of Unsupervised Environment Design (UED, Dennis et al.
(2020)) is the problem of automatically generating an adaptive distribution of environments which will lead to policies that successfully transfer within a target domain.
The domain of possible environments is represented by an Underspeciﬁed POMDP (UPOMDP), which adds a set of free parameters to the standard deﬁnition of a POMDP, along which each concrete instantiation, or level , of the UPOMDP.
For instance, these free parameters can be the position of obstacles in a maze, or friction coefﬁcients in a physics-based task.
Formally a UPOMDP is deﬁned as a tuple M = (cid:104) A, O, Θ , S , T , I , R , γ (cid:105) , where A is a set of actions, O is a set of observations, Θ is a set of free parameters, S is a set of states, T : S ×
A × Θ → ∆ ( S ) is a transition function, I : S
→ O is an observation (or inspection) function, R :
S
→ R is a reward function, and γ is a discount factor.
UED typically approaches the curriculum design problem as training a teacher agent that co-evolves an adversarial curriculum for a student agent, for example, by maximizing the student’s regret.
We will focus on a recent UED algorithm called Robust Prioritized Level Replay (PLR ⊥ , Jiang et al., 2021b), which performs environment design via random search.
PLR maintains a buffer of the most useful levels for training, according to some learning potential score—typically based on a regret approximation, such as the positive value loss—and with probability p , actively samples the next training level from this level buffer instead of the ground-truth training distribution.
This selective-sampling mechanism has been demonstrated to greatly improve sample-efﬁciency and generalization in several domains, while provably leading to a minimax regret policy for the student at NE.
In maximizing regret, PLR curricula naturally avoid unsolvable levels, which have no regret.
2. O FF -B ELIEF L EARNING
In cooperative MARL, self-play promotes the formation of cryptic conventions—arbitrary sequences of actions that allow agents to communicate information about the environment state.
These conventions are learned jointly among all agents during training, but are arbitrary and hence, indecipherable to independently-trained agents or humans at test time.
Crucially, this leads to policies that fail to perform zero-shot coordination (ZSC, Hu et al., 2020), where independentlytrained agents must cooperate successfully without additional learning steps, or ad-hoc team play.
Off-Belief Learning (OBL) resolves this problem by forcing agents to assume their co-players act according to a ﬁxed, known policy π 0 until the current time t , and optimally afterwards, conditioned on this assumption.
If π 0 is playing uniformly random, this removes the possibility of forming arbitrary conventions.
Formally, let G be a decentralized, partially-observable MDP (Dec-POMDP, Bernstein et al., 2002), with state s , joint action a , observation function
I i ( s ) for each player i , and transition function
T ( s, a ) .
Let the historical trajectory τ =
( s 1 , a 1 , ...a t − 1 , s t ) , and the action-observation history (AOH) for agent i be τ
i =
( I i ( s 1 ) , a 1 , ..., a t − 1 , I i ( s t )) .
Further, let π 0 be an arbitrary policy, such as a uniformly random policy, and B π 0 ( τ | τ i )
=
P ( τ t | τ it , π 0 ) , a belief model predicting the current state, conditioned on the AOH of agent i and the assumption of co-players playing policy π 0 until the current time t , and optimally according to π 1 from t and beyond.
OBL aims to ﬁnd the policy πwith the optimal, counter-factual value function ,
[Equation]
As the agent conditions its policy on the realized AOH τ i , while transition dynamics are based on states sampled from B , this mechanism is called a ﬁctitious transition .
In Section 5, we show how
OBL’s ﬁctitious transition can be adapted to the single-agent curriculum learning setting to address
CICS, by interpreting the curriculum designer in UED as a co-player.
R ELATED W ORK
The mismatch between training and testing distributions of input features is referred to as covariate shift , and has long served as a fundamental problem for the machine learning community.
Covariate shifts have been extensively studied in supervised learning (Vapnik & Chervonenkis, 1971;
Huang et al., 2006; Bickel et al., 2009; Arjovsky et al., 2019).
In RL, prior works have largely focused on covariate shifts due to training on off-policy data (Sutton et al., 2016;
Rowland et al., 2020;
Espeholt et al., 2018; Hallak & Mannor, 2017; Gelada & Bellemare, 2019; Thomas & Brunskill, 2016) including the important case of learning from demonstrations (Pomerleau, 1988; Ross &
Bagnell, 2010).
Recent work also aims to learn invariant representations robust to covariate shifts (Zhang et al., 2019; 2021).
More generally, CICS can be interpreted as a kind of sample-selection bias (Heckman, 1979).
We believe this work to be the ﬁrst to formalize and provide a solution to the problem of covariate shifts in reinforcement learning due to curriculum learning.
Our method ﬁxes a critical ﬂaw that can cause curricula to fail under CICS—an important problem as curricula have been shown to be essential for training RL agents across many of the most challenging domains, including combinatorial gridworlds (Zhong et al., 2020), Go (Silver et al., 2016), StarCraft(Vinyals et al., 2019), and achieving comprehensive task mastery in open-ended environments (Stooke et al., 2021).
While this work focuses on PLR, other recent methods include minimax adversarial curricula (Wang et al., 2019; 2020) and curricula based on changes in return (Matiisen et al., 2017;
Portelas et al., 2020).
Most similar to our work, OFFER (Ciosek & Whiteson, 2017) adapts a curriculum over transition functions and uses importance sampling to correct for biased gradient estimates.
Unlike this work, Ciosek & Whiteson (2017) requires whitebox access to the transition function and does not directly study the impact of CICS on the learning dynamics.
Curriculum methods have also been studied in goal-conditioned RL (Florensa et al., 2018; Campero et al., 2021;
Sukhbaatar et al., 2018; OpenAI et al., 2021), though CICS does not occur here as goals are observed by the agent.
Lastly, domain randomization (DR, Sadeghi & Levine, 2017; Peng et al., 2017) can be seen as a degenerate form of UED, though curriculum-based extensions of DR have also been studied (Jakobi, 1997; Tobin et al., 2017).
Prior work has also investigated methods for learning Bayes optimal policies under uncertainty about the task (Zintgraf et al., 2020;
Osband et al., 2013), based on the framework of Bayes-adaptive
MDPs (BAMDPs) (Bellman, 1956; Duff, 2002).
In this setting, the agent can adapt to an unknown
MDP over several episodes by acting to reduce its uncertainty about the identity of the MDP.
In contrast, SAMPLR learns a robustly Bayes optimal policy for the case of zero-shot transfer.
Further unlike these works, our setting assumes the distribution of certain aleatoric parameters are biased during training, which would lead to biased a posteriori uncertainty estimates with respect to the ground-truth distribution when optimizing for the BAMDP objective.
Instead, SAMPLR proposes a means to correct for this bias assuming knowledge of the true environment parameters for each level, to which we can safely assume access in curriculum learning.
C URRICULUM -I NDUCED C OVARIATE S HIFT
As UED algorithms formulate curriculum learning as a multi-agent game between teacher and student agents, we can formalize when CICS become problematic by considering the equilibrium point of this game: Let θ be the environment parameters controlled by UED, P ( θ ) , the ground truth distribution of θ , and P ( θ ) , the curriculum distribution at equilibrium.
Further, let τ t be ( o 1 , a 1 , ..., a t − 1 , o t ) , the student agent’s action-observation history (AOH) until time t (though we will use simply τ when clear from context).
The optimal action-value function Q ∗ with respect to P ( θ ) can then be expressed as a marginalization over θ :
[Equation]
From Equation 2, we see that Q ∗ ( a t | τ t ) remains optimal under different values of P ( θ ) at each time t as long as it is possible to infer θ deterministically from τ t , implying that P ( θ (cid:48) | τ t )
= 1 for some θ , in which case the RHS of Equation 1 reduces to the LHS.
If P ( θ | τ t ) < 1 for all θ , then some subset θ (cid:48) ⊂
Θ results in irreducible uncertainty at time t , thereby constituting aleatoric parameters.
Letting Q ∗ ( a t | τ t ) be the optimal action-value function when P ( θ ) is replaced with P ( θ ) in Equation 2, we can then state that curriculum-induced covariate shift results in suboptimal policies with respect to the ground truth distribution when
[Equation]
Moreover, this formulation highlights how this effect results from the presence of aleatoric parameters.
This description categorizes errors made by a policy trained on P evaluated on levels drawn from P ( θ ) into two categories: The ﬁrst type of error, which we call a mistake , simply arises when P ( θ (cid:48)(cid:48) )
=for some non-aleatoric θ (cid:48)(cid:48) where P ( θ (cid:48)(cid:48) ) > 0 , i.e. the policy did not train on trajectories needed to learn to behave optimally under some set of otherwise identiﬁable level parameters.
This is distinct from the second kind of error, which we call a misunderstanding , and which corresponds to the problematic mismatch between training and test time distributions of speciﬁcally aleatoric parameters θ (cid:48) , so that P ( θ (cid:48) ) (cid:54) =
P ( θ (cid:48) ) , as we just previously characterized.
This taxonomy also clariﬁes errors in cooperative MARL, where the co-players shape environment interactions, and thus play a similar role to the UED teacher.
Failures in ZSC can then be diagnosed as due to (i) a mistake, because the co-players fail to generate trajectories that occur with test-time co-players; or (ii), a misunderstanding, because the train-time co-players shift the training distribution of aleatoric parameters P ( θ (cid:48) ) , which impacts the inference of P ( τ )
= (cid:80) θ (cid:48)
P ( θ (cid:48) ) P ( τ | θ (cid:48) ) needed for optimal cooperation—for example, through the use of cryptic conventions.
This view then connects generalization errors in cooperative MARL to those in single-agent RL, implying that methods like
OBL devised to solve one type of error in one of the settings may be adapted for the other.
Indeed, we now describe our method, which does exactly this:
By adapting OBL to single-agent curriculum learning, we can ground the values of P ( θ (cid:48) | τ t ) by forcing P ( θ (cid:48) )
=
P ( θ (cid:48) ) , thereby ensuring that the distribution of the aleatoric parameters at equilibrium is equivalent to their ground truth distribution.
S AMPLE -M ATCHED PLR (SAMPLR)
We now describe how OBL’s ﬁctitious transition can be adapted for PLR ⊥ (Jiang et al., 2021a) to address CICS, resulting in Sample-Matched PLR (SAMPLR).
To avoid CICS, we must ground θ (cid:48) to the ground-truth distribution P ( θ (cid:48) ) , while allowing the remaining parameters in Θ to vary under
UED, so as to still beneﬁt from a curriculum.
To achieve this, we adapt the ﬁctitious transition to single-agent curriculum learning by treating the UED teacher as a co-player—one that performs the single action of choosing the level θ at the start of each episode, and subsequently performs no-ops.
Under this ﬁctitious transition, we ground the teacher’s choice of θ such that the aleatoric parameters θ (cid:48) are assumed to be sampled from P ( θ ) .
Thus, at each time t , the agent takes actions a t based on its AOH as usual, but estimates the advantage
A ( a t , s t ) using ﬁctitious transitions, which assume subsequent state transitions and rewards occur with Θ (cid:48) ﬁxed to θ (cid:48) ∼ P ( θ (cid:48) ) , sampled at the start of the episode.
More formally, the ﬁctitious transition is performed as s (cid:48) t ∼ B ( s (cid:48) t | τ ) , a t ∼ π ( ·| τ ) , s (cid:48) t +1 = T ( s (cid:48) t , a t ) , and r (cid:48) t = R ( s t +1 ) , where τ is the
Randomly initialize policy π ( φ ) , an empty level buffer Λ of size K , and belief model B ( s t | τ ) .
while not converged do
Sample replay-decision Bernoulli, d ∼ P D ( d ) if d = 0 or | Λ | = 0 then
Sample level θ from level generator Collect π ’s trajectory τ on θ , with a stop-gradient φ ⊥
Use observed ground-truth states to update B else
Use PLR to sample a replay level from the level store, θ ∼ Λ
Collect ﬁctitious trajectory τ (cid:48) on θ , based on s (cid:48)
t ∼ B
Update π with rewards R ( τ (cid:48) ) end
Compute PLR score, S = score ( τ (cid:48) , π )
Update Λ with θ using score S end
AOH of the student.
Figure 2 summarizes this transition mechanism.
Here, the belief model B ( s t | τ ) can be expressed as
[Equation]
This shows that, assuming blackbox simulator access, we can generally implement B as follows:
Periodically during training, we sample N trajectories, such that each is generated under a level with θ (cid:48) ∼ P ( θ (cid:48) ) .
We use these { ( θ (cid:48) k , τ k ) } Nk =1 pairs to update a posterior model P ( θ (cid:48) | τ ) that predicts the underlying θ (cid:48) given τ .
We can then sample from B by ﬁrst sampling θ ∼ P ( θ (cid:48) | τ ) , followed by stepping forward a parallel simulator that has been initially reset to the current AOH τ , with ﬁxed Θ (cid:48)
= θ (cid:48) , thereby yielding a desired sample of s t according to B .
In practice, it is often the case that θ (cid:48) can be uniquely identiﬁed by some revelatory event by time t , so that P ( θ (cid:48) | τ ) = 1 for some θ (cid:48) and τ , and P ( s t | τ, θ (cid:48) )
[-= 
-]
=
P ( s t | τ ) otherwise.
In this case, we can implement the ﬁctitious transition by setting θ (cid:48) ∼ P ( θ (cid:48) ) at the start of each episode; subsequent transitions will then be consistent with B .
For example, in the fruit choice example, whether apple or banana was the right goal is deterministically revealed by the ﬁnal reward, and otherwise, does not impact transition dynamics.
Additionally, we often only have limited access to P ( θ ) throughout training, for example, if sampling P ( θ ) is costly.
In this case, we can learn an estimate ˆ P ( θ (cid:48) ) using the samples we do collect from P ( θ ) , which can occur online.
We then use ˆ P ( θ (cid:48) ) in sampling ﬁctitious transitions during UED.
We refer to the resulting ˆ P ( θ (cid:48) ) as a learned belief prior .
SAMPLR, summarized in Algorithm 1, incorporates this ﬁctitious transition by replacing the advantages of trajectories on replay levels sampled by PLR ⊥ with their ﬁctitious counterparts, as only these trajectories are used by PLR ⊥ for training.
To reduce the cost of sampling P ( θ (cid:48) ) , we can use the new levels regularly sampled by PLR ⊥ to estimate the learned belief prior P ( θ (cid:48) ) , and use
P ( θ (cid:48) ) in sampling ﬁctitious transitions on replay levels.
G ROUNDED P OLICIES A RE R OBUSTLY B AYES O PTIMAL
We can view OBL-based correction as a method for training a policy to be optimal with respect to the ground-truth value function, with levels sampled from some generating distribution Λ deﬁned as:
[Equation]
Note that when Λ = P ( θ ) this reduces to the ground-truth value function notated simply as V ( π ) .
First, we will note that, for any UED method, our OBL-based correction will ensure that, in equilibrium, the resulting policy is Bayes-optimal on the ground truth beliefs, on any trajectory sampled from M Λ ( π ) , the distribution of trajectories of π in levels sampled from Λ .
Remark 1.
If π is optimal with respect to the grounded value function V Λ ( π ) then it is Bayes optimal with respect to the ground-truth parameter distribution P ( θ ) on the support of M Λ ( π ) .
Proof.
By deﬁnition we have that π ∈ arg max π ∈ Π { V Λ ( π ) } =
arg max π ∈ Π { E τ ∼M Λ ( π ) (cid:104)
V π ( τ ) (cid:105) } .
Since π can condition on the initial trajectory τ , the action selected after each trajectory can be independently optimized.
Thus we have, for all τ ∈ M Λ ( π ) , π ∈ arg max π ∈ Π { V π ( τ ) } implying that π is the optimal grounded policy and V π = V ∗ .
Thus, assuming the base RL algorithm ﬁnds Bayes optimal policies, a UED method that optimizes the grounded value function, as done by SAMPLR, will result in Bayes optimal performance over the ground-truth distribution.
When the UED method aims to maximize worst-case regret, we can prove an even stronger property we call robust (cid:15)
-Bayes optimality .
Let V θ ( π ) be the value function for π evaluated on a speciﬁc level θ . We will say that a policy is robustly (cid:15) -Bayes optimal iff for all θ in the domain of P ( θ ) and for all π (cid:48)
we have
[Equation]
Note how this differs from being only (cid:15) -Bayes optimal, which means for all π (cid:48) ,
[Equation]
With robust (cid:15) -Bayes optimality, we must be (cid:15) -optimal even on levels which are rarely sampled from the ground-truth distribution.
We will show that if SAMPLR is in an (cid:15) -Nash Equilibrium, then a policy is robustly (cid:15) -Bayes optimal with respect to the grounded value function V ( π ) rather than only (cid:15) -Bayes optimal as one would expect from training directly on the true distribution of levels.
Theorem 1.
If π is (cid:15) -Bayes optimal by V Λ ( π ) for Λ minimizing worst-case regret as is done in SAMPLR, then it is robustly (cid:15) -Bayes optimal with respect to the grounded value function, V ( π ) .
Proof.
Let π be (cid:15) -optimal
with respect to V Λ ( π ) where Λ minimizing worst-case regret with respect to π .
Let π ∗ be an optimal grounded policy, and let θ be arbitrary.
Then we have:
[Equation]
Where the ﬁrst inequality follows from Λ minimizing worst-case regret with respect to π , and the second follows from π being (cid:15) -optimal on Λ .
Rearranging terms gives the desired condition.
E XPERIMENTS
We investigate the performance of SAMPLR with respect to the the standard PLR and domain randomization in environments based on MiniHack (Samvelyan et al., 2021), a library for creating custom environments based on the runtime of the NetHack Learning Environment (NLE) (Küttler et al., 2020).
Acting optimally in our environments requires grounding to the ground-truth distribution.
Our agents are trained using PPO (Schulman et al., 2017), using the best hyperparameters found via grid search, and use the policy architecture in Küttler et al.
We tune the PLR-speciﬁc hyperparameters shared among PLR ⊥ and SAMPLR variants for each environment, based on the performance of PLR ⊥ .
Full details of our environments, agent architecture, and hyperparameters are provided in Appendix A, and our SAMPLR implementation, in Appendix B.
We compare both standard SAMPLR and a variant called LP-SAMPLR that learns a belief prior over the true goal, against PLR ⊥ and standard PPO baselines.
These baselines allows us to separate the relative changes in performance metrics due to curriculum learning and our proposed correction for CICS.
[Figure]
In our experiments, we ﬁrst adapt the apple and banana example, depicted in Figure 3 into a fully procedurally-generated RL environment set in the world of NetHack.
For each level, the agent can only learn to choose optimally in expectation at test time by grounding to the distribution of correct choices in the deployment domain.
Speciﬁcally, in each level, the agent must traverse between one to eight randomly generated rooms, and in the ﬁnal room, the agent must choose to eat the apple or the banana.
The correct choice is ﬁxed for each level, but indiscernible to the agent.
Thus, the identity of the true goal acts as the aleatoric parameter.
Figure 3 shows example levels from this environment.
This environment presents a hard exploration challenge for standard RL algorithms, as it requires learning to both navigate multiple rooms, as well as the NLE-speciﬁc skills of kicking doors and eating.
The doors opening into adjacent rooms are locked.
In order to go from one room to the next, the agent must learn to, potentially repeatedly, kick the locked door until it opens.
Likewise, upon reaching a piece of fruit in the ﬁnal room, the agent must learn to deliberately choose to eat the fruit.
If the right choice of fruit were determinable per episode, we expect PLR ⊥ ’s adaptive curriculum to improve learning by selectively sampling levels at the threshold of the agent’s abilities.
Let π A be the policy in which the agent always chooses the apple, and π B , the banana.
If the probability of the goal being apple
P ( A ) =
q , the expected return of the agent is R A q under π A and R B (1 − q ) under π B .
The optimal policy is then to act according to π A when q > R B / ( R A + R B ) , and according to π B otherwise.
We expect training under domain randomization, which samples each level completely at random from the ground-truth distribution P ( θ ) , deﬁned by the environment parameterization, would converge to the correct choice of π A or π B , assuming the environment is learnable by the choice of RL algorithm.
In contrast, PLR evolves an adversarial curriculum, incentivizing
PLR to shift the distribution over goals throughout training.
For example, PLR is incentivized to ﬂip the curriculum distribution to favor eating apples whenever the agent begins to consistently succeed at eating bananas.
We thus expect the PLR curriculum to continually oscillate the preferred choice of goal.
In our experiments, we set R A
= 3 , R B = 10 , and q in { 0 . 7 , 0 . 5 , 0 . 3 } , making it always optimal to follow the banana-eating policy π B , but with the marginal beneﬁt of doing so varying with q .
We report train and test performance of each agent over 200M training steps in Figure 4.
We ﬁnd that
DR struggles to learn an effective policy, plateauing at an expected return under 1 . 0 ; PLR performs even worse, due to its adversarial curriculum shifting the distribution over the correct goal, leading to rapid oscillations in the optimal choice of fruit under the curriculum distribution, as visible in the high-variance oscillations in the proportion of apple goals selected by PLR for each value of q in 4).
This makes it difﬁcult for the agent to settle on the optimal policy with respect to any ground-truth distribution.
The proportion of each choice outcome, shown in Figure 5, reveals that both DR and
PLR policies fail to eat any fruit most of the time, even after 200M training steps.
In contrast, both
SAMPLR and LP-SAMPLR realize a marked improvement in test performance by grounding PLR’s
[Figure]
P r opo r t i on s e l e c t ed
[Figure]
[Figure]
otherwise wild shifts in q .
Figure 4 shows this improvement is most pronounced when the expected difference in π A and π B is smaller, and therefore easier for PLR to ﬂip the agent’s policy.
Moreover, we see in Figure 5 that both SAMPLR methods present levels with higher room counts on average, indicating that early in training, it is able to discover easier, few-room levels in which the agent can capture reward signal, and which can then be made more complex to push the agent along its threshold of abilities.
We next turn to a more challenging environment that introduces additional NetHack-speciﬁc dynamics.
The FireDungeon environment (see Figure 6 for example levels), requires the agent to navigate through up to 13 chambers, and ultimately choose between chamber A or B, each containing a valid goal.
Reaching this goal ends the episode and provides the agent with a sparse reward.
Further, in this penultimate chamber, there is an armor, which if worn grants the agent with ﬁre resistance with probability q .
Chamber B is marked by a red gemstone by the door and contains enemy units, whose ﬁre attack will instantly kill the agent.
Killing each enemy grants the agent with +1 ﬁnal reward, which is only provided upon reaching either goal.
The agent thus stands to attain a higher reward by attacking the enemies in Chamber B, before reaching the goal, only if the armor is ﬁreproof.
This environment presents an even more difﬁcult exploration problem for the agent, yet we see in
Figure 6, both SAMPLR variants begin to learn to solve this environment with signiﬁcantly greater sample-efﬁciency than DR, while PLR again struggles to learn.
As in the case of the previous stochastic choice environment, we see that PLR rapidly oscillates the key aleatoric parameter, which in this case, determines whether the armor is ﬁreproof.
C ONCLUSION
Using the formal notions of environment parameterizations in the framework of UED, we deﬁned the problem of curriculum-induced covariate shift in RL.
Our deﬁnition highlights the issues that can arise when there is persisitent uncertainty over the environment parameters, either because the uncertainty is irreducible or because reducing the uncertainty is costly.
We then adapted a ﬁctitious transition mechanism previously used to improve zero-shot coordination in cooperative MARL to correct for this covariate shift.
We demonstrated that our resulting algorithm, SAMPLR, avoids the pitfalls of this type of covariate shift, while preserving the beneﬁts of curriculum learning. A E XPERIMENTAL D ETAILS AND H YPERPARAMETERS
In this section, we expand on stochastic binary choice environment featured in our experiments, as well as our choice of agent architecture and hyperparameters.
Additionally, we provide details on how we implemented SAMPLR and LP-SAMPLR for this environment.
A. E NVIRONMENT D ETAILS
Multi-Room Stochastic Fruit Choice We make use of MiniHack (Samvelyan et al., 2021), a library built on top of the NetHack Learning Environment (NLE, Küttler et al., 2020), for creating custom environments in the NetHack runtime.
In particular, our environment, MultiRoomBinaryChoice-N8-Random, expands the example in Figure 1 into a challenging, procedurallygenerated exploration problem.
The agent must navigate through up to eight rooms in each level, and in the ﬁnal room, choose the correct piece of fruit, either the apple or banana, to receive a reward.
If the agent eats the wrong fruit for the level, it receives a reward of 0 .
With probability q , the apple is the correct fruit to eat.
Eating either fruit terminates the episode.
The episode also terminates once the budget of 250 steps is reached.
Notably, passage into adjacent rooms requires ﬁrst kicking down a locked door.
As per NLE game dynamics, locked doors may require a random number of kicks before they give way.
To complicate the learning of this kicking skill, kicking the stone walls of the room will lower the agent’s health points; multiple misguided kicks can then lead to the agent dying, ending the episode.
FireDungeon Each level of the FireDungeon environment corresponds to a procedurally-generated sequence of connected dungeon chambers.
Except for the penultimate chamber, which opens to two goal chambers, each chamber only has one adjacent chamber, connected by a locked door.
Entering any adjacent chamber thus requires ﬁrst kicking open the locked door.
The penultimate chamber is connected via locked doors to two goal chambers, and contains a piece of armor of a random variety.
With probability q , the armor is of the ﬁreproof variety.
The ﬁrst room is empty aside from a goal, which when reached, terminates the episode and grants the agent +1 reward.
The second room contains a goal, in addition to three enemy units (pyrolisks), whose ranged ﬁre attack can instantly kill the agent.
However, the agent is immune to the pyrolisk’s attacks when wearing ﬁreproof armor.
Killing each pyrolisk earns the agent an additional +1 reward, creating a natural choice for the agent: Whether to brave the room with enemies for a greater return or choose the safer goal with a guaranteed +1 reward.
The optimal choice depends on whether the armor is ﬁreproof, the key aleatoric parameter for this environment.
Observation space The agent’s observation consists of two primary elements: The nethack glyph and blstats tensors.
The glyph tensor represents a 2D symbolic observation of the dungeon.
This glyph tensor contains a 21 × 79 window of glyph identiﬁers, which can each be one of thepossible glyphs in NetHack, which represent monsters, items, environment features, and other game entities.
The blstats vector contains character-centric values, such as the agent’s coordinates and the information in the “bottom-line stats” of the game, such as the agent’s health stats, attribute levels, armor class, and experience points.
Action space
The agent’s actions include the eight navigational actions, corresponding to moving toward each cell in the agent’s Moore neighborhood, in addition to two additional actions for kicking (doors) and eating (apples and bananas).
Melee attacks in NLE correspond to simply walking towards and colliding with an enemy.
A. A RCHITECTURE
We make use of the same agent architecture from Küttler et al.
The policy applies a
ConvNet to all visible glyph embeddings, and a separate ConvNet to a 9 × 9 egocentric crop around the agent—which was found to improve generalization—producing two latent vectors.
These are then concatenated with an MLP encoding of the blstats vector, and the resulting vector is processed by an MLP, and ﬁnally, input through an LSTM to produce the action distribution.
We used the implementation of this policy architecture available at https://github.com/
A. C HOICE OF H YPERPARAMETERS
Our choice of PPO hyperparameters, shared across all baselines and SAMPLR variants in our experiments, was based on a grid search, in which we train agents with domain randomization on a 15 × 15 maze, in which up to 50 walls can be randomly placed in each position
[Table]
and the agent’s starting position along with the positions of the goal and up to 50 walls are randomly placed in each level.
We chose this environment to perform the grid search, as it allows for faster training than the multi-room environment featured in our main experiments.
Speciﬁcally, we swept over the following hyperparameter values: number of PPO epochs in 5 , 20 , number of PPO minibatches in 1 , 4 , PPO clip parameter in { 0 . 1 , 0 . 2 } , learning rate in { 1 e- 3 , 1 e- 4 } , and discount factor γ in { 0 . 99 , 0 . 995 } .
Fixing these hyperparameters to the best setting found, we then performed a separate grid search PLR’s replay rate p in { 0 . 5 , 0 .
95 } and replay buffer size in { 4000 , 5000 , 8000 } , evaluating settings based on evaluation levels sampled via domain randomization, after 50M steps of training.
Our choice of hyperparameters is summarized in Table
[Table]
[Table]
B SAMPLR I MPLEMENTATION D ETAILS
As the aleatoric parameter θ (cid:48) in MultiRoom-BinaryChoice-N8-Random corresponds to a Bernoulli determining whether the goal is to eat the apple, its value does not impact transition dynamics except potentially at the ﬁnal time step, in which it determines the return.
We can therefore implement
SAMPLR’s ﬁctitious transition by simply setting Θ (cid:48)
= θ (cid:48) ∼ P ( θ (cid:48) )
at the beginning of each episode.
In our experiments, LP-SAMPLR learns a belief prior over the Bernoulli parameter q , corresponding to the probability that the correct goal for a level is apple (or if the armor is ﬁreproof), that is P
( A ) =
q , and consequently, the probability the goal is banana is P ( B ) = 1 − q .
We implement posterior inference by assuming a prior Beta (1 , 1) over whether the correct goal is apple or banana.
Given N A apple levels and N B banana levels thus far sampled from the ground-truth distribution during the nonreplay training episodes, the posterior predictive is then P ( A ) =
( N A + 1) /
(cid:16)(cid:80)
i ∈{ A,B } N i + 1 (cid:17) .
As seen in Figure 7, LP-SAMPLR quickly converges to the approximate ground-truth distribution over the correct goal, where here, q = 0 .
[Figure]