Grounding Aleatoric Uncertainty in Unsupervised Environment Design
Anonymous Author(s)
AffiliationAddress email
Abstract
Adaptive curricula in reinforcement learning (RL) have proven effective forproducing policies robust to discrepancies between the train and test environment.
Recently, the Unsupervised Environment Design (UED) framework generalized RLcurricula to generating sequences of entire environments, leading to new methodswith robust minimax regret properties.
Problematically, in partially-observable orstochastic settings, optimal policies may depend on the ground-truth distributionover aleatoric parameters of the environment in the intended deployment setting,while curriculum learning necessarily shifts the training distribution.
We formalizethis phenomenon as curriculum-induced covariate shift (CICS), and describe howits occurrence in aleatoric parameters can lead to suboptimal policies.
Directlysampling these parameters from the ground-truth distribution avoids the issue, butthwarts curriculum learning.
We propose SAMPLR, a minimax regret UED methodthat optimizes the ground-truth utility function, even when the underlying trainingdata is biased due to CICS.
We prove, and validate on challenging domains, that ourapproach preserves optimality under the ground-truth distribution, while promotingrobustness across the full range of environment settings.
Introduction
Adaptive curricula, which dynamically adjust the distribution of training environments to bestfacilitate learning, have played a key role in many recent achievements in deep reinforcement learning(RL).
Applications have spanned both single-agent RL [29, 47, 52, 20] where adaptation occurs overenvironment variations and multi-agent RL, where adaptation can additionally occur over co-players[37, 46, 38].
These methods demonstrably improve the sample efficiency and robustness of the finalpolicy [23, 7, 19, 18], e.g. by presenting the agent with challenges at the threshold of its abilities.
In this paper we introduce and address a fundamental problem relevant to adaptive curriculum learningmethods for RL, which we call curriculum-induced covariate shift (CICS).
Analogous to the covariateshift that occurs in supervised learning (SL) [16], CICS refers to a mismatch between the inputdistribution at training and test time.
In the case of RL, we will show this becomes problematic whenthe shift occurs over the aleatoric parameters of the environment—those aspects of the environmentholding irreducible uncertainty even in the limit of infinite experiential data [8].
While in some cases,
CICS may impact model performance in SL, adaptive curricula for SL have generally not been foundto be as impactful as in RL [49].
Therefore, we focus on addressing CICS specifically as it arises inthe RL setting, leaving investigation of its potential impact in SL to future work.
To establish precise language around adaptive curricula, we cast our discussion under the lens of
Unsupervised Environment Design [UED, 7].
UED provides a formal problem description for whichan optimal curriculum is the solution, by defining the Underspecified POMDP (UPOMDP see
Section 2), which expands the classic POMDP with a set of free parameters Θ , representing the
[Figure]
aspects of the environment that may vary.
UED then seeks to adapt distributions over Θ to maximizesome objective, potentially tied to the agent’s performance.
UED allows us to view adaptive curriculaas emerging via a multi-player game between a teacher that proposes environments with parametersθ ∼ P (Θ) and a student that learns to solve them.
In addition to notational clarity, this formalismenables using game theoretic constructs, such as Nash equilibria [NE, 24], to analyze curricula.
This game-theoretic view has led to the development of curriculum methods with principled robustnessguarantees, such as PAIRED [7] and Robust Prioritized Level Replay [PLR ⊥ , 18], which aim tomaximize a student’s regret and lead to minimax regret [34] policies at NE.
Thus, at NE, the studentcan solve all solvable environments within the training domain.
However, in their current form the
UED robustness guarantees are misleading: if the UED curriculum deviates from a ground-truthdistribution P (Θ) of interest, i.e. the distribution at deployment, with respect to aleatoric parametersΘ ′
⊂ Θ , the resulting policies may be suboptimal under the ground-truth distribution P .
For a concrete example of how CICS can be problematic, consider the case of training a self-drivingcar to navigate potentially icy roads, when icy conditions rarely occur under P .
When present, theice is typically hard to spot in advance; thus, the aleatoric parameters Θ ′ correspond to whether eachsection of the road is icy.
A priori, a curriculum should selectively sample more challenging icysettings to facilitate the agent’s mastery over such conditions.
However, this approach risks producingan overly-pessimistic agent (i.e. one that assumes that ice is common), driving slowly even in fairweather.
Such a policy leads to inadequate performance on P , which features ice only rarely.
We can preserve optimality on P by grounding the policy —that is, ensuring that the agent actsoptimally with respect to the ground-truth utility function for any action-observation history τ andthe implied ground-truth posterior over Θ : [Equation]
[Equation]
) where the ground-truth utility, conditioned on X , U ( π | X ) , is defined to be E τ,θ ∼ P ( θ | X ) [ (cid:80) ∞ t [-=
-]=0 γ t r t ] ,for rewards r t and a discount γ .
We can ground the policy by grounding the training distribution , which means constraining thetraining distribution of aleatoric parameters P (Θ ′ ) to match P (Θ ′ ) .
This is trivially accomplished bydirectly sampling θ ′ ∼ P (Θ ′ ) , which we call naive grounding .
Unfortunately, this approach makesmany curricula infeasible by removing the ability to selectively sample environment settings overaleatoric parameters.
Applying this strategy to the self-driving agent may result in a policy that isoptimal in expectation under P where there is rarely ice, but nevertheless fails to drive safely on ice.
We wish to maintain the ability to bias a training distribution, since it is required for curriculumlearning, while ensuring the resulting decisions remain optimal in expectation under P .
This goal iscaptured by the following objective:
[Equation]
where D is the training distribution of τ .
Under naive grounding, D is equal to P ( τ ) and Equation2 reduces to U ( π ) .
To overcome the limitations of naive grounding, we develop an approach thatallows D to deviate from P ( τ ) , e.g. by prioritizing levels most useful for learning, but still groundsthe policy by evaluating decisions following potentially biased training trajectories τ according to
U ( π | τ ) .
Figure 1 summarizes this approach, and contrasts it with an ungrounded adaptive curriculum.
In summary this work presents the following contributions: i)
We first formalize the problem of CICSin RL in Section 3. ii)
Then, we present SAMPLR, which extends PLR ⊥ , a state-of-the-art UEDmethod, to preserve optimality on P while training under a usefully biased training distribution in
Section 4.
iii)
We prove in Section 5 that SAMPLR promotes Bayes-optimal policies that are robustover all environment settings θ ∼ P (Θ) .
iv)
Our experiments validate these conclusions in twochallenging domains, where SAMPLR learns highly robust policies while PLR ⊥ fails due to CICS.
Background2.1 Unsupervised Environment Design
Unsupervised Environment Design [UED, 7] is the problem of automatically generating an adaptivedistribution of environments which will lead to policies that successfully transfer within a targetdomain.
The domain of possible environment settings is represented by an Underspecified POMDP(UPOMDP). This models each environment instantiation, or level , as a specific setting of the freeparameters that control how the environment varies across instances.
Examples of free parameters arethe position of walls in a maze, or friction coefficients in a physics-based task.
Formally a UPOMDPis defined as a tuple M = ⟨ A, O, Θ , S , T , I , R , γ ⟩ , where A is the action space, O is the observationspace, Θ is the set of free parameters, S is the state space, T : S ×
A × Θ → ∆ ( S ) is the transitionfunction, I : S
→ O is the observation function, R :
S
→ R is the reward function, and γ is thediscount factor.
UED typically approaches the curriculum design problem as training a teacher agentthat co-evolves an adversarial curriculum for a student agent, e.g. by maximizing the student’s regret.
2.2 Prioritized Level Replay
We focus on a recent UED algorithm called Robust Prioritized Level Replay [PLR ⊥ , 19], whichperforms environment design via random search.
PLR ⊥ maintains a buffer of the most useful levelsfor training, according to an estimate of learning potential—typically based on regret, approximatedby a function of the temporal-difference (TD) errors incurred on each level.
For each episode, withprobability p , PLR ⊥ actively samples the next training level from this buffer, and otherwise evaluatesregret on a new level θ ∼ P (Θ) without training.
This sampling mechanism provably leads to aminimax regret policy for the student at NE, and has been shown to improve sample-efficiency andgeneralization.
The resultant regret-maximizing curricula naturally avoid unsolvable levels, whichhave no regret.
We provide implementation details for PLR ⊥ in Appendix A.
Curriculum-Induced Covariate Shift
Since UED algorithms formulate curriculum learning as a multi-agent game between teacher andstudent agents, we can formalize when CICS becomes problematic by considering the equilibriumpoint of this game: Let Θ be the environment parameters controlled by UED, P (Θ) , the groundtruth distribution, and P (Θ) , the curriculum distribution at equilibrium.
Further, let τ Ot and τ At be( o 1 , ..., o t ) and ( a 1 , ..., a t − 1 ) respectively, that is, the student agent’s action and observation historiesuntil time t .
We will use τ t = τ Ot × τ At to refer to the joint action-observation history (AOH) of thestudent (and simply τ when clear from context).
Letting V ( π | τ t ) denote the value function underthe curriculum distribution P (Θ) , we characterize an instance of CICS over Θ as problematic if theoptimal policy under P (Θ) differs from that under the ground-truth P (Θ) for some τ t , so that:
[Equation]
The value function V ( π | τ t ) with respect to P (Θ) can be expressed as a marginalization over θ :
[Equation]
Here, the notation P ( θ ) refers to P (Θ = θ ) .
Importantly, V ( π | τ t ) corresponds to Equation 3 with P replaced by P .
We see that V ( π | τ t ) remains optimal for each history τ t when P ( θ ) is replacedby different choices of P ( θ ) as long as P ( θ ∗ | τ t )
= 1 for some θ ∗ and P shares support with P , asthen P ( τ Ot | θ, τ
At ) = 1 if θ = θ ∗ and zero elsewhere.
In this case, the sums reduce to V ( π | τ t , θ ∗ ) ,regardless of changing P to P .
In other words, when Θ is fully determined given the current historyτ , covariate shifts over Θ with respect to P (Θ) have no impact on policy evaluation and thus thevalue function for the optimal policy.
If this condition does not hold, the uncertainty over the value ofsome subset Θ ′ ⊂
Θ is irreducible given τ , making Θ ′ aleatoric parameters for the history τ .
Thus,assuming the curriculum shares support with the ground-truth distribution, covariate shifts can onlyalter the optimal policy at τ when they occur over aleatoric parameters given τ .
Such parameters canarise when the environment is inherently stochastic or when the cost of reducing uncertainty is high.
Crucially, our analysis assumes P and P share support over Θ .
When this assumption is broken,the policy trained under the curriculum can be suboptimal for environment settings θ , for which
P ( θ ) = 0 and P ( θ ) > 0 .
In this paper, we specifically assume that P and P share support, and focuson addressing suboptimality under the ground-truth P due to CICS over the aleatoric parameters Θ ′ .
This discussion thus makes clear that problematic CICS can be resolved by enforcing the constraint P (Θ ′ | τ ) =
P (Θ ′ | τ ) for the aleatoric parameters of the environment.
As discussed, naive groundingsatisfies this constraint by directly sampling θ ′ ∼ P (Θ ′ )
, at the cost of curricula over Θ ′ .
This workdevelops an alternative for satisfying this constraint while admitting curricula over Θ ′ .
Sample-Matched PLR (SAMPLR)
Randomly initialize policy π ( ϕ ) , an empty level buffer Λ of size K , and belief model B ( s t | τ ) . while not converged do
Sample replay-decision Bernoulli, d ∼ P D ( d ) if d = 0 or | Λ | = 0 then
Sample level θ from level generator Collect π ’s trajectory τ on θ , with a stop-gradient ϕ ⊥ else
Use PLR to sample a replay level from the level store, θ ∼ Λ
Collect fictitious trajectory τ ′ on θ , based on s ′ t ∼ B
Update π with rewards R ( τ ′ ) end
Compute PLR score, S = score ( τ ′ , π )
Update Λ with θ using score S end
We now describe a general strategyfor addressing CICS, and apply it to
PLR ⊥ , resulting in Sample-Matched PLR(SAMPLR).
This new UED method featuresthe robustness properties of PLR ⊥ whilemitigating the potentially harmful effects of
CICS over the aleatoric parameters Θ ′ .
As discussed in Section 3, CICS becomeproblematic when the covariate shift occursover some aleatoric subset Θ ′ of theenvironment parameters Θ , such that theexpectation over Θ ′ influences the optimalpolicy.
Adaptive curriculum methods like
PLR ⊥ prioritize sampling of environmentsettings where the agent experiences the mostlearning.
While such a curriculum lets theagent focus on correcting its largest errors, thecurriculum typically changes the distributionover aleatoric parameters Θ ′ , inducing bias inthe resulting decisions.
Ideally, we can eliminate this bias, ensuring the resulting policy makes optimaldecisions with respect to the ground-truth utility function, conditioned on the current trajectory:
[Equation]
A naive solution for grounding is to simply exclude Θ ′ from the set of environment parameters undercurriculum control.
That is, for each environment setting proposed by the curriculum, we resampleθ ′ ∼ P .
We refer to this approach as naive grounding .
Naive grounding forces the expected reward and next state under each transition at the current AOH τ to match that under P .
Thus, optimalpolicies under naive grounding must be optimal with respect to the ground-truth distribution over θ ′ .
While technically simple, naive grounding suffers from lack of control over Θ ′ .
This limitation is ofno concern when the value of Θ ′ does not alter the distribution of τ until the terminal transition, e.g.when Θ ′ is the correct choice in a binary choice task, thereby only influencing the final, sparse rewardwhen the right choice is made.
In fact, our initial experiment in Section 6 shows naive groundingperforms well in such cases.
However, when the value of Θ ′ changes the distribution of τ beforethe terminal transition, the agent may benefit from a curriculum that actively samples levels whichpromote learning robust behaviors under unlikely events.
Enabling the full benefits of the curriculumin such cases requires the curriculum to selectively sample values of Θ ′ .
Instead of naive grounding, we aim to ground only the policy updates, allowing the curriculum tobias the training distribution.
This can be accomplished by optimizing the following objective:
[Equation]
To achieve this we directly replace the reward r t and next state s t +1 with counterfactual values thatwould be experienced if θ ′ were consistent with τ and P , so that θ ′ ∼ P ( θ ′ | τ ) .
This substitutionoccurs by simulating a fictitious transition , where the fictitious state is sampled as s ′ t ∼ B ( s ′ t | τ ) ,the action as a t ∼ π ( ·| τ ) (as per usual), the fictitious next state as s ′ t +
=
T ( s ′ t , a t ) , and thefictitious reward as r ′ t =
R ( s ′ t +1 ) .
Here, the belief model B ( s ′ t | τ ) provides the ground-truth posteriordistribution of the current state given τ :
[Equation]
Fictitious transitions, summarized in Figure 2, ground the observed rewards and state transitions to P .
Should training on these transitions lead to an optimal policy over Θ , this policy will also be optimalwith respect to P .
We prove this property in Section 5.
Fictitious transitions thus provide the benefitof naive grounding without giving up curriculum control over Θ ′ .
In general, we implement B as follows:
Given P (Θ ′ ) as a prior, wemodel the posterior P ( θ ′ | τ ) with Bayesian inference.
The posteriorcould be learned via supervised learning with trajectories collectedfrom the environment for a representative selection of θ ′ .
Further,we may only have limited access to P (Θ) throughout training, forexample, if sampling P (Θ) is costly.
In this case, we can learn anestimate ˆ P (Θ ′ ) from samples we do collect from P (Θ) , which canoccur online.
We can then use ˆ P (Θ ′ ) to inform the belief model.
SAMPLR, summarized in Algorithm 1, incorporates this fictitioustransition into PLR ⊥ by replacing the transitions experienced inreplay levels sampled by PLR ⊥ with their fictitious counterparts,as PLR ⊥ only trains on these trajectories.
PLR ⊥ uses PPO withthe Generalized Advantage Estimator [GAE, 35] as the base RLalgorithm, where both advantage estimates and value losses can bewritten in terms of one-step TD errors δ t at time t .
Training onfictitious transitions then amounts to computing these TD errorswith fictitious states and rewards: δ t = r ′ t + V ( s ′ t ) − V ( s ′ t +1 ) .
Importantly, because PLR ⊥ provably leads to policies that minimize worst-case regret over all θ at
NE, SAMPLR enjoys the same property for θ ∼ P (Θ) .
A proof of this fact is provided in Section 5.
Applying SAMPLR requires two key assumptions:
First, the simulator can be reset to a specific state,which is often true, as RL largely occurs in resettable simulators or those that can be made to do so.
Second, we have knowledge of P (Θ ′ ) .
Often, we know P a priori , e.g. via historical data or as partof the domain specification, as in games of chance. Further, we can often estimate P empirically.
The Grounded Optimality of SAMPLR
Training on fictitious transitions is a method for learning an optimal policy with respect to the groundtruth utility function U D ( π ) over the distribution D of training trajectories τ , defined in Equation 5.
When D corresponds to the distribution of trajectories on levels θ ∼ P (Θ) , U D ( π ) reduces to theground-truth utility function, U ( π ) .
For any UED method, our approach ensures that, in equilibrium,the resulting policy is Bayes-optimal with respect to P (Θ) for all trajectories in the support of D .
Remark 1.
If π ∗ is optimal with respect to the ground-truth utility function U D ( π ) then it is optimalwith respect to the ground-truth distribution P (Θ) of environment parameters on the support of D .
Proof.
By definition we have π ∗ ∈ arg max π ∈ Π { U D ( π ) }
= arg max π ∈ Π
{ E τ ∼D (cid:2) U
( π | τ ) (cid:3) } .
Since π cancondition on the initial trajectory τ , the action selected after each trajectory can be independentlyoptimized.
Thus we have, for all τ ∈ D , π ∗ ∈ arg max π ∈ Π { U ( π | τ ) } implying that π ∗ is the optimalpolicy maximizing U ( π | τ ) .
Thus, assuming the base RL algorithm finds Bayes-optimal policies, a UED method that optimizesthe ground-truth utility function, as done by SAMPLR, results in Bayes-optimal performance overthe ground-truth distribution.
If the UED method maximizes worst-case regret, we can prove an evenstronger property we call robust ϵ -Bayes optimality .
Let U θ ( π ) be the ground-truth utility function for π on the distribution D πθ of initial trajectoriessampled from level θ , so that U θ ( π ) =
U D πθ ( π ) .
Given a policy π maximizing U θ ( π ) , we say that πis robustly ϵ -Bayes optimal iff for all θ in the domain of P (Θ) and all π ′ , we have
U θ ( π ) ≥ U θ ( π ′ )
− ϵ.
Note how this property differs from being simply ϵ -Bayes optimal, which would only imply that
U ( π ) ≥ U ( π ′ )
− ϵ.
Robust ϵ -Bayes optimality requires π to be ϵ -optimal on all levels θ in the support of the groundtruth distribution, even those rarely sampled under P (Θ) .
We will show that at ϵ -Nash equilibrium,
SAMPLR results in a robustly ϵ -Bayes optimal policy for the ground-truth utility function U θ ( π ) .
Incontrast, training directly on levels θ ∼ P (Θ) results a policy that is only ϵ -Bayes optimal.
Theorem 1.
If π ∗ is ϵ -Bayes optimal with respect to U (cid:98) D ( π ) for the distribution (cid:98) D of trajectoriessampled under π over levels maximizing the worst-case regret of π , as occurs under SAMPLR, thenπ ∗ is robustly ϵ -Bayes optimal with respect to the ground-truth utility function, U ( π ) .
Proof.
Let π ∗ be ϵ -optimal with respect to U (cid:98) D ( π ) where (cid:98) D is the trajectory distribution under π onlevels maximizing the worst-case regret of π .
Let π ∗ be an optimal grounded policy.
Then for any θ ,
[Equation]
The first inequality follows from (cid:98) D being trajectories from levels that maximize worst-case regretwith respect to π ∗ , and the second follows from π ∗ being ϵ -optimal on U (cid:98) D ( π ) .
Rearranging termsgives the desired condition.
Experiments
Our experiments first focus on a discrete, stochastic binary choice task, with which we validate ourtheoretical conclusions by demonstrating that CICS can indeed lead to suboptimal policies.
Moreover,we show that naive grounding suffices for learning robustly optimal policies in this setting.
However,as we have argued, naive grounding gives up control of the aleatoric parameters Θ ′ and thus lacksthe ability to actively sample scenarios helpful for learning robust behaviors—especially importantwhen such scenarios are infrequent under the ground-truth distribution P (Θ) .
SAMPLR inducespotentially biased curricula, but retains optimality under P (Θ) by matching transitions under P (Θ ′ )with those under P (Θ ′ ) .
We assess the effectiveness of this approach in our second experimentaldomain, based on the introductory example of driving icy roads.
In this continuous-control drivingdomain, we seek to validate whether SAMPLR does in fact learn more robust policies that transfer totail cases under P (Θ ′ ) , while retaining high expected performance over the entire P (Θ ′ ) .
All agents were trained using PPO [36] with the best hyperparameters found via grid search using aset of validation levels.
We provide extended descriptions of both environments alongside the fulldetails of our architecture and hyperparameter choices in Appendix C.
[Figure]
6.1 Stochastic Fruit Choice
We aim to demonstrate the phenomenon of CICS in Stochastic Fruit Choice, a binary choice task,where the aleatoric parameter determines the correct choice.
This task requires the agent to traverse upto eight rooms, and in the final room, decide to eat either the apple or banana.
The correct choice θ ′ isfixed for each level, but hidden from the agent.
Optimal decision-making depends on the ground-truthdistribution over the correct fruit, P (Θ ′ ) .
This task benefits from a curriculum over the number ofrooms, but a curriculum that selectively samples over both room layout and correct fruit choice canlead to suboptimal policies.
Figure 3 shows example levels from this environment.
This domain presents a hard exploration challenge for RL agents, requiring robust navigation acrossmultiple rooms.
Further, this environment is built on top of MiniHack [33], enabling integration ofselect game dynamics from the NetHack Learning Environment [21], which the agent must master tosucceed:
To go from one room to the next, the agent needs to learn to kick the locked door until itopens.
Upon reaching the final room, the agent must then apply the eat action on the correct fruit.
Let π A be the policy that always chooses the apple, and π B , the banana.
If the probability that thegoal is the apple is P ( A ) =
q , then the expected return is R A q under π A and R B (1 − q ) under π B .
The optimal policy is π A when q > R B / ( R A + R B ) , and π B otherwise.
Domain randomization(DR), which directly samples each level θ ∼ P ( θ ) , optimizes for the correct ground-truth P (Θ ′ ) , butwill predictably struggle to solve the exploration challenge.
PLR ⊥ may induce curricula easing theexploration problem, but can be expected make the correct fruit choice oscillate throughout trainingto maximize regret, leading to problematic CICS.
We set R A = 3 , R B = 10 , and q = 0 .
7 , making π B optimal with an expected return of 3 . 0 .
Wecompare the train and test performance of agents trained with DR, PLR ⊥ , and PLR ⊥ with naivegrounding over 200M training steps in Figure 3.
In this domain, SAMPLR reduces to naive grounding,as θ ′ only effects the reward of a terminal transition, making fictitious transitions equivalent to realtransitions for all intermediate time steps.
We see that DR struggles to learn an effective policy,plateauing at a mean return around 1 . 0 , while PLR ⊥ performs the worst.
Figure 6 in Appendix Bshows that the PLR ⊥ curriculum exhibits much higher variance in q , rapidly switching the optimalchoice of fruit to satisfy its regret-maximizing incentive, making learning more difficult.
In contrast,
PLR ⊥ with naive grounding constrains q = 0 .
7 , while still exploiting a curriculum over an increasingnumber of rooms, as visible in Figure 6.
This grounded curriculum results in a policy that solvesmore complex room layouts at test time.
Figures 5 and 6 in Appendix B additionally show how the
SAMPLR agent’s choices converge to π B and how the size of SAMPLR’s improvement varies underalternative choices of q in { 0 . 5 , 0 . 3 } .
6.2 Zero-Shot Driving Formula 1 Tracks with Black Ice
We now turn to a domain where the aleatoric parameters influence the distribution of τ t at each t ,thereby creating opportunities for a curriculum to actively sample specific θ ′ to promote learning onbiased distributions of τ t .
We base this domain on the black ice driving scenario from the introductionof this paper, by modifying the CarRacingBezier environment in [18].
In our version, each track tilehas black ice with probability q , in which case its friction coefficient is 0, making acceleration and
[Figure]
t
[Figure]
r
[Figure]
grounding
SAMPLR braking impossible.
This task especially difficult, since the agent cannot see black ice in its pixelobservations.
Figure 4 shows example tracks with ice rendered for illustration purposes.
The episodicreturns scale proportionately with how much of the track is successfully driven and how quickly thisis accomplished.
As success requires learning to navigate the challenging dynamics over ice patches,a curriculum targeting more difficult ice configurations should lead to policies more robust to blackice.
Here, the ground-truth distribution P ( θ ) models the realistic assumption that most days see littleto no ice.
We therefore model the probability of ice per tile as q ∼ Beta ( α, β ) , where α = 1 , β = 15 .
[Table]
We test the hypothesis that SAMPLR’sregret-maximizing curriculum results inpolicies that preserve optimal performanceon the ground-truth distribution P ( θ ′ ) ,while being more robust to tail casescompared to DR and PLR ⊥ with naivegrounding.
We expect standard PLR ⊥ tounderperform all methods due to CICS,leading to policies that are either toopessimistic or too optimistic with respectto the amount of ice.
These baselines provide the controls needed to distinguish performance changesdue to the two grounding approaches and those due to the underlying curriculum learning method.
We train agents using each method for 5M steps and test zero-shot generalization performance on the
Formula 1 tracks taken from the CarRacingF1 benchmark, extended to allow for each track segmentto have black ice with probability q in { 0 . 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } .
These test tracks are significantlylonger and more complex than those seen at training, as well as having a higher rate of black ice.
To implement SAMPLR’s belief model, we use a second simulator as a perfect model of theenvironment.
At each time step, this second simulator, which we refer to as the fictitious simulator ,resets to the exact physics state of the primary simulator, and its icy tiles are resampled according tothe exact posterior over the aleatoric parameter q = θ ′ , such that θ ′ ∼ P
θ ′ | τ ) , ensuring the futureuncertainty is consistent with the past.
The agent decides on action a t based on the current realobservation o t , and observes the fictitious return r ′ t and next state s ′ t +1 determined by the fictitioussimulator after applying a t in s ′ t ∼ P ( s ′ t | τ, θ ′ ) .
This dual simulator arrangement, fully detailed in Appendix A.2, allows us to measure the impact of training on fictitious transitions independently ofthe efficacy of a model-based RL approach.
Further, as the training environment in RL is most oftensimulation (e.g. in sim2real), this approach is widely applicable.
SAMPLR outperforms all baselines in zero-shot transfer to higher ice rates on the full F1 benchmarkand attains a statistically significant improvement at p < 0 .
001 when transferring to q = 0 .
4 andq = 0 . 6 , and p < 0 .
05 when q = 0 . 8 .
Importantly, SAMPLR outperforms PLR ⊥ with naivegrounding, indicating that SAMPLR exploits specific settings of Θ ′ to better robustify the agentagainst rare icy conditions in the tail of P (Θ) .
Indeed, Figure 4 shows that on average, SAMPLRexposes the agent to more ice per track tile driven, while PLR ⊥ underexposes the agent to icecompared to DR and naive grounding, suggesting that under PLR ⊥ agents attain higher regret onice-free tracks—a likely outcome as ice-free tracks are easier to drive and lead to returns, with which regret scales.
Unfortunately, this results in PLR ⊥ being the worst out of all methods on theground-truth distribution.
SAMPLR and naive grounding avoid this issue by explicitly matchingtransitions to those under P at τ .
As reported in Table 1, SAMPLR matches the performance ofthe baselines on P ( θ ) , indicating its active sampling of challenging θ ′ preserves performance under P (Θ) , i.e. the agent does not become overly cautious.
Related Work
The mismatch between training and testing distributions of input features is referred to as covariateshift , and has long served as a fundamental problem for the machine learning community.
Covariateshifts have been extensively studied in supervised learning [45, 16, 4, 2].
In RL, prior works havelargely focused on covariate shifts due to training on off-policy data [41, 31, 10, 13, 12, 43] includingthe important case of learning from demonstrations [28, 30].
Recent work also aimed to learn invariantrepresentations robust to covariate shifts [50, 51].
More generally, CICS is a form of sample-selectionbias [14].
Previous methods like OFFER [6] considered correcting biased transitions via importancesampling [40] when optimizing for expected return on a single environment setting, rather than robustpolicies over all environments settings.
We believe our work provides the first general formalizationand solution strategy addressing curriculum-induced covariate shifts (CICS) for RL.
The importance of addressing CICS is highlighted by recent results showing curricula to be essentialfor training RL agents across many of the most challenging domains, including combinatorialgridworlds [52], Go [37], StarCraft 2 [46], and achieving comprehensive task mastery in open-endedenvironments [38].
While this work focuses on PLR ⊥ , other methods include minimax adversarialcurricula [47, 48] and curricula based on changes in return [23, 29].
Curriculum methods have alsobeen studied in goal-conditioned RL [11, 5, 39, 25], though CICS does not occur here as goals areobserved by the agent.
Lastly, domain randomization [DR, 32, 27] can be seen as a degenerate formof UED, though curriculum-based extensions of DR have also been studied [17, 44].
Prior work has also investigated methods for learning Bayes optimal policies under uncertaintyabout the task [53, 26], based on the framework of Bayes-adaptive MDPs (BAMDPs)
Inthis setting, the agent can adapt to an unknown MDP over several episodes by acting to reduce itsuncertainty about the identity of the MDP.
In contrast, SAMPLR learns a robustly Bayes-optimalpolicy for zero-shot transfer.
Further unlike these works, our setting assumes the distribution of somealeatoric parameters is biased during training, which would bias the a posteriori uncertainty estimateswith respect to the ground-truth distribution when optimizing for the BAMDP objective.
Instead,
SAMPLR proposes a means to correct for this bias assuming knowledge of the true environmentparameters, to which we can safely assume access in curriculum learning.
Deeply related, Off-Belief Learning [OBL, 15] trains cooperative agents in self-play using fictitioustransitions assuming the action histories of co-players follow a base policy, e.g. a uniformly randomone.
Enforcing this assumption prevents the agents from developing conventions that communicateprivate information to co-players via arbitrary action sequences.
Such conventions hinder coordinationwith independently trained agents or, importantly, humans.
SAMPLR can be viewed as adapting OBL to single-agent curriculum learning, where a co-player sets the environment parameters at thestart of each episode.
This connection highlights how single-agent curriculum learning is inherently amulti-agent problem, and therefore problems afflicting multi-agent learning will also surface in thissetting; moreover, methods that address such issues in one setting can then be adapted to the other.
Conclusion
This work characterized how CICS over aleatoric environment parameters Θ ′ can lead to suboptimalpolicies with respect to the ground truth distribution over environment settings, P (Θ) .
We introduceda general strategy for correcting such covariate shifts, by training the agent on fictitious rewardsand next states whose distribution is guaranteed to match what would be experienced under theground-truth distribution P (Θ ′ ) .
Our method SAMPLR augments PLR ⊥ with this correction.
Bytraining on fictitious transitions, SAMPLR can actively sample specific values of θ ′ that inducetrajectories with greater learning potential, while still grounding the training data to P ( θ ′ ) .
Crucially,our experiments showed that SAMPLR outperforms both PLR ⊥ and PLR ⊥ with naive grounding,learning highly performant policies where the competing baselines fail.