Grounding Aleatoric Uncertainty in Unsupervised Environment Design
Anonymous Author(s)
AffiliationAddress email
Abstract
Adaptive curricula in reinforcement learning (RL) have proven effective forproducing policies robust to discrepancies between the train and test environment.
Recently, the Unsupervised Environment Design (UED) framework generalized RLcurricula to generating sequences of entire environments, leading to new methodswith robust minimax regret properties.
Problematically, in partially-observable orstochastic settings, optimal policies may depend on the ground-truth distributionover aleatoric parameters of the environment in the intended deployment setting,while curriculum learning necessarily shifts the training distribution.
We formalizethis phenomenon as curriculum-induced covariate shift (CICS), and describe howits occurrence in aleatoric parameters can lead to suboptimal policies.
Directlysampling these parameters from the ground-truth distribution avoids the issue, butthwarts curriculum learning.
We propose SAMPLR, a minimax regret UED methodthat optimizes the ground-truth utility function, even when the underlying trainingdata is biased due to CICS.
We prove, and validate on challenging domains, that ourapproach preserves optimality under the ground-truth distribution, while promotingrobustness across the full range of environment settings.
Introduction
Adaptive curricula, which dynamically adjust the distribution of training environments to bestfacilitate learning, have played a key role in many recent achievements in deep reinforcement learning(RL).
Applications have spanned both single-agent RL [32, 50, 55, 22] where adaptation occurs overenvironment variations and multi-agent RL, where adaptation can additionally occur over co-players[40, 49, 41].
These methods demonstrably improve the sample efficiency and robustness of the finalpolicy [25, 8, 21, 20], e.g. by presenting the agent with challenges at the threshold of its abilities.
In this paper we introduce and address a fundamental problem relevant to adaptive curriculum learningmethods for RL, which we call curriculum-induced covariate shift (CICS).
Analogous to the covariateshift that occurs in supervised learning (SL) [18], CICS refers to a mismatch between the inputdistribution at training and test time.
In the case of RL, we will show this becomes problematic whenthe shift occurs over the aleatoric parameters of the environment—those aspects of the environmentholding irreducible uncertainty even in the limit of infinite experiential data [9].
While in some cases,
CICS may impact model performance in SL, adaptive curricula for SL have generally not been foundto be as impactful as in RL [52].
Therefore, we focus on addressing CICS specifically as it arises inthe RL setting, leaving investigation of its potential impact in SL to future work.
To establish precise language around adaptive curricula, we cast our discussion under the lens of
Unsupervised Environment Design [UED, 8].
UED provides a formal problem description for whichan optimal curriculum is the solution, by defining the Underspecified POMDP (UPOMDP see
Section 2), which expands the classic POMDP with a set of free parameters Θ , representing the
[Figure]
aspects of the environment that may vary.
UED then seeks to adapt distributions over Θ to maximizesome objective, potentially tied to the agent’s performance.
UED allows us to view adaptive curriculaas emerging via a multi-player game between a teacher that proposes environments with parametersθ ∼ P (Θ) and a student that learns to solve them.
In addition to notational clarity, this formalismenables using game theoretic constructs, such as Nash equilibria [NE, 26], to analyze curricula.
This game-theoretic view has led to the development of curriculum methods with principled robustnessguarantees, such as PAIRED [8] and Robust Prioritized Level Replay [PLR ⊥ , 20], which aim tomaximize a student’s regret and lead to minimax regret [37] policies at NE.
Thus, at NE, the studentcan solve all solvable environments within the training domain.
However, in their current form the
UED robustness guarantees are misleading: if the UED curriculum deviates from a ground-truthdistribution P (Θ) of interest, i.e. the distribution at deployment, with respect to aleatoric parametersΘ ′
⊂ Θ , the resulting policies may be suboptimal under the ground-truth distribution P .
For a concrete example of how CICS can be problematic, consider the case of training a self-drivingcar to navigate potentially icy roads, when icy conditions rarely occur under P .
When present, theice is typically hard to spot in advance; thus, the aleatoric parameters Θ ′ correspond to whether eachsection of the road is icy.
A priori, a curriculum should selectively sample more challenging icysettings to facilitate the agent’s mastery over such conditions.
However, this approach risks producingan overly-pessimistic agent (i.e. one that assumes that ice is common), driving slowly even in fairweather.
Such a policy leads to inadequate performance on P , which features ice only rarely.
We can preserve optimality on P by grounding the policy —that is, ensuring that the agent actsoptimally with respect to the ground-truth utility function for any action-observation history τ andthe implied ground-truth posterior over Θ :
[Equation]
where the ground-truth utility conditioned on X , U ( π | X ) , is defined to be E τ,θ ∼ P ( θ | X ) [ (cid:80) ∞ t {+=
+} γ t r t ] ,for rewards r t and a discount γ .
We can ground the policy by grounding the training distribution , which means constraining thetraining distribution of aleatoric parameters P (Θ ′ ) to match P (Θ ′ ) .
This is trivially accomplished bydirectly sampling θ ′ ∼ P (Θ ′ ) , which we call naive grounding .
Unfortunately, this approach makesmany curricula infeasible by removing the ability to selectively sample environment settings overaleatoric parameters.
Applying this strategy to the self-driving agent may result in a policy that isoptimal in expectation under P where there is rarely ice, but nevertheless fails to drive safely on ice.
We wish to maintain the ability to bias a training distribution, since it is required for curriculumlearning, while ensuring the resulting decisions remain optimal in expectation under P .
This goal iscaptured by the following objective:
[Equation]
where D is the training distribution of τ .
Under naive grounding, D is equal to P ( τ ) and Equation2 reduces to U ( π ) .
To overcome the limitations of naive grounding, we develop an approach thatallows D to deviate from P ( τ ) , e.g. by prioritizing levels most useful for learning, but still groundsthe policy by evaluating decisions following potentially biased training trajectories τ according to
U ( π | τ ) .
Figure 1 summarizes this approach, and contrasts it with an ungrounded adaptive curriculum.
In summary this work presents the following contributions: i)
We first formalize the problem of CICSin RL in Section 3. ii)
Then, we present SAMPLR, which extends PLR ⊥ , a state-of-the-art UEDmethod, to preserve optimality on P while training under a usefully biased training distribution in
Section 4.
iii)
We prove in Section 5 that SAMPLR promotes Bayes-optimal policies that are robustover all environment settings θ ∼ P (Θ) .
iv)
Our experiments validate these conclusions in twochallenging domains, where SAMPLR learns highly robust policies, while PLR ⊥ fails due to CICS.
Background2.1 Unsupervised Environment Design
Unsupervised Environment Design [UED, 8] is the problem of automatically generating an adaptivedistribution of environments which will lead to policies that successfully transfer within a targetdomain.
The domain of possible environment settings is represented by an Underspecified POMDP(UPOMDP), which models each environment instantiation, or level , as a specific setting of the freeparameters that control how the environment varies across instances.
Examples of free parameters arethe position of walls in a maze or friction coefficients in a physics-based task.
Formally a UPOMDPis defined as a tuple M = ⟨ A, O, Θ , S , T , I , R , γ ⟩ , where A is the action space, O is the observationspace, Θ is the set of free parameters, S is the state space, T : S ×
A × Θ → ∆ ( S ) is the transitionfunction, I : S
→ O is the observation function, R :
S
→ R is the reward function, and γ is thediscount factor.
UED typically approaches the curriculum design problem as training a teacher agentthat co-evolves an adversarial curriculum for a student agent, e.g. by maximizing the student’s regret.
2.2 Prioritized Level Replay
We focus on a recent UED algorithm called Robust Prioritized Level Replay [PLR ⊥ , 21], whichperforms environment design via random search.
PLR ⊥ maintains a buffer of the most useful levelsfor training, according to an estimate of learning potential—typically based on regret, approximatedby a function of the temporal-difference (TD) errors incurred on each level.
For each episode, withprobability p , PLR ⊥ actively samples the next training level from this buffer, and otherwise evaluatesregret on a new level θ ∼ P (Θ) without training.
This sampling mechanism provably leads to aminimax regret policy for the student at NE, and has been shown to improve sample-efficiency andgeneralization.
The resultant regret-maximizing curricula naturally avoid unsolvable levels, whichhave no regret.
We provide implementation details for PLR ⊥ in Appendix A.
Curriculum-Induced Covariate Shift
Since UED algorithms formulate curriculum learning as a multi-agent game between a teacher and astudent agent, we can formalize when CICS becomes problematic by considering the equilibriumpoint of this game: Let Θ be the environment parameters controlled by UED, P (Θ) , their groundtruth distribution, and P (Θ) , their curriculum distribution at equilibrium.
We use τ t to refer to thejoint action-observation history (AOH) of the student until time t (and simply τ when clear fromcontext).
Letting V ( π | τ t ) denote the value function under the curriculum distribution P (Θ) , wecharacterize an instance of CICS over Θ as problematic if the optimal policy under P (Θ) differsfrom that under the ground-truth P (Θ) for some τ t , so that
[Equation]
The value function V ( π | τ t ) with respect to P (Θ) can be expressed as a marginalization over θ :
[Equation]
Here, the notation P ( θ ) means P (Θ = θ ) , and the tilde on the ˜ P and ˜ V terms indicates independencefrom any distribution over Θ , as they both condition on θ .
Importantly, the value function underthe curriculum distribution V ( π | τ t ) corresponds to Equation 3 with P replaced by P .
We see that V ( π | τ t ) is unchanged for a given τ t when P ( θ ) is replaced with P ( θ ) if 1) P ( θ ∗ | τ t )
= 1 for someθ ∗ , and 2) P shares support with P . Then ˜ P ( τ t | θ )
= 1 iff θ = θ ∗ and zero elsewhere.
In this case,the sums reduce to V ( π | τ t , θ ∗ ) , regardless of changing the ground-truth distribution P to P .
In otherwords, when Θ is fully determined given the current history τ , covariate shifts over Θ with respect to P (Θ) have no impact on policy evaluation and thus the value function for the optimal policy.
If thefirst condition does not hold, the uncertainty over the value of some subset Θ ′ ⊂
Θ is irreduciblegiven τ , making Θ ′ aleatoric parameters for the history τ .
Thus, assuming the curriculum sharessupport with the ground-truth distribution, covariate shifts only alter the optimal policy at τ whenthey occur over aleatoric parameters given τ .
Such parameters can arise when the environment isinherently stochastic or when the cost of reducing uncertainty is high.
Crucially, our analysis assumes P and P share support over Θ .
When this assumption is broken,the policy trained under the curriculum can be suboptimal for environment settings θ , for which
P ( θ ) = 0 and P ( θ ) > 0 .
In this paper, we specifically assume that P and P share support and focuson addressing suboptimality under the ground-truth P due to CICS over the aleatoric parameters Θ ′ .
This discussion thus makes clear that problematic CICS can be resolved by grounding the trainingdistribution , i.e. enforcing the constraint P (Θ ′ | τ )
= P (Θ ′ | τ ) for the aleatoric parameters of theenvironment.
This constraint results in grounding the policy , i.e. ensuring it is optimal with respect tothe ground-truth utility function based on P (Equation 1).
As discussed, naive grounding satisfiesthis constraint by directly sampling θ ′ ∼ P (Θ ′ )
, at the cost of curricula over Θ ′ .
This work developsan alternative for satisfying this constraint while admitting curricula over Θ ′ .
Sample-Matched PLR (SAMPLR)
Randomly initialize policy π level buffer Λ size belief t | τ while not converged do
Sample replay-decision Bernoulli, d ∼ P D ( d ) if d = 0 or | Λ | = 0 then
Sample level θ from level generator Collect π ’s trajectory τ on θ , with a stop-gradient ϕ ⊥ else
Use PLR to sample a replay level from the level store, θ ∼ Λ
Collect fictitious trajectory τ ′ on θ , based on s ′ t ∼ B
Update π with rewards R ( τ ′ ) end
Compute PLR score, S = score ( τ ′ , π )
Update Λ with θ using score S end
We now describe a general strategyfor addressing CICS, and apply it to
PLR ⊥ , resulting in Sample-Matched PLR(SAMPLR).
This new UED method featuresthe robustness properties of PLR ⊥ whilemitigating the potentially harmful effects of
CICS over the aleatoric parameters Θ ′ .
As discussed in Section 3, CICS becomeproblematic when the covariate shift occursover some aleatoric subset Θ ′ of theenvironment parameters Θ , such that theexpectation over Θ ′ influences the optimalpolicy.
Adaptive curriculum methods like
PLR ⊥ prioritize sampling of environmentsettings where the agent experiences the mostlearning.
While such a curriculum lets theagent focus on correcting its largest errors, thecurriculum typically changes the distributionover aleatoric parameters Θ ′ , inducing bias inthe resulting decisions.
Ideally, we can eliminate this bias, ensuring the resulting policy makes optimaldecisions with respect to the ground-truth utility function, conditioned on the current trajectory:
[Equation]
A naive solution for grounding is to simply exclude Θ ′ from the set of environment parameters undercurriculum control.
That is, for each environment setting proposed by the curriculum, we resampleθ ′ ∼ P .
We refer to this approach as naive grounding .
Naive grounding forces the expected rewardand next state under each transition at the current AOH τ to match that under P .
Thus, optimalpolicies under naive grounding must be optimal with respect to the ground-truth distribution over θ ′ .
While technically simple, naive grounding suffers from lack of control over Θ ′ .
This limitation is ofno concern when the value of Θ ′ does not alter the distribution of τ until the terminal transition, e.g.when Θ ′ is the correct choice in a binary choice task, thereby only influencing the final, sparse rewardwhen the right choice is made.
In fact, our initial experiment in Section 6 shows naive groundingperforms well in such cases.
However, when the value of Θ ′ changes the distribution of τ beforethe terminal transition, the agent may benefit from a curriculum that actively samples levels whichpromote learning robust behaviors under unlikely events.
Enabling the full benefits of the curriculumin such cases requires the curriculum to selectively sample values of Θ ′ .
Instead of naive grounding, we aim to ground only the policy updates, allowing the curriculum tobias the training distribution.
This can be accomplished by optimizing the following objective:
[Equation]
To achieve this we directly replace the reward r t and next state s t +1 with counterfactual values thatwould be experienced if θ ′ were consistent with τ and P , so that θ ′ ∼ P ( θ ′ | τ ) .
This substitutionoccurs by simulating a fictitious transition , where the fictitious state is sampled as s ′ t ∼ B ( s ′ t | τ ) ,the action as a t ∼ π ( ·| τ ) (as per usual), the fictitious next state as s ′ t +
=
T ( s ′ t , a t ) , and thefictitious reward as r ′ t =
R ( s ′ t +1 ) .
Here, the belief model B ( s ′ t | τ ) provides the ground-truth posteriordistribution of the current state given τ :
[Equation]
Fictitious transitions, summarized in Figure 2, ground the observed rewards and state transitions to P .
Should training on these transitions lead to an optimal policy over Θ , this policy will also be optimalwith respect to P .
We prove this property in Section 5.
Fictitious transitions thus provide the benefitof naive grounding without giving up curriculum control over Θ ′ .
In general, we implement B as follows:
Given P (Θ ′ ) as a prior, wemodel the posterior P ( θ ′ | τ ) with Bayesian inference.
The posteriorcould be learned via supervised learning with trajectories collectedfrom the environment for a representative selection of θ ′ .
Further,we may only have limited access to P (Θ) throughout training, forexample, if sampling P (Θ) is costly.
In this case, we can learn anestimate ˆ P (Θ ′ ) from samples we do collect from P (Θ) , which canoccur online.
We can then use ˆ P (Θ ′ ) to inform the belief model.
SAMPLR, summarized in Algorithm 1, incorporates this fictitioustransition into PLR ⊥ by replacing the transitions experienced inreplay levels sampled by PLR ⊥ with their fictitious counterparts,as PLR ⊥ only trains on these trajectories.
PLR ⊥ uses PPO withthe Generalized Advantage Estimator [GAE, 38] as the base RLalgorithm, where both advantage estimates and value losses can bewritten in terms of one-step TD errors δ t at time t .
Training onfictitious transitions then amounts to computing these TD errorswith fictitious states and rewards: δ t = r ′ t + V ( s ′ t ) − V ( s ′ t +1 ) .
Importantly, because PLR ⊥ provably leads to policies that minimize worst-case regret over all θ at
NE, SAMPLR enjoys the same property for θ ∼ P (Θ) .
A proof of this fact is provided in Section 5.
Applying SAMPLR requires two key assumptions:
First, the simulator can be reset to a specific state,which is often true, as RL largely occurs in resettable simulators or those that can be made to do so.
When a resettable simulator is not available, a possible solution is to learn a model of the environmentwhich we leave for future work.
Second, we have knowledge of P (Θ ′ ) .
Often, we know P a priori ,e.g. via empirical data or as part of the domain specification, as in games of chance.
The Grounded Optimality of SAMPLR
Training on fictitious transitions is a method for learning an optimal policy with respect to the groundtruth utility function U D ( π ) over the distribution D of training trajectories τ , defined in Equation 5.
When D corresponds to the distribution of trajectories on levels θ ∼ P (Θ) , U D ( π ) reduces to theground-truth utility function, U ( π ) .
For any UED method, our approach ensures that, in equilibrium,the resulting policy is Bayes-optimal with respect to P (Θ) for all trajectories in the support of D .
Remark 1.
If π ∗ is optimal with respect to the ground-truth utility function U D ( π ) then it is optimalwith respect to the ground-truth distribution P (Θ) of environment parameters on the support of D .
Proof.
By definition we have π ∗ ∈ arg max π ∈ Π { U D ( π ) }
= arg max π ∈ Π
{ E τ ∼D (cid:2) U
( π | τ ) (cid:3) } .
Since π cancondition on the initial trajectory τ , the action selected after each trajectory can be independentlyoptimized.
Therefore, for all τ ∈ D , π ∗ ∈ arg max π ∈ Π { U ( π | τ ) } implying that π ∗ is the optimal policymaximizing U ( π | τ ) .
Thus, assuming the base RL algorithm finds Bayes-optimal policies, a UED method that optimizesthe ground-truth utility function, as done by SAMPLR, results in Bayes-optimal performance overthe ground-truth distribution.
If the UED method maximizes worst-case regret, we can prove an evenstronger property we call robust ϵ -Bayes optimality .
Let U θ ( π ) be the ground-truth utility function for π on the distribution D πθ of initial trajectoriessampled from level θ , so that U θ ( π ) =
U D πθ ( π ) .
Given a policy π maximizing U θ ( π ) , we say that πis robustly ϵ -Bayes optimal iff for all θ in the domain of P (Θ) and all π ′ , we have
U θ ( π ) ≥ U θ ( π ′ )
− ϵ.
Note how this property differs from being simply ϵ -Bayes optimal, which would only imply that
U ( π ) ≥ U ( π ′ )
− ϵ.
Robust ϵ -Bayes optimality requires π to be ϵ -optimal on all levels θ in the support of the groundtruth distribution, even those rarely sampled under P (Θ) .
We will show that at ϵ -Nash equilibrium,
SAMPLR results in a robustly ϵ -Bayes optimal policy for the ground-truth utility function U θ ( π ) .
Incontrast, training directly on levels θ ∼ P (Θ) results in a policy that is only ϵ -Bayes optimal.
Theorem 1.
If π ∗ is ϵ -Bayes optimal with respect to U (cid:98) D ( π ) for the distribution (cid:98) D of trajectoriessampled under π over levels maximizing the worst-case regret of π , as occurs under SAMPLR, thenπ ∗ is robustly ϵ -Bayes optimal with respect to the ground-truth utility function, U ( π ) .
Proof.
Let π ∗ be ϵ -optimal with respect to U (cid:98) D ( π ) where (cid:98) D is the trajectory distribution under π onlevels maximizing the worst-case regret of π .
Let π ∗ be an optimal grounded policy.
Then for any θ ,
[Equation]
The first inequality follows from (cid:98) D being trajectories from levels that maximize worst-case regretwith respect to π ∗ , and the second follows from π ∗ being ϵ -optimal on U (cid:98) D ( π ) .
Rearranging termsgives the desired condition.
Experiments
Our experiments first focus on a discrete, stochastic binary choice task, with which we validate ourtheoretical conclusions by demonstrating that CICS can indeed lead to suboptimal policies.
Moreover,we show that naive grounding suffices for learning robustly optimal policies in this setting.
However,as we have argued, naive grounding gives up control of the aleatoric parameters Θ ′ and thus lacksthe ability to actively sample scenarios helpful for learning robust behaviors—especially importantwhen such scenarios are infrequent under the ground-truth distribution P (Θ) .
SAMPLR inducespotentially biased curricula, but retains optimality under P (Θ) by matching transitions under P (Θ ′ )with those under P (Θ ′ ) .
We assess the effectiveness of this approach in our second experimentaldomain, based on the introductory example of driving icy roads.
In this continuous-control drivingdomain, we seek to validate whether SAMPLR does in fact learn more robust policies that transfer totail cases under P (Θ ′ ) , while retaining high expected performance on the whole distribution P (Θ ′ ) .
All agents are trained using PPO [39] with the best hyperparameters found via grid search using a setof validation levels.
We provide extended descriptions of both environments alongside the full detailsof our architecture and hyperparameter choices in Appendix C.
[Figure]
6.1 Stochastic Fruit Choice
[Figure] We aim to demonstrate the phenomenon of CICS in Stochastic Fruit Choice, a binary choice task,where the aleatoric parameter determines the correct choice.
This task requires the agent to traverse upto eight rooms, and in the final room, decide to eat either the apple or banana.
The correct choice θ ′ isfixed for each level, but hidden from the agent.
Optimal decision-making depends on the ground-truthdistribution over the correct fruit, P (Θ ′ ) .
This task benefits from a curriculum over the number ofrooms, but a curriculum that selectively samples over both room layout and correct fruit choice canlead to suboptimal policies.
Figure 3 shows example levels from this environment.
This domain presents a hard exploration challenge for RL agents, requiring robust navigation acrossmultiple rooms.
Further, this environment is built on top of MiniHack [36], enabling integration ofselect game dynamics from the NetHack Learning Environment [23], which the agent must master tosucceed:
To go from one room to the next, the agent needs to learn to kick the locked door until itopens.
Upon reaching the final room, the agent must then apply the eat action on the correct fruit.
Let π A be the policy that always chooses the apple, and π B , the banana.
If the probability that thegoal is the apple is P ( A ) =
q , then the expected return is R A q under π A and R B (1 − q ) under π B .
The optimal policy is π A when q > R B / ( R A + R B ) , and π B otherwise.
Domain randomization(DR), which directly samples each level θ ∼ P ( θ ) , optimizes for the correct ground-truth P (Θ ′ ) , butwill predictably struggle to solve the exploration challenge.
PLR ⊥ may induce curricula easing theexploration problem, but can be expected make the correct fruit choice oscillate throughout trainingto maximize regret, leading to problematic CICS.
We set R A = 3 , R B = 10 , and q = 0 .
7 , making π B optimal with an expected return of 3 . 0 .
Wecompare the train and test performance of agents trained with DR, PLR ⊥ , and PLR ⊥ with naivegrounding over 200M training steps in Figure 3.
In this domain, SAMPLR reduces to naive grounding,as θ ′ only effects the reward of a terminal transition, making fictitious transitions equivalent to realtransitions for all intermediate time steps.
We see that DR struggles to learn an effective policy,plateauing at a mean return around 1 . 0 , while PLR ⊥ performs the worst.
Figure 6 in Appendix Bshows that the PLR ⊥ curriculum exhibits much higher variance in q , rapidly switching the optimalchoice of fruit to satisfy its regret-maximizing incentive, making learning more difficult.
In contrast,
PLR ⊥ with naive grounding constrains q = 0 .
7 , while still exploiting a curriculum over an increasingnumber of rooms, as visible in Figure 6.
This grounded curriculum results in a policy that solvesmore complex room layouts at test time.
Figures 5 and 6 in Appendix B additionally show how the
SAMPLR agent’s choices converge to π B and how the size of SAMPLR’s improvement varies underalternative choices of q in { 0 . 5 , 0 . 3 } .
6.2 Zero-Shot Driving Formula 1 Tracks with Black Ice
We now turn to a domain where the aleatoric parameters influence the distribution of τ t at each t ,thereby creating opportunities for a curriculum to actively sample specific θ ′ to promote learning onbiased distributions of τ t .
We base this domain on the black ice driving scenario from the introductionof this paper, by modifying the CarRacingBezier environment in [20].
In our version, each track tilehas black ice with probability q , in which case its friction coefficient is 0, making acceleration and
[Figure]
e t u r n
[Figure]
PLR
Naive grounding
SAMPLR braking impossible.
This task is especially difficult, since the agent cannot see black ice in its pixelobservations.
Figure 4 shows example tracks with ice rendered for illustration purposes.
The episodicreturns scale linearly with how much of the track is driven and how quickly this is accomplished.
As success requires learning to navigate the challenging dynamics over ice patches, a curriculumtargeting more difficult ice configurations should lead to policies more robust to black ice.
Here, theground-truth distribution P (Θ ′ ) models the realistic assumption that most days see little to no ice.
We therefore model the probability of ice per tile as q ∼ Beta ( α, β ) , where α = 1 , β = 15 .
[Table]
We test the hypothesis that SAMPLR’sregret-maximizing curriculum results inpolicies that preserve optimal performanceon the ground-truth distribution P (Θ ′ ) ,while being more robust to tail casescompared to DR and PLR ⊥ with naivegrounding.
We expect standard PLR ⊥ tounderperform all methods due to CICS,leading to policies that are either toopessimistic or too optimistic with respectto the amount of ice.
These baselines provide the controls needed to distinguish performance changesdue to the two grounding approaches and those due to the underlying curriculum learning method.
We train agents with each method for 5M and test zero-shot generalization performance on the
Formula 1 (F1) tracks from the CarRacingF1 benchmark, extended to allow each track segment tohave black ice with probability q in { 0 . 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } .
These test tracks are significantly longerand more complex than those seen at training, as well as having a higher rate of black ice.
To implement SAMPLR’s belief model, we use a second simulator as a perfect model of theenvironment.
At each time step, this second simulator, which we refer to as the fictitious simulator ,resets to the exact physics state of the primary simulator, and its icy tiles are resampled according tothe exact posterior over the aleatoric parameter q = θ ′ , such that θ ′ ∼ P
θ ′ | τ ) , ensuring the futureuncertainty is consistent with the past.
The agent decides on action a t based on the current realobservation o t , and observes the fictitious return r ′ t and next state s ′ t +1 determined by the fictitioussimulator after applying a t in state s ′ t ∼ P ( s ′ t | τ, θ ′ ) .
This dual simulator arrangement, fully detailedin Appendix A.2, allows us to measure the impact of training on fictitious transitions independentlyof the efficacy of a model-based RL approach.
Further, as the training environment in RL is mostoften simulation (e.g. in sim2real), this approach is widely applicable.
SAMPLR outperforms all baselines in zero-shot transfer to higher ice rates on the full F1 benchmarkand attains a statistically significant improvement at p < 0 .
001 when transferring to q = 0 .
4 andq = 0 . 6 , and p < 0 .
05 when q = 0 . 8 .
Importantly, SAMPLR outperforms PLR ⊥ with naivegrounding, indicating that SAMPLR exploits specific settings of Θ ′ to better robustify the agentagainst rare icy conditions in the tail of P (Θ ′ ) .
Indeed, Figure 4 shows that on average, SAMPLRexposes the agent to more ice per track tile driven, while PLR ⊥ underexposes the agent to icecompared to DR and naive grounding, suggesting that under PLR ⊥ agents attain higher regret onice-free tracks—a likely outcome as ice-free tracks are easier to drive and lead to returns, with which regret scales.
Unfortunately, this results in PLR ⊥ being the worst out of all methods on theground-truth distribution.
SAMPLR and naive grounding avoid this issue by explicitly matchingtransitions to those under P at τ .
As reported in Table 1, SAMPLR matches the baselines in meanperformance across all F1 tracks under P (Θ ′ ) , indicating that despite actively sampling challengingθ ′ , it preserves performance under P (Θ ′ ) , i.e. the agent does not become overly cautious.
Related Work
The mismatch between training and testing distributions of input features is referred to as covariateshift , and has long served as a fundamental problem for the machine learning community.
Covariateshifts have been extensively studied in supervised learning [48, 18, 5, 2].
In RL, prior works havelargely focused on covariate shifts due to training on off-policy data [44, 34, 11, 14, 13, 46] includingthe important case of learning from demonstrations [31, 33].
Recent work also aimed to learn invariantrepresentations robust to covariate shifts [53, 54].
More generally, CICS is a form of sample-selectionbias [15].
Previous methods like OFFER [7] considered correcting biased transitions via importancesampling [43] when optimizing for expected return on a single environment setting, rather than robustpolicies over all environments settings.
We believe our work provides the first general formalizationand solution strategy addressing curriculum-induced covariate shifts (CICS) for RL.
The importance of addressing CICS is highlighted by recent results showing curricula to be essentialfor training RL agents across many of the most challenging domains, including combinatorialgridworlds [55], Go [40], StarCraft 2 [49], and achieving comprehensive task mastery in open-endedenvironments [41].
While this work focuses on PLR ⊥ , other methods include minimax adversarialcurricula [30, 50, 51] and curricula based on changes in return [25, 32].
Curriculum methods havealso been studied in goal-conditioned RL [12, 6, 42, 27], though CICS does not occur here as goalsare observed by the agent.
Lastly, domain randomization [DR, 35, 29] can be seen as a degenerateform of UED, and curriculum-based extensions of DR have also been studied [19, 47].
Prior work has also investigated methods for learning Bayes optimal policies under uncertaintyabout the task [56, 28], based on the framework of Bayes-adaptive MDPs (BAMDPs)
Inthis setting, the agent can adapt to an unknown MDP over several episodes by acting to reduce itsuncertainty about the identity of the MDP.
In contrast, SAMPLR learns a robustly Bayes-optimalpolicy for zero-shot transfer.
Further unlike these works, our setting assumes the distribution of somealeatoric parameters is biased during training, which would bias the a posteriori uncertainty estimateswith respect to the ground-truth distribution when optimizing for the BAMDP objective.
Instead,
SAMPLR proposes a means to correct for this bias assuming knowledge of the true environmentparameters, to which we can often safely assume access in curriculum learning.
Deeply related, Off-Belief Learning [OBL, 16] trains cooperative agents in self-play using fictitioustransitions assuming all past actions of co-players follow a base policy, e.g. a uniformly random one.
Enforcing this assumption prevents agents from developing conventions that communicate privateinformation to co-players via arbitrary action sequences.
Such conventions hinder coordination withindependently trained agents or, importantly, humans.
SAMPLR can be viewed as adapting OBL tosingle-agent curriculum learning, where a co-player sets the environment parameters at the start ofeach episode (see Appendix D).
This connection highlights how single-agent curriculum learning isinherently a multi-agent problem, and thus problems afflicting multi-agent learning also surface inthis setting; moreover, methods addressing such issues in one setting can then be adapted to the other.
Conclusion
This work characterized how curriculum-induced covariate shifts (CICS) over aleatoric environmentparameters Θ ′ can lead to suboptimal policies under the ground-truth distribution over theseparameters, P (Θ ′ ) .
We introduced a general strategy for correcting CICS, by training the agent onfictitious rewards and next states whose distribution is guaranteed to match what would be experiencedunder P (Θ ′ ) .
Our method SAMPLR augments PLR ⊥ with this correction.
By training on fictitioustransitions, SAMPLR actively samples specific values of θ ′ that induce trajectories with greaterlearning potential, while still grounding the training data to P (Θ ′ ) .
Crucially, our experiments inchallenging environments with aleatoric uncertainty showed that SAMPLR produces robust policiesoutperforming those trained with competing baselines that do not correct for CICS. Checklist For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 4.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
Appendix E.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section
In particular, we highlight that our proof of SAMPLR’s robustly ϵ -Bayes optimal
property assumes the underlying optimization procedure reaches an approximate Nash
equilibrium.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 5.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)?
[Yes]
The
code reproducing the experimental results is included in the supplemental material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 6 and
Appendix C.
(c) Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)?
[Yes] See figure captions in Section 6.
(d) Did you include the total amount of compute and the type of resources used (e.g.,
type of GPUs, internal cluster, or cloud provider)?
[Yes] See compute estimates in
Appendix C.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[Yes] We include the license for the code
reproducing our experiments in the supplemental material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include the code reproducing our experimental results in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]