Return Dispersion as an Estimator of Learning Potential for Prioritized Level Replay
Abstract
Prioritized Level Replay (PLR) has been shown to induce adaptive curricula that improve the sample-efﬁciency and generalization of reinforcement learning policies in environments featuring multiple tasks or levels.
PLR selectively samples training levels weighed by a function of recent temporal-difference (TD) errors experienced on each level.
We explore the dispersion of returns as an alternative prioritization criterion to address certain issues with TD error scores.
Introduction
Recently, there has been a growing interest in testing the generalization and sample efﬁciency of
RL algorithms [1–5].
This led to the development of new benchmarks such as Procgen [6], which consists of 16 procedurally-generated game environments, allowing for distinct train and test levels.
One method which has been shown to be effective is Prioritized Level Replay (PLR) [7].
PLR performs active sampling of training levels in environments featuring multiple levels by selectively sampling the next training level in proportion to its learning potential, as estimated by a function of recent temporal-difference (TD) errors experienced on each level.
For Proximal Policy Optimization (PPO) [8], the best results were achieved when prioritizing based on the average L1 value loss, or equivalently, the magnitude of the generalized advantage estimator (GAE) [9].
PLR improves both sample-efﬁciency and generalization by actively sampling for the largest opportunities for reducing TD errors or the value loss, and thus the epistemic uncertainty—the uncertainty intrinsic to the agent’s own predictive modules.
Despite the effectiveness of PLR, its usage of value losses makes it susceptible to unreliable scores under aleatoric uncertainty—sources of irreducible uncertainty due to inherent randomness in the environment [10, 11]—as the value loss does not distinguish between epistemic and aleatoric uncertainty.
In this paper, we propose using a measure of episodic return dispersion for assessing a level’s learning potential.
We discuss how value losses can struggle in the presence of aleatoric uncertainty, and how the proposed dispersion-based method can help us to partially address this issue.
However, we ﬁnd PLR using return dispersion leads to inconsistent improvements in test performance across the environments in the Procgen Benchmark, improving test returns in some environments, while hurting it in others.
Moreover, in its current instantiation, our idea is impractical due to its sample inefﬁciency in requiring multiple rollouts per level, although we will conclude the paper by discussing how this inefﬁciency can be addressed in future work. [Table]
Background: Prioritized Level Replay
Prioritized Level Replay is an adaptive curriculum learning algorithm which varies the probability that an RL agent trains on particular levels (e.g. conﬁgurations of the environment or task) as a function of the level’s priority.
The algorithm prioritizes levels based on a mixture of a scoring metric aimed at estimating the learning potential of the agent on the level, and of how long ago the level was last acted on by the policy (as a mechanism to keep learning potential estimates up-to-date).
As it pertains to learning potential, the scoring criterion is the only component of PLR that we will focus on here.
Suppose we run a policy π θ on a level l i for T steps until termination, resulting in a trajectory τ .
PLR computes the level’s score as a magnitude of the GAE averaged across T steps:
[Equation]
[Equation]
where δ t =
r t + γV
( s t +1 )
− V ( s t ) is the one-step TD error.
The λ hyperparameter trades off the bias and variance of the advantage estimator.
The two extreme cases with the most bias and most variance correspond to λ = [-
-]0 and λ = 1 : A GAE ( γ, 0) t = r t + γV
( s t +1 ) − V ( s t ) and ˆ A GAE ( γ, 1) t = (cid:80) Tk =
γ k r t + k − V
( s t ) .
Unsurprisingly, the latter Monte-Carlo estimate proved unsuitable as a PLR score due to its high variance, while GAE ( γ, 0) and GAE ( γ, 0 . 95) work well [7].
Jiang et al.
[7] interprets the per step score in Eq.
1 as the L1 value loss, (cid:12)(cid:12) V target t − V t (cid:12)(cid:12) .
This inspired the formulation of the Value Correction Hypothesis, which states that the levels most conducive to learning are those with the highest value losses.
PLR computes these value losses based on the single, most recent trajectory for each level.
However, in settings with aleatoric uncertainty, such scores can become unreliable.
Consider the case of a partially-observable level, where the agent ﬁnds itself in the middle of a long corridor, and the unobserved goal can be at either end.
Even if the agent has a perfect strategy for checking both sides of the corridor, it will still incur non-zero value losses.
Relatedly, value losses can be arbitrarily high if the rewards are stochastic, as we increase the reward variance.
In both of these cases, value losses cannot disentangle epistemic and aleatoric sources of uncertainty, thereby potentially wrongly prioritizing between levels with high aleatoric uncertainty, but in which the policy is already close to optimal, and levels with low aleatoric uncertainty, in which the policy can still do better.
Therefore, value losses are liable to improperly capturing the desired notion of learning potential in stochastic settings.
Despite this shortcoming, PLR with L1 value loss scores leads to signiﬁcant generalization gains on the Procgen Benchmark.
However, as pointed out in Raileanu and Fergus [12], even though Procgen levels have deterministic transitions, the value function will still capture irreducible uncertainty due to observational feature aliasing across levels—the overlap between state representations as learned by the value network.
This effect can be a consequence of using unsuitable network architectures, or directly stem from partial observability as in the corridor example above.
In both cases, it leads to less reliable value loss scores.
PLR with Dispersion Scores
[Figure]
SD, unlike CV, is not scale-invariant, so multiplying rewards by a factor of two leads to larger σ .
We set CV to 0 when both σ and µ are 0.
Consider a level that is easy for the agent to solve.
On such a level, the agent will consistently attain high returns.
Likewise, for a level that is too hard, the agent will consistently receive low returns.
In both cases, there is little signal from which the agent can learn.
In contrast, levels on the frontier of the agent’s current abilities will be solved sporadically, leading to returns that vary across episodes, leading to a rich learning signal.
To identify those levels, one can measure how dispersed the returns are, for instance, using the standard deviation (SD) or the coefﬁcient of variation (CV): the ratio of the standard deviation to the mean.
CV is scale-invariant and its asymmetric behaviour as shown in Figure 1 reﬂects one way of how we might want to prioritize levels: difﬁcult levels that are rarely solved would be given a higher priority.
Similarly to the value loss, the proposed return dispersion score conﬂates the epistemic and aleatoric sources of randomness in the case of stochastic returns.
However, in the case of aleatoric uncertainty due to partial observability, return dispersion scores can help alleviate potential inaccuracies in level scores resulting from value function bias due to feature aliasing.
Return dispersion scores have the added beneﬁt of being easier to combine with various base RL algorithms.
RL algorithms vary significantly in how their value losses are computed, making it necessary to customize implementations of value-loss scores for each algorithm.
In contrast, computing return dispersion is agnostic to the underlying RL algorithm.
As we point out in the Appendix, there are, however, connections between value losses and the return dispersion.
Experiments
For our experiments, we extend the original PLR implementation 1 , and use the default hyperparameters unless mentioned explicitly.
In Figure 2, we use the Procgen Benchmark to compare variants of PLR with the following scoring functions: average GAE magnitude as in Eq.1 (L1 value loss PLR), CV of returns, and SD of returns.
The latter two scores are computed based on the undiscounted and unnormalized returns from 12 episodes for each level sampled during a training iteration.
These additional episodes are only used to compute the CV or SD of returns, and are not used for PPO updates.
In the Appendix, we include an experiment with a higher number of evaluation episodes, and provide some extra results for more variants of value-based PLR.
[Figure]
a n t e s t e p i s o d e r e t u r n
[Figure]
Our motivation for measuring the dispersion of returns mainly suits environments where a single reward is given for a successful completion of a level.
In Procgen, these environments are Coinrun,
Jumper, Leaper, Heist, Maze and Ninja.
On each of these games, PLR with either CV or SD performs better or no worse than the L1 value loss.
In other environments, like Bigﬁsh, Chaser, Dodgeball,
Miner, Climber and Bossﬁght, the agent aims to obtain a sequences of rewards accumulating over the course of an episode.
Here, our strategy of computing the dispersion of returns is less suitable.
Caveﬂyer is an example of an environment where dispersion-based scores may fail.
In this environment, the agent must navigate through a cave to reach a goal, at which point the episode terminates and the agent receives a completion reward.
However, the agent can obtain additional rewards by destroying objects along the way.
These rewards can inﬂate the return variance, rendering CV or
SD scores uninformative.
Meanwhile, Plunder is an example environment where CV-based PLR performs particularly poorly.
One explanation for this could be that it prioritizes levels that are too difﬁcult from the start, making it impossible for the agent to learn a good policy.
We see that SD struggles with environments like Starpilot and Bigﬁsh.
We attribute this to SD being unsuitable for comparing levels featuring large shifts in mean returns.
On the other hand, CV, which measures the variability in relation to the mean, performs slightly better than L1 value loss on these two environments.
To see how these methods perform on average across all 16 environments, we ran the aggregate analysis from Agarwal et al.
[13] using the ﬁnal scores on 1000 test levels.
Figure 3 shows no improvements in comparison to PLR with L1 value loss.
However, we see that choosing the best performing method out of CV or SD for each environment, as we do post hoc for PLR-SD-CV, results in a marked improvement in test performance, suggesting that an automated mechanism for choosing between CV or SD may make dispersion-based scores a stronger alternative to L1 value loss in this domain.
Discussion
We discussed two ways in which aleatoric uncertainty can lead to unreliable PLR scores, namely through stochastic rewards or feature aliasing.
In order to address the feature-aliasing case, we proposed to score levels based on the dispersion of returns.
We empirically showed that, with
PLR, such dispersion scores can improve upon value loss scores.
Nevertheless, the inconsistent improvements across the Procgen Benchmark using SD and CV scores show that the choice of the dispersion measure has a signiﬁcant impact on performance.
Thus, an interesting continuation of this work would be to determine whether there are better dispersion measures for PLR.
Alternatively, we might be able to ﬁnd a suitable reward transformation that makes SD work well across all environments, or, perhaps, a mechanism for choosing the dispersion measure upfront depending on the properties of the environment.
In our current implementation, evaluating CV and SD scores requires collecting additional episodes, making it extremely sample-inefﬁcient compared to standard PLR.
Future work is required to make our idea practical in terms of sample efﬁciency, and existing estimators for the variance of returns [14] can help in this regard.
Further, while dispersion scores improve the reliability of PLR scores when dealing with aleatoric uncertainty due to feature-aliasing, they do not address the case of stochastic rewards.
We plan to extend the ideas developed here—namely using PLR scores based on the return distribution—to handle the case of stochastic rewards in future work.
Lastly, while PLR has been primarily motivated via the Value Correction Hypothesis, this explanation only provides a heuristic argument for the success of this method.
We hypothesize its effectiveness may be tied to the update mechanics of the underlying RL algorithm.
Jiang et al.
[7] used PPO, for which GAE is used in the policy gradient estimator, ˆ g = ˆ E t [ ∇ θ log π θ ( a t | s t )
ˆ A GAE ( γ,λ ) t ] .
Larger
GAE magnitudes (or L1 value losses) correlate with larger parameters updates, which may lead to both faster training and improved generalization [15].
This may provide an alternative explanation for the beneﬁts of PLR that may be worth further investigation.
Appendix A.1 Additional experimental results PLR with other value-based scores
It is useful to recall the following deﬁnition of the advantage function A ( a t , s t ) whose estimates ˆ A t are used in the construction of the value loss:
[Equation]
From this follows the interpretation of A ( s t , a t ) as the advantage, measured by the difference in expected returns, of taking an action a t in state s t in comparison to all other actions under our policy.
Positive advantages point us to better actions, whose probability under π θ should be increased.
Usually, there are very few good actions in comparison to the number of bad actions one could take in each state, and there are many more ways in which an action can be bad.
As Schulman et al. [9] points out, this asymmetry can be exploited for prioritization.
In our case, instead of the GAE magnitude, we can use max( ˆ A t , 0) or exp( ˆ A t ) .
This scoring function, also called the positive value loss , has previously been studied in the context of PLR, and shown to be effective in a continuous control domain [16].
As we show in Figure 4, PLR with positive value loss, i.e. max( ˆ A t , 0) averaged across time steps, tends to perform worse than PLR with L1 value loss in the Procgen domain.
In Figure 4, we also show the results of PLR with L1 value loss whose V target t
[Equation]
Monte-Carlo (MC) estimate of the return.
For the majority of Procgen games, it performs worse than the uniform-sampling baseline.
This is likely because of the high variance of the MC estimate.
[Figure]
M e a n t e s t e p
i s o d e r e t u r n
[Figure]
Experiments with more evaluation episodes
Estimates of the CV based on 12 episodes can be noisy, and so we checked if it helps to increase this number.
Figure 5 shows the results for the Leaper environment with 12 and 48 episodes used for estimating the CV.
While the latter converges on a slightly higher value, the improvements are marginal in comparison to the amount of extra compute.
However, it indicates possible gains from having less noisy estimates of the dispersion.
5M 10M 15M 20M 25M Steps
M e a n t e s t e p
i s o d e r e t u r n
PLR L1 value loss
PLR CV 12 episodes
PLR CV 48 episodes
A.2 Connection between values losses and the variance of returns
To see the connection between advantage estimates and the variance of returns, let us start from the deﬁnition of GAE ( γ, 1) :
[Equation]
Compare this to the variance of a random variable R t = (cid:80) ∞ k =0 γ k r t + k for some ﬁxed s t :
[Equation]

We see that (cid:16) ˆ A GAE ( γ, 1) ( s t , a t ) (cid:17) 2 estimates Var ( R t | s t ) using a single sample, i.e. a trajectory that starts from s t and a t , and the mean of R t given by the value function.
In our approach, when computing the SD of returns, we effectively estimate Var ( R 0 | s 0 ) based onsamples of the random variable R
= (cid:80) Tt =0 r t .
For GAE ( γ, 0) , we can follow similar steps:
[Equation]
In this case, the random variable is R t =
r t + V
( s t +1 ) and its variance is:
[Equation]
Thus, GAE ( γ, 0) is an estimator of the variance of R t for a ﬁxed s t and a sample of a t , r t and s t +1 .