Perturbed Quantile Regression for Distributional Reinforcement Learning
Abstract
Distributional reinforcement learning aims to learn distribution of return under stochastic environments.
Since the learned distribution of return contains rich information about the stochasticity of the environment, previous studies have relied on descriptive statistics, such as standard deviation, for optimism in the face of uncertainty.
However, using the uncertainty from an empirical distribution can hinder convergence and performance when exploring with the certain criterion that has an one-sided tendency on risk in these methods.
In this paper, we propose a novel distributional reinforcement learning that explores by randomizing risk criterion to reach a risk-neutral optimal policy.
First, we provide a perturbed distributional Bellman optimality operator by distorting the risk measure in action selection.
Second, we prove the convergence and optimality of the proposed method by using the weaker contraction property.
Our theoretical results support that the proposed method does not fall into biased exploration and is guaranteed to converge to an optimal return distribution.
Finally, we empirically show that our method outperforms other existing distribution-based algorithms in various environments including 55 Atari games.
Introduction
Distributional reinforcement learning (DRL) learns the stochasticity of returns in the reinforcement learning environments and has shown remarkable performance in several benchmark tasks.
Its model generates the approximated distribution of returns, where the mean value implies the traditional
Q-value [1, 4, 10].
Learning procedure with stochasticity through return distribution is represented by parametric (epistemic) uncertainty , which is due to insufficient or inaccurate data and intrinsic (aleatoric) uncertainty , which is inherently possessed randomness in the environment [5, 9].
The learned stochasticity gives rise to the notion of risk-sensitivity, and some distributional reinforcement learning algorithms distort the learned distribution to create a risk-averse or risk-seeking policy.
Another way to employ the uncertainty is to design an efficient exploration method which is essential to find an optimal behavior with a few trials.
Optimism in the face of uncertainty (OFU) is one of the fundamental exploration principles that employs parametric uncertainty to promote exploring less understood behaviors and to construct confidence set.
Most OFU algorithms select an action with the highest upper-confidence bound (UCB) of parametric uncertainty which can be considered as the optimism at the moment [3, 7].
In deep RL, several OFU studies often model the parametric uncertainty explicitly through the Bayesian posterior, which is estimated by using neural networks.
However, learning the representation of high-dimensional state-action space and Bellman update simultaneously leads to unstable propagation [32].
[Figure] [Figure]
On the other hand, DRL, which aims to capture the intrinsic uncertainty can provide more statistical information during control such as mode, median, or variance by addressing full characteristics of the return distribution.
Despite the richness of risk-sensitive information for return distribution, only a few DRL methods have tried to employ the benefits of distributional perspective for exploration [8, 17, 19, 29, 34] by utilizing the estimated uncertainty from distributional output that is composed of a mixture of intrinsic and parametric uncertainty.
Unfortunately, separating these two types of uncertainty during learning is not a trivial task.
Mavrin et al.
[17] propose a distribution-based OFU exploration that schedules a decaying bonus rate to suppress the effect of intrinsic uncertainty, which unintentionally induces a risk-seeking policy.
Although OFU based approaches try to reduce parametric uncertainty by revisiting the state with high uncertainty, there exists the side effect that the criteria unfortunately force the agent to chase the intrinsic uncertainty (risk) simultaneously due to the indistinguishability of two distinct uncertainties during updates.
In Figure 1, DLTV which is based on optimism fails to reach the threshold while its baseline algorithm, QR-DQN, can achieve the goal in an environment with high intrinsic uncertainty.
In the entangled case, relying on specific criteria causes a one-sided tendency on risk and makes an agent consistently select certain actions during exploration that degrades performance.
We call this phenomenon ‘fixedness’ and present a simple, yet effective approach to resolve such an issue.
In this paper, we propose perturbed quantile regression (PQR) which perturbs the criterion on uncertainty by randomizing the risk criterion in action selection to avoid a one-sided tendency on risk.
We define the distributional perturbation on return distribution to re-evaluate the estimate of return by distorting the learned distribution with perturbation weight.
Unlike the typical worst-case approach in risk-sensitive settings or OFU based approaches, we instead randomly sample a risk measure from an ambiguity set, which represents that the risk setting is ambiguous when the characteristics of a given environment are unknown.
In summary, our contributions are as follows.
• A risk-neutral strategy called perturbed quantile regression (PQR) is proposed, which improves over naive risk-seeking strategies.
• A sufficient condition for convergence is provided for the proposed Bellman operator with a weaker contraction property.
Backgrounds & Related works 2.1 Distributional RL
We consider a Markov decision process (MDP) which is defined as a tuple ( S , A , P, R, γ ) where S is a finite state space, A is a finite action space, P : S ×
A × S → [0 , 1] is the transition probability, R is the random variable of rewards in [ − R max , R max ] , and γ ∈ [0 , 1) is the discount factor.
We define a stochastic policy π ( ·| s ) which is a conditional distribution over A given state s .
For a fixed policy π , we denote Z π
( s, a ) as a random variable of return distribution of state-action pair ( s, a ) following the policy π .
We attain Z π
( s, a ) = (cid:80) ∞
t =
γ t R ( S t , A t ) , where S t +1 ∼ P ( ·| S t , A t ) , A t ∼ π ( ·| S t ) and S
= s,
A 0 = a .
Then, we define an action-value function as Q π ( s, a ) =
E [ Z π ( s, a )] in [ − V max , V max ] where V max = R max / (1 − γ ) .
For regularity, we further notice that the space of action-value distributions Z has the first moment bounded by V max :
[Equation]
In distributional RL, the return distribution for the fixed π can be computed via dynamic programming with the distributional Bellman operator defined as,
[Equation]
where D = denotes that both random variables share the same probability distribution.
We can compute the optimal return distribution by using the distributional Bellman optimality operator defined as,
[Equation]
Bellemare et al.
[1] have shown that τ π is a contraction in a maximal form of the Wasserstein metric but τ is not a contraction in any metric.
Combining with the expectation operator, E τ is a contraction so that we can guarantee that the expectation of Z converges to the optimal state-action value, while the convergence of a return distribution itself is not guaranteed.
2.2 Exploration on Distributional Reinforcement Learning
To combine with deep RL, a parametric distribution Z θ is used to learn a return distribution by using τ .
Dabney et al.
[10] have employed a quantile regression to approximate the full distribution by letting Z θ ( s, a )
= 1 N (cid:80) Ni =1 δ
θ
i
( s,a ) where the parameter θ represents the locations of a mixture of N
Dirac delta functions.
Each θ i represents the value where the cumulative probability is τ i = iN .
By using the quantile representation with the distributional Bellman optimality operator, the problem can be formulated as a minimization problem as,
[Equation]
N where ( s t , a t , r t , s t +1 ) is a given transition pair, ˆ τ i =
τ i −
+ τ i, a
[Equation]
:= argmax a
[Equation]
Z [ Z θ ( s
[Equation]
+1 , a ′ )] , ρ κ ˆ τ i ( x ) :=
| ˆ τ i − δ { x< 0 } |L κ ( x ) , and L κ ( x ) := x 2 / 2 for | x | ≤ κ and L κ ( x ) :
= κ ( | x |− 12 κ ) , otherwise.
Based on the quantile regression, Dabney et al.
[10] have proposed a quantile regression deep Q network (QR-DQN) that shows better empirical performance than the categorical approach [1], since the quantile regression does not restrict the bounds for return.
As deep RL typically did, QR-DQN adjusts ϵ -greedy schedule, which selects the greedy action with probability 1 − ϵ and otherwise selects random available actions uniformly.
The majority of QR-DQN variants [9, 30] rely on the same exploration method.
However, such approaches do not put aside inferior actions from the selection list and thus suffers from a loss [21].
Hence, selecting a statistically plausible action is crucial for efficient exploration.
In recent studies, Mavrin et al.
[17] modifies the criterion of selecting an action for efficient exploration with optimism in the face of uncertainty.
Using left truncated variance as a bonus term to estimate optimistic way and decaying ratio c t to suppress the intrinsic uncertainty, DLTV was proposed as an uncertainty-based exploration in DRL without using ϵ -greedy exploration.
At timestep t , the action selection of DLTV can be described as:
[Equation]
where θ i ’s are the values of quantile level τ i .
DLTV shows that a constant schedule degrades the performance significantly compared to a decaying schedule.
𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠
Start : 𝑎𝑎: 𝑎𝑎 1 no-op : 𝑎𝑎 2 ~ 𝑎𝑎: 𝑎𝑎 ∗ (optimal)
2.3 Risk in Distributional RL
Instead of an expected value, risk-sensitive RL tries to maximize a certain risk measure such as MeanVariance [33], Value-at-Risk (VaR) [6], or Conditional Value-at-Risk (CVaR) [24, 25], which result in different classes of optimal policy.
Especially, Dabney et al.
[9] interprets risk measures as the expected utility function of the return, i.e., E Z
[ U ( Z ( s, a ))] .
Under this interpretation, risk-sensitive
RL can be formulated as the maximization problem with various types of utility functions.
If the utility function U is linear, the policy obtained under such risk measure is called risk-neutral .
If U is concave or convex, the resulting policy is termed as risk-averse or risk-seeking , respectively.
In general, a distortion risk measure is a generalized expression of risk measure generated from the distortion function.
Definition 2.1.
Let h : [0 , 1] → [0 , 1] be a distortion function such that h (0) = 0 , h (1) = 1 and non-decreasing.
Given a probability space (Ω , F , P ) and a random variable Z : Ω → R , a distortion risk measure ρ h corresponding to a distortion function h is defined by:
[Equation]
where F Z is the cumulative distribution function of Z .
In fact, non-decreasing property of h makes it possible to distort the distribution of Z while satisfying the fundamental property of CDF.
Note that the concavity or the convexity of distortion function also implies risk-averse or seeking behavior, respectively.
Dhaene et al.
[11] showed that any distorted expectation can be expressed as weighted averages of quantiles.
In other words, generating a distortion risk measure is equivalent to choosing a reweighting distribution.
Fortunately, distributional RL has a suitable configuration to apply those uncertainty-based approaches that could naturally expand the class of policies.
Chow et al.
[5] and Stanko and Macek [28] considered risk-sensitive RL with a CVaR objective, where risk is related to robust decision making.
Dabney et al.
[9] expanded the class of policies on arbitrary distortion risk measures and investigated the effects of a distinct distortion risk measures by changing the sampling distribution for quantile targets τ .
Unlike the usual risk-sensitive RL, DLTV applied the risk measure only on action selection, while it keeps the standard objective to obtain a risk-neutral optimal policy.
Our paper will also utilize the risk measure only to select action, and focus on achieving the original risk-neutral purpose.
Perturbation in Distributional RL 3.1 Motivation
Distribution-based OFU exploration [15, 18] was proposed to give a bonus for the uncertainty that can be extracted from the distribution.
However, we found that keeping optimism on uncertainty tends to select sub-optimal behaviors over a long period.
For example, suppose we choose a criterion based on mean-standard deviation with decaying coefficient c t .
Consider two actions a 1 , a 2 with mean µ 1 , µand variance σ 1 , σ 2 respectively, under the following conditions: µ 1 ≥ µ 2 , σ 1 ≤ σ 2 , and µ 1 + c t σ 1 ≤ µ 2 + c t σ 2 .
Then, the agent prefers to select a 2 based on OFU.
To change the decision towards the true optimal action a 1 , the following steps η = min (cid:110) t ′ > t : c t ′ ≤ µ 1 − µ 2 σ 2 − σ 1 (cid:111) − t. need to be spent.
Hence, if there is a bias in the criterion itself, such fixedness often occurs and degrades the performance as the agent has no experience with the optimal policy during that period.
To empirically demonstrate shortcomings of OFU exploration in distributional RL, we build a representative environment that is easy to interpret intuitively among the cases in which intrinsic
[Figure]
experiment
A schematic diagram of the environment is shown in Figure given and and the terminates of reached the
N and the rightmost
12 N (5 ,
1 2 ) + 12 N ( has mean the toward the is
Despite the simple configuration, the possibility to obtain a higher reward in the suboptimal state than the optimal state makes the agent difficult which policy is optimal until it experiences enough to detect the characteristics of each distribution.
Thus, the goal of our toy experiment is to evaluate how quickly each algorithm could find a risk-neutral optimal policy.
Rather than maintaining optimism with decaying schedule to suppress intrinsic uncertainty, we modify the optimism into a randomized risk criterion and named the perturbed variant as p-DLTV.
In short, p-DLTV is a simple modification of DLTV where randomness is given to the coefficient c t through normal distribution.
We compare QR-DQN and DLTV with our randomized variants of two algorithms PQR and p-DLTV to examine the effect of randomized risk criteria.
We describe our main algorithm, PQR, in detail in Section 3.4.
Pseudocode are given in Appendix C.5.
In Figure 3(a), DLTV fails to estimate the true optimal return distribution of action a 1 .
Due to the erroneous estimation, the agent takes longer to recognize its error.
Hence, the deterministic selection based on a fixed criterion could mislead toward exploitation rather than exploration.
This indicates that DLTV may not gather experiences well in stochastic environment.
Hence, decaying schedule while maintaining optimism is not sufficient to avoid risk-seeking behavior.
In Figure 3(b), we count the number of timesteps when the optimal policy was actually performed for each algorithm to show the occurrence of fixedness.
Since the optimal policy consists of the same index a 1 , we plot the total count of performing the optimal action with 10 different seeds.
The interval with a slope of 1 implies that the optimal policy was performed every time.
From the slope of each line, it is observed that DLTV selects the suboptimal action even if the optimal policy was initially performed.
Although the mean return of a 1 (move left) is estimated to be superior, the agent only selects a 2 (move right) during training due to its consistent optimism on uncertainty.
Even if DLTV has spent enough number of time steps to choose the true optimal policy, the remaining procedure is already close to greedy selection as it starts from a decreased coefficient.
In contrast, p-DLTV alleviates the fixedness early and finds the true optimal policy, which implies that a randomized criterion is a simple but effective on training process.
By applying this approach to risk measure, we propose a novel distributional Bellman operator which converges with a weaker contraction property, and build a practical algorithm called PQR, which produces steeper line by quickly obtaining the optimal policy.
Our key idea is to select the statistically plausible action which can be maximal in a certain risk measure.
Compared to QR-DQN,
PQR improves efficiency by excluding inferior actions implicitly that cannot be maximal in any risk criterion.
In addition, Even-Dar et al.
[13] theoretically showed action elimination , which reduces the size of the action sets to be searched by explicitly eliminating sub-optimal action early, can speedup learning to find an optimal policy.
In Section 3.2, we derive the first theoretical sufficient condition for the convergence of exploration method in DRL, which implies that the proposed operator has the same unique fixed point as the standard distributional Bellman equation.
3.2 Perturbed Distributional Bellman Optimality Operator
To choose statistically plausible actions which may be maximal for certain risk criterion, we will generate a distortion risk measure involved in a pre-defined constraint set called an ambiguity set .
The ambiguity set, originated from distributionally robust optimization (DRO) literature, is a family of distribution characterized by a certain statistical distance such as ϕ -divergence or Wasserstein distance [12, 26].
In this paper, we will examine the ambiguity set defined by the discrepancy between distortion risk measure and expectation.
We say the sampled reweighting distribution ξ as (distributional) perturbation and define it as follows:
Definition 3.1.
(Perturbation, Perturbation Gap, and Ambiguity Set)
Given a probability space (Ω , F , P ) , let X : Ω → R be a random variable and Ξ = (cid:8) ξ : ξ ( w ) ≥ 0 , (cid:82) w ∈ Ω ξ ( w ) P ( dw ) =
1 (cid:9) be a set of probability density functions.
For a given constraint set U ⊂ Ξ , we say ξ ∈ U as a (distributional) perturbation from U and denote the ξ − weighted expectation of X as follows:
[Equation]
which can be interpreted as the expectation of X under perturbed probability distribution ξ P .
We further define d ( X ; ξ ) =
| E [ X ] − E
ξ
[ X ] | as perturbation gap of X with respect to ξ .
Then, for a given constant ∆ ≥ 0 , we define the ambiguity set with the bound ∆ as
[Equation]
For brevity, we omit the input w from a random variable unless confusing.
Since ξ is a probability density function, E ξ [ X ] is an induced risk measure with respect to a reference measure P .
Intuitively, ξ ( w ) can be viewed as a distortion to generate a different probability measure and allow to vary the risk tendency.
The aspect of using distortion risk measures looks similar to IQN [9].
However, instead of changing the sampling distribution of quantile level τ implicitly, we reweight each quantile from the ambiguity set.
This allows us to control the maximum allowable distortion with bound ∆ , whereas in IQN the risk measure does not change throughout learning.
In Section 3.4, we suggest a practical method to construct the ambiguity set.
Now, we characterize perturbed distributional Bellman optimality operator (PDBOO)
τ ξ for a fixed perturbation ξ ∈ U ∆ ( Z ) written as below:
[Equation]
Notice that ξ ≡ 1 corresponds to a base expectation, i.e., E ξ,P = E P , which recovers the standard distributional Bellman optimality operator τ .
In risk-sensitive DRL or distributionally robust RL, the Bellman optimality equation is reformulated for a pre-defined risk measure [5, 27, 31].
PDBOO has a significant distinction in that it performs dynamic programming that adheres to the risk-neutral optimal policy while randomizing the risk criterion at every step.
If we consider the time-varying bound of ambiguity set, scheduling ∆ t is a key ingredient to determine whether PDBOO will efficiently explore or converge.
Intuitively, if an agent continues to sample the distortion risk measure from a fixed ambiguity set with a constant ∆ , there is a possibility of selecting sub-optimal actions after sufficient exploration, which may not guarantee eventual convergence.
Based on the quantile model Z θ , our algorithm can be summarized into two parts.
First, we aim to minimize the expected discrepancy between Z θ and τ ξ Z θ − where ξ is sampled from ambiguity set U ∆ .
To clarify notation, we write E ξ [ · ] as a ξ − weighted expectation and E ξ ∼ P ( U ∆ ) [ · ] as an expectation with respect to ξ which is sampled from U ∆ .
Then, our goal is to minimize the perturbed distributional Bellman objective with sampling procedure P :
[Equation]
where we use the Huber quantile loss as a discrepancy on Z θ ′ and τ ξ Z θ − at timestep t .
It is clearly different from DRO which performs the worst-case optimization by using a minimax objective.
By using min-expectation instead of min-max operator, we investigate risk-neutral exploration that can avoid overly pessimistic policies.
Second, considering a sequence ξ t which converges uniformly to 1 so that τ ξ t converges uniformly to standard τ , we derive a sufficient condition of ∆ t that the expectation of any composition of the operators E τ ξ n :1 :=
E τ ξ n τ ξ n − 1 · · · τ ξ 1 has the same unique fixed point as the standard.
3.3 Convergence of the perturbed distributional Bellman optimality operator
In this section, we provide the theoretical result of PDBOO about its convergence through E [ Z ( n ) ] .
We denote the iteration as Z ( n +1) :
= τ ξ n +1 Z ( n ) , Z (0) =
Z for each timestep n > 0 , and the intersection of ambiguity set as ¯ U ∆ n ( Z ( n − 1) ) :
= (cid:84) s,a U ∆ n (cid:0) Z
( n − 1) ( s, a ) (cid:1) .
Assumption 3.2.
Suppose that the bound ∆ n satisfies the condition, (cid:80) ∞ n =1 ∆ n < ∞ .
Practically, satisfying the above assumption is not strict to characterize the landscape of scheduling.
For this update rule, we first state our main theorem below.
Theorem 3.3.
(Weaker Contraction Property) Let ξ n be sampled from ¯ U ∆ n ( Z ( n − 1) ) for every iteration.
If Assumption 3.2 holds, then the expectation of any composition of operators E τ ξ n : converges, i.e., E τ ξ n :
[ Z ] → E [ Z ∗ ] .
Moreover, the following bound holds,
[Equation]
Although the above theorem does not imply γ -contraction property which guarantees an unique fixed point generally, we can show that E [ Z ∗ ] is the unique fixed point for the operator E τ ξ n :1 in the following theorem.
Theorem 3.4.
If Assumption 3.2 holds, E [ Z ∗ ] is the unique fixed point of Bellman optimality equation for any Z ∈ Z .
We now know that the converged value E [ Z ∗ ] is the unique solution of the standard Bellman optimality equation.
It means that PDBOO, which only has weaker contraction property, can achieve the unique fixed point of standard Bellman operator by Assumption 3.2.
Unlike the previous distribution-based or risk-sensitive approaches, PDBOO can be considered as a novel operator which has the compatibility for obtaining a risk-neutral optimal policy by randomizing risk measure during exploration.
3.4 Practical Algorithm with Distributional Perturbation
We propose a perturbed quantile regression (PQR) that is a practical algorithm for distributional reinforcement learning.
Our quantile model is updated by minimizing the objective function (1) induced by PDBOO.
To compute the target distribution of (1), we propose a sampling method of ξ from ambiguity set U ∆ .
Since we employ a quantile model, sampling a reweight function ξ can be reduced into sampling an N -dimensional weight vector ξ :=
[ ξ 1 , · · · , ξ N ] where (cid:80) Ni =1 ξ i =
N and ξ i ≥ 0 for all i ∈ { 1 , · · · , N } .
Based on the QR-DQN setup, note that the condition (cid:82) w ∈ Ω ξ ( w ) P ( dw ) = 1 turns into (cid:80) Ni =1 1 N ξ i = 1 , since the quantile level is set as τ i = iN .
A key issue is how to construct an ambiguity set with bound ∆ t and then sample ξ .
A natural class of distribution for practical use is the symmetric Dirichlet distribution with concentration β , which represents distribution over distributions.
(i.e. x ∼ Dir ( β ) .)
If β is small, most of the mass is concentrated on a few elements.
Otherwise, all elements are similar to each other and produce evenly distributed weight.
By using the Dirichlet distribution, we sample a random vector, x ∼ Dir( β ), and define the reweight distribution as ξ := 1 N + α ( N x − 1 N ) .
From the construction of ξ , we have 1 − α ≤ ξ i ≤ 1 + α ( N − 1) for all i
and it follows that | 1 − ξ i | ≤ α ( N − 1) .
By controlling α , we can bound the deviation of ξ i from 1 and bound the perturbation gap as
[Equation]
Ground Truth µ = 8 . 1 σ = 0 .∆ µ (ˆ µ − µ ) ∆ σ (ˆ σ − σ )
QR-DQN 1.
DLTV -1.
p-DLTV 0.
PQR(ours)
Hence, letting α ≤ ∆ ( N − 1) V max is sufficient to obtain d ( Z ; ξ ) ≤ ∆ in the quantile setting.
We set β = 0 .
· 1 N to generate a constructive perturbation ξ n which gap is close to the bound ∆ n .
To satisfy Assumption 3.2, we set ∆ t = ∆ 0 t − (1+ ϵ ) where ∆ 0 is a hyperparameter.
The detailed procedure is summarized in Algorithm 2 in Appendix C.5.
Experimental Results and Details
Our experiments aim to answer the following questions: (1) Does our PQR method successfully escape from the fixedness phenomenon in stochastic environments?
(2) Can PQR with perturbed action selection accurately capture a stochastic return distribution?
(3) Can a randomized perturbation based exploration serve as a good behavior policy for the full Atari benckmark?
We compare our algorithm to various DRL baselines, which has been shown to achieve better performance on stochastic RL environments.
This comparison is particularly interesting since the proposed methods outperform the advanced DRL models, such as ϵ -greedy for QRDQN, IQN, and Rainbow by only applying the randomized exploration strategy.
The detailed experimental setup and implementation details can be found in Appendix C.
[Figure]
4.1 Learning on Stochastic Environments with High Intrinsic Uncertainty
N-Chain.
As the mean of each return is designed to be similar, it is useful to examine the learning behavior of the empirical return distribution for each algorithm.
Figure 4 shows the empirical PDF of return distribution by using Gaussian kernel density estimation.
Notably, p-DLTV made a much better estimate than DLTV only by changing from optimism to a randomized scheme.
Although the optimal policy was performed, QR-DQN overestimates the optimal Q-value of ( s 2 , a 1 ) as ˆ µ = 9 .
33 , while the ground truth is computed as µ = 10 γ 2 = 8 .
However, PQR estimates the ground truth much better than other baselines with much closer mean and standard-deviation.
LunarLander-v2.
Figure 1 shows that p-DLTV and PQR reach the threshold faster than the other baselines.
Surprisingly, p-DLTV with the randomized approach successfully reach the goal with high extreme reward, but DLTV failed to reach the landing pad.
The result implies that maintaining optimism on intrinsic uncertainty leads to longer fixedness during training and therefore randomized approach could alleviate fixedness within an affordable time budget.
Atari Games with sticky actions.
To avoid the deterministic dynamics of Atari games, [16] proposes injecting stochasticity scheme, called sticky actions , by forcing to repeat the previous action with probability p = 0 .
Sticky actions protocol prevents agents from relying on memorization and allows to evaluate the robustness during training.
In Figure 5, as we expected, PQR shows the steeper learning curves by escaping fixedness earlier without any support of advanced schemes, such as n -step updates for Rainbow.
PQR dramatically improves over IQN and Rainbow in A SSAULT ,
B ATTLEZONE , B EAMRIDER , B EZERK and B OWLING at 50 million steps.
4.2 Full Atari Results
Finally, we evaluate the performance on full set of Atari games, each of which contained intrinsic uncertainty in different ways.
Even if it is well known that Atari benchmarks without sticky actions do not have ‘enough’ stochasticity, intrinsic uncertainty is still prevalent in various manners.
In Table 1, we evaluated 55 Atari results at 50M frames comparing with published score of QRDQN [10], IQN [9], and Rainbow [14] via the report of DQN-Zoo benchmark [22] for reliability.
[Table]
across 55 games on 50M frames, measured as percentages of human baseline.
Reference values are from Quan and Ostrovski [22].
While PQR cannot enjoy the environmental stochasticity by the deterministic dynamics compared to sticky action protocol, PQR achieved 562% performance gain in the mean of human-normalized score over QR-DQN, which is comparable results to IQN.
From the raw scores of 55 games, PQR winsgames against QR-DQN and 34 games against IQN.
Note that IQN benefits from the generalized form of distributional outputs which reduce the approximation error from the number of quantiles output.
While Rainbow is a combination of several orthogonal improvements such as double q-learning, prioritized replay, dueling networks, and n-step updates, PQR has another orthogonal benefit from exploration strategies which are based on the rich information of distributional output and shows the competitive performance with Rainbow.
4.3 Discussion
In section 4.1, a notable consistent result is that just adding randomness to the coefficient c t on DLTV shows the significant improvement supporting that the randomized risk criterion was superior to
OFU in distributional RL.
In section 4.2, PQR successfully escapes the fixedness better than ϵ -greedy methods.
In most RL environments with intrinsic uncertainty, we observe that OFU and ϵ -greedy have difficulty in making a decision that matches risk-neutral purpose, because two uncertainties are intertwined during learning.
Conclusions
In this paper, we proposed a general framework of perturbation in distributional RL which is based on the characteristics of a return distribution.
Without resorting to a pre-defined risk criterion, we revealed and resolved fixedness where one-sided tendency on risk can lead to biased action selection under the stochastic environment.
To our best knowledge, this paper is the first attempt to integrate risk-sensitivity and exploration by using time-varying Bellman objective with theoretical analysis.
In order to validate the effectiveness of PQR, we evaluate on various environments including
Atari games with several distributional RL baselines.
Without separating the two uncertainties, the results show that perturbing the risk criterion is an effective approach to resolve the issue of fixedness.
We believe that PQR can be combined with other distributional RL or risk-sensitive algorithms as a perturbation-based exploration method without sacrificing their original objectives.
Acknowledgements
This work is in part supported by National Research Foundation of Korea (NRF, 2021R1A2C2014504), Artificial Intelligence Innovation Hub (2021-0-02068) grant funded by the
Ministry of Science and ICT (MSIT), Center for Applied Research in Artificial Intelligence(CARAI,
UD190031RD) grant Funded by Defense Acquisition Program Administration(DAPA), Agency for
Defense Development(ADD), INMAC Seoul National University, and BK21-plus.
[-For the rest, the best value of c = 0 . 05 was chosen from [50 , 5 , 0 . 5 , 0 . 
-]