A D ISTRIBUTIONAL P ERSPECTIVE ON A CTOR -C RITIC F RAMEWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent distributional reinforcement learning methods, despite their successes, still contain fundamental problems that can lead to inaccurate representations of value distributions, such as distributional instability, action type restriction, and conﬂation between samples and statistics.
In this paper , we present a novel distributional actor critic framework GMAC to address such problems
Adopting a stochastic policy removes the ﬁrst two problems, and the conﬂation in the approximation is alleviated by minimizing the Cram ´ er distance between the value distribution and its Bellman target distribution.
In addition GMAC improves data efﬁciency by generating the Bellman target distribution through the Sample-
Replacement algorithm denoted by SR λ which provides a distributional generalization of multi -step policy evaluation algorithms
We empirically show that our method captures the multimodality of value distributions and improves the performance of a conventional actor -critic method with low computational cost in both discrete and continuous action spaces, using Arcade Learning Environment (ALE) and
PyBullet environment.
I NTRODUCTION
The ability to learn complex representations via neural networks has enjoyed success in various applications of reinforcement learning (RL), such as pixel-based video gameplays (Mnih et al., 2015), the game of Go (Silver et al., 2016), robotics (Levine et al., 2016), and high dimensional controls like humanoid robots (Lillicrap et al., 2016; Schulman et al., 2015).
Starting from the seminal work of
Deep Q-Network (DQN) (Mnih et al., 2015), the advance in value prediction network, in particular, has been one of the main driving forces for the breakthrough.
Among the milestones of the advances in value function approximation, distributional reinforcement learning (DRL) further develops the scalar value function to a distributional representation.
The distributional perspective offers various beneﬁts by providing more information on the characteristics and the behavior of the value.
One such beneﬁt is the preservation of multimodality in value distributions, which leads to more stable learning of the value function (Bellemare et al., 2017a).
Despite the development, several issues remain, hindering DRL from becoming a robust framework.
First, a theoretical instability exists in the control setting of value-based DRL methods (Bellemare et al., 2017a).
Second, previous DRL algorithms are limited to a single type of action space, either discrete (Bellemare et al., 2017a; Dabney et al., 2018b;a) or continuous (Barth-Maron et al., 2018;
Singh et al., 2020).
Third, a common choice of loss function is Huber quantile regression loss which is vulnerable to conﬂation between samples and statistics without an imputation strategy (Rowland et al., 2019).
The instability issue is not present if a trainable policy is introduced, i.e., the evaluation setting of the Bellman operator is used, as shown by the convergence of distributional Bellman operator (Bellemare et al., 2017a).
In addition, the general form of the stochastic policy gradient method does not assume a speciﬁc type of action space, e.g. discrete or continuous (Williams, 1988; 1992;
Sutton et al., 1999).
Because Wasserstein distance has biased sample gradients (Bellemare et al., 2017b), in practice, directly minimizing the Wasserstein distance is often not preferred as a loss function of a neural network and thus some of the exemplary works (Dabney et al., 2018b;a) of deep DRL minimizes the Huber quantile regression loss (Huber, 1964) instead.
To this end, we treat the methods which minimize the Huber quantile loss as our baseline.
However, as proven in Rowland
[Figure]
et al. (2019), representing a distribution using the Huber quantiles instead of samples can lead to conﬂation between statistics and samples, and thus an imputation strategy is required to learn a more accurate representation of a distribution.
Despite its theoretical soundness, the imputation strategy can introduce computational overheads depending on the statistics (e.g. quantiles, expectiles).
To avoid this issue, we formulate the DRL problem through samples and parameters instead of statistics, and minimize the Cram ´ er distance between the value distributions.
Combining these solutions, we arrive at a distributional actor-critic with Cram ´ er distance as the value distribution loss.
On the other hand, many actor-critic methods suffer from data inefﬁciency and often include multistep algorithms like the λ -return algorithm
(Watkins, 1989) and off-policy updates, e.g. importance sampling.
Here we address this problem by adapting multi-step off-policy updates to the distributional perspective, by deﬁning a generalized form of the multi-step distributional Bellman targets.
Furthermore, we introduce a novel value-distribution learning method which we call SampleReplacement, denoted by SR( λ ).
We show that the expectation of the target distribution from SR( λ ) is equivalent to the scalar λ -return.
Additionally, we propose to parameterize the value distribution as a Gaussian mixture model (GMM).
When combining GMM with the Cram ´ er distance, we can derive an analytic solution and obtain unbiased sample gradients at a much lower computational cost compared to the method using Huber quantile loss.
Altogether, we call our framework GMAC (Gaussian mixture actor-critic).
We present experimental results to demonstrate that this framework successfully outperforms the baseline algorithm with scalar value function in discrete action space, and can be expanded to continuous action spaces without any architectural or algorithmic modiﬁcation.
Furthermore, we show that more accurate representation of value distributions is learned, with a less computational cost.
R ELATED W ORKS
Bellemare et al.
(2017a) has shown that the distributional Bellman operator derived from the distributional Bellman equation is a contractor in a maximal form of the Wasserstein distance.
Based on this point, Bellemare et al. (2017a) proposed a categorical distributional model, C51, which is later discussed to be minimizing the Cram´er distance in the projected distributional space (Rowland et al., 2018;
Bellemare et al., 2019).
Dabney et al.
(2018b)
proposed quantile regression-based models, QR-DQN, which parameterizes the distribution with a uniform mixture of Diracs and uses sample-based Huber quantile loss (Huber, 1964).
Dabney et al.
(2018a) later expand it further so that a full continuous quantile function can be learned through the implicit quantile network (IQN).
Yang et al.
(2019) then further improved the approximation of the distribution by adjusting the set of quantiles.
Rowland et al.
(2019) proposed expectile regression in place of quantile regression for learning categorical distribution to address the error in the Bellman target approximation.
Choi et al.
(2019) suggested parameterizing the value distribution using Gaussian mixture and minimizing the
Tsallis-Jenson divergence as the loss function on a value-based method.
Outside of RL, Bellemare et al.
(2017b) proposed to use Cram´er distance in place of Wasserstein distance used in WGAN due to its unbiasedness in sample gradients (Arjovsky et al., 2017).
There have been many applications of the distributional perspective, which exploit the additional information from value distribution.
Dearden et al.
(1998) modeled parametric uncertainty and
Morimura et al.
(2010a;b) designed a risk-sensitive algorithm using a distributional perspective, which can be seen as the earliest concept of distributional RL.
Mavrin et al.
(2019) utilized the idea of the uncertainty captured from the variance of value distribution.
Nikolov et al.
(2019) has also utilized the distributional representation of the value function by using information-directedsampling for better exploration of the value-based method.
While multi-step Bellman target was considered (Hessel et al., 2018), the sample-efﬁciency was directly addressed by combining multistep off-policy algorithms like Retrace( λ ) (Gruslys et al., 2017).
Just as C51 has been expanded deep RL to disitributional perspective, Barth-Maron et al.
(2018) studied a distributional perspective on DDPG (Lillicrap et al., 2016), an actor-critic method, by parameterizing a distributional critic as categorical distribution and Gaussian mixture model.
Singh et al.
(2020) has further expanded the work by using an implicit quantile network for the critic.
Several works (Duan et al., 2020; Kuznetsov et al., 2020; Ma et al., 2020) have proposed a distributional version of the soft-actor-critic (SAC) framework to address the error from over-estimating the value.
However, previous works concentrated on extending a speciﬁc actor-critic framework to the distributional setting.
Therefore, we aim to suggest methods which may be easily adopted in the process of expanding a scalar value methods to a distributional perspective, along with an attempt to address the previously mentioned issues present in the value-based ditributional algorithms.
D ISTRIBUTIONAL R EINFORCEMENT L EARNING
We consider a conventional RL setting, where an agent’s interaction with its environment is described by a Markov Decision Process (MDP)
( X , A , R, P, γ ), where X and A are state and action spaces, R ( x, a ) is the stochastic reward function for a pair of state x and action a , P ( x (cid:48) | x, a ) is the transition probability of observing x (cid:48) given the pair ( x, a ) , and γ ∈ (0,1) is a time discount factor.
A policy π ( ·| x ) maps a state x ∈ X to a probability distribution over actions a ∈ A .
The objective of RL is to maximize the expected return, E [ G t ] where G t = (cid:80) ∞ t =
γ t R ( x t , a t ) is the sum of discounted rewards from state x t given a policy π at time t .
Then for any state x t , the value
V and state-action value Q under the given policy π can be deﬁned as
[Equation]
A recursive relationship in the value in terms of the reward and the random transition is described by the Bellman equation (Bellman, 1957) given by
[Equation]
where the ﬁrst expectation is calculated over a given state-action pair ( x, a ) and the second expectation is taken over the next possible states x (cid:48) ∼ P ( ·| x, a ) and actions a (cid:48) ∼ π ( ·| x ) .
DRL extends the Bellman equation to an analogous recursive equation, termed the distributional
Bellman equation (Morimura et al., 2010a;b; Bellemare et al., 2017a), using a distribution of the possible sum of discounted rewards Z
[Equation] [Equation]
where D = denotes having equal distributions and Q ( x, a ) =
E [ Z ( x, a )] .
Then Z is learned through distributional Bellman operator T π deﬁned as
[Equation]
where P π : Z → Z is a state transition operator under policy π , P π Z ( x, a ) D :
= Z ( x (cid:48) , a (cid:48) ) .
Analogously, the distributional Bellman optimality operator T can be deﬁned as
[Equation]
X
X ← r t + γ X //
Bellman operator
Z ( λ ) t ← (cid:80) mi = δ X
i // empirical distribution using m Diracs end end
The distributional Bellman operator has been proven to be a γ -contraction in a maximal form of
Wasserstein distance (Bellemare et al., 2017a), which has a practical deﬁnition given by
[Equation]
where U, V are random variables and F U , F V are their cumulative distribution functions (cdf).
However, unlike the distributional Bellman operator, the distributional Bellman optimality operator is not a contractor in any metric (Bellemare et al., 2017a).
Thus, the distance d p ( T Z 1 , T Z 2 ) between some random variables Z 1 , Z 2 may not converge to a unique solution.
This issue has been discussed in Bellemare et al.
(2017a), with an example of oscillating value distribution caused by a speciﬁc tie-breaker design of the argmax operator.
One way to remove this issue from consideration is by learning the value distributions via expected Bellman operator with a trainable stochastic policy and ﬁnding an optimal policy under principles of conservative policy iteration by Kakade & Langford (2002).
See Appendix A for more discussion.
A LGORITHM
In this section, we incrementally develop the building blocks of our proposed method.
First, we present an efﬁcient distributional version of λ -return algorithm called Sample-Replacement, denoted by SR( λ ).
Then, we show that minimizing the energy distance, which is equivalent to a speciﬁc form of the Cram ´ er distance, between Gaussian mixtures can be a better solution than using quantile regressions when working with distributional actor-critic and SR( λ ).
4. SR( λ ): S AMPLE -R EPLACEMENT FOR λ - RETURN D ISTRIBUTION
The actor-critic method is a temporal-difference (TD) learning method in which the value function, the critic, is learned through the TD error deﬁned by the difference between the TD target given by n -step return, G ( n ) t
[Equation]
and the current value estimate V ( x t ) .
A special case of TD method, called TD( λ ) (Sutton, 1988), generates a weighted average of n -step returns for the TD target, also known as the λ -return,
[Equation]
to mitigate the variance and bias trade-off between Monte Carlo and the TD(0) return to enhance data efﬁciency.
From an alternative perspective, equation 7 can be thought of as ﬁnding a TD target via taking the expectation of a random variable ˜ G whose sample space is the set of all n -step returns, { G (0) t , . . .
, G ( ∞ ) t } .
Then the probability distribution of ˜ G is given by
[Equation]
Similar to G ( n ) t , we deﬁne n -step approximation of the value distribution as
[Equation]
where E [ Z ( n ) t ] = G ( n ) t .
Then, a distributional analogy of equation 8 can be written as
[Equation]
where ˜ Z is a random variable whose sample space is a set of all n -step approximations, { Z (0) t , . .
, Z ( ∞ ) t } .
Unlike ˜ G , we cannot directly calculate an expectation over a set of random variables, ˜ Z .
Instead, we redeﬁne equation 10 in terms of cdfs:
[Equation]
F Z ( n ) t denotes the cdf of the n -step return Z ( n ) t , and ˜ F = { F Z (0) t , . .
. , F Z ( ∞ ) t } is a random variable over the set of F Z ( n ) t .
Then, for any z , we can successfully rewrite equation 11 as a linear combination of F Z ( n ) t
[Equation]
Let us deﬁne a random variable Z ( λ ) t that has E [ ˜ F ] as its cdf.
Then the expectation of Z ( λ ) t and the expectation of Z ( n ) t have an analogous relationship to equation 7 (see Appendix B), meaning that its behavior in expectation is equal to that of the λ -return.
Therefore, we treat the resulting random variable Z ( λ ) t as a distributional analogy of the λ -return.
We note that, in practice, collecting inﬁnite horizon trajectory is infeasible and thus we use a truncated sum (Cichosz, 1995; van Seijen et al., 2011):
[Equation]
Given a trajectory of length N , na ¨ ıvely speaking, ﬁnding Z ( λ ) t for each time step requires ﬁnding
N different Z ( n ) t .
As a result, we need to ﬁnd total of O ( N 2 ) different distributions to ﬁnd Z t ( λ ) for all states in the given trajectory.
The number of distributions to ﬁnd can be reduced to O ( N ) , in practice, by approximating the distribution of Z ( n ) t with a mixture of diracs, as described in Dabney et al. (2018b), because the Bellman operations can be applied to the same set of samples to ﬁnd Z ( n ) t for different t ’s. Then, the approximation takes the form of
[Equation]
where θ : X → R m is some parametric model.
To obtain the target samples for Z θ ( x t ) , we start with m samples of the last value distribution in the sampled trajectory.
Traversing through the sampled trajectory in a reversed order, we replace each sample with a new sample from the next state with a probability of 1 − λ .
Then the obtained set X m is equivalent to a set of samples from the approximated distribution of the λ -returns, Z ( λ ) t .
A more detailed description of the algorithm can be found in Algorithm 1.
For sample-based methods like the implicit quantile network, one can directly use this set X m as the target samples.
However, Rowland et al.
(2019) has shown that the quantiles predicted from the Huber quantile loss cannot be interpreted as samples, thus an imputation strategy is required to generate a distribution from the statistics.
We propose to minimize the Cram´er distance instead in which the predicted parameters can be samples or the parameters of the distribution itself.
4. C RAM ´ ER D ISTANCE
Let P and Q be probability distributions over R .
If we deﬁne the cdf of P, Q as F P , F Q respectively, the l p family of divergence between P and Q is
[Equation]
When p = 2 , it is termed the Cram´er distance.
The distributional Bellman operator in the evaluation setting is a | γ | 1 /p -contraction mapping in the Cram´er metric space (Rowland et al., 2019), whose worked out proof can be found in Appendix C.
A notable characteristic of the Cram´er distance is the unbiasedness of the sample gradient,
[Equation]
[Equation]
where X m := { X 1 , ..., X m } are samples drawn from P , ˆ P m := 1 m (cid:80) mi =
δ X i is the empirical distribution, and Q θ is a parametric approximation of a distribution.
The unbiased sample gradient makes it suitable to use Cram´er distance with stochastic gradient descent method and empirical distributions for updating the value distribution.
Sz ´ ekely (2002) showed that, in the univariate case, the squared Cram ´ er distance is equivalent to one half of energy distance ( l 22 ( P, Q )
= 12 E ( P, Q ) ) deﬁned as
[Equation]
where U, U (cid:48) and V, V (cid:48) are random variables that follow P, Q , respectively.
4. E NERGY
So far, we have described the components required to formulate a general setting of the suggested distributional extension to actor-critic methods.
Here, we take one step further to enhance the approximation accuracy and computational efﬁciency by considering the parameterized model of the value distribution as a Gaussian mixture model (Choi et al., 2019; Barth-Maron et al., 2018).
Following the same assumption used for equation 14 the approximation using Gaussian mixture is given using parametric models µ, σ, w : X → R K [Equation] , where
[Equation]
If random variables X
[Equation]
Y follow the distributions P, Q parameterized as GMM
[Equation]
the energy distance has the following closed-form
[Equation]
where δ ( U, V ) =
(cid:88)
[Equation]
Here, µ xi refers to the i th component for random variable X and same applies for σ and w .
With Gaussian mixtures, the closed-form solution of the energy distance deﬁned in equation 19 has a computational advantage over sample-based approximations like the Huber quantile loss.
When using GMM, the analytic approximation of equation 14 can be derived as
[Equation]
where µ nk refers to k th component of µ Z ( n ) t for simplicity of notation, n is sampled from Geo (1 − λ ) and the index k ∈ { 1 , . .
, K } is sampled proportional to the mixture weights { w 1 , . . .
, w K } .
This is equivalent to having a mixture of m Gaussians, thus we can simply perform sample replacement on the parameters ( µ, σ 2 ) , instead of realizations of the random variables as in equation 14.
Then, the loss function described in equation 19 can easily be applied.
When bringing all the components together, we have a distributional actor-critic framework with
SR( λ ) that minimizes the Cram´er distance between Gaussian mixture value distributions.
We call this method GMAC.
A brief sketch of the algorithm is shown in Appendix E.
S...
S S
T γ γ r 4 ∼ U{− 1 , 1 } r 5 ∼ N (0 , 0 .
[Figure]
E XPERIMENTS
In this section, we present experimental results for three different distributional versions of Proximal
Policy Optimization (PPO) with SR( λ ): IQAC (IQN + Huber quantile), IQAC-E (IQN + energy distance), and GMAC (GMM + energy distance), in the order of the progression of our suggested approach.
The performance of the scalar version of PPO with value clipping (Schulman et al., 2016) is used as the baseline for comparison.
Details about the loss function of each method can be found in Appendix D. For a fair comparison, we keep all common hyperparameters consistent across the algorithms except for the value heads and their respective hyperparameters (see Appendix F).
The results demonstrate three contributions of our proposed DRL framework: 1) the ability to correctly capture the multimodality of value distributions, 2) generalization to both discrete and continuous action spaces, and 3) signiﬁcantly reduced computational cost.
Representing Multimodality As discussed throughout Section 4, we expect minimizing the
Cram´er distance to produce a more appropriate depiction of a distribution compared to minimizing the Huber quantile loss.
First, we demonstrate this with a simple value regression problem for an MDP of ﬁve sequential states, as shown in Figure 2 (a).
The reward function r i of last two state
S i is stochastic, with r 4 from a uniform discrete distribution and r 5 from a normal distribution.
Then the value distribution of S 1 should be bimodal with expectation of zero (Figure 2 (b)).
In this example, minimizing the Huber quantile loss (labeled as IQ-naive) of a implicit quantile network underestimates the variance of S 1 due to conﬂation and does not capture the locations of the modes.
Applying an imputation strategy as suggested in Rowland et al.
(2019), improvement on the underestimation of variance can be seen.
On the other hand, minimizing the Cram´er distance converges to correct mode locations using both implicit quantile network and Gaussian mixture model, labeled as
IQE and GM respectively in the ﬁgure.
More details about the experimental setup and further results can be found in G. The comparison can be extended to more complex tasks such as the Atari games, of which a sample result is shown in Figure 1, and additional visualizations of the value distribution during the learning process from different games can be found in Appendix G.
So what can be achieved with correct modality?
By capturing the correct modes of the distribution, an additional degree of freedom on top of the expected value can be accurately obtained, from which richer information can be derived to distinguish states by their value distributions.
In particular, the extra information may be utilized as an intrinsic motivation in sparse-reward exploration tasks.
To demonstrate, we compare using Cram´er distance between value distributions as intrinsic reward to using TD error between scalar value estimates in a sparse reward environment of Montezuma’s
Revenge in Figure 2 (c), which shows a clear improvement in the performance.
Discrete and Continuous Action Spaces Experimental results from the ALE (Bellemare et al., 2013) and the PyBullet environments (Coumans & Bai, 2016–2020) show that our algorithm can be
[Figure]
applied to both discrete and continuous action spaces.
While GMAC shows overall improvement in 61 Atari games (Appendix G), we present the full learning curves of 3 Atari games in Figure 3 in which previous distributional methods have shown improvements over the scalar version.
One can observe that the results from GMAC show the most signiﬁcant improvement from the baseline.
All experiments are run over 5 random seeds with consistent hyperparameters.
In Figure 3, a solid line represents an average of mean scores over 100 recent episodes and the shaded area represents the standard deviation over the seeds.
For visual clarity, the plots are smoothed over 5M frames and 1.25M frames in Atari and PyBullet, respectively.
Non-smoothed learning curves on more tasks and the ﬁnal scores over the 61 ALE tasks can be found in Appendix G.
Computational Cost Table 1 shows the number of parameters and the number of ﬂoating-point operations (FLOPs) required for a single inference and update step of each agent.
We emphasize three points here.
First, the implicit quantile network requires more parameters due to the intermediate embeddings of random quantiles.
Second, the difference between the FLOPs for a single update in IQAC and IQAC-E indicates that the proposed energy distance requires less computation than the Huber quantile regression.
Last, the results for GMAC show that using Gaussian Mixtures can greatly reduce the cost even to match the numbers of PPO while having improved performance.
[Table]
C ONCLUSION
In this paper, we have developed the distributional perspective of the actor-critic framework which integrates the SR( λ ) method, Cram´er distance, and Gaussian mixture models for improved performance in both discrete and continuous action spaces at a lower computational cost.
Furthermore, we show that our proposed method can capture the correct modality in the value distribution, while the extension of the conventional method with the stochastic policy fails to do so.
Capturing the correct modality of value distributions can improve the performance of various policybased RL applications that exploit statistics from the value distribution.
Such applications may include training risk-sensitive policies and learning control tasks with sparse rewards that require heavy exploration, where transient information from the value distribution can give beneﬁt to the learning process.
We leave further development of these ideas as future works
Appendices A D ISCUSSION ON THE CHOICE OF P ROXIMAL P OLICY O PTIMIZATION AS A BASELINE
A general learning process of RL can be described using policy iteration, which consists of two iterative phases: policy evaluation and policy improvement (Sutton & Barto, 1998).
In policy iteration, the value function is assumed to be exact, meaning that given policy, the value function is learned until convergence for the entire state space, which results in a strong bound on the rate of convergence to the optimal value and policy (Puterman, 1994).
But the exact value method is often infeasible from resource limitation since it requires multiple sweeps over the entire state space.
Therefore, in practice, the value function is approximated, i.e. it is not trained until convergence nor across the entire state space on each iteration.
The approximate version of the exact value function method, also known as asynchronous value iteration , still converges to the unique optimal solution of the Bellman optimality operator.
However, the Bellman optimality only describes the limit convergence, and thus the best we can practically consider is to measure the improvement on each update step.
Bertsekas & Tsitsiklis (1996) have shown that, when we approximate the value function V π of some policy π with ˜ V , the lower bound of a greedy policy π (cid:48) is given by
[Equation]
(21) where ε = max x | ˜ V ( x ) − V π ( x ) | is the L ∞ error of value approximation ˜ V .
This means a greedy policy from an approximate value function guarantees that its exact value function will not degrade more than 2 γε 1 − γ .
However, there is no guarantee on the improvement, i.e. V π (cid:48)
( x ) > V π ( x ) (Kakade & Langford, 2002).
As a solution to this issue, Kakade & Langford (2002) have proposed a policy updating scheme named conservative policy iteration ,
[Equation]
which has an explicit lower bound on the improvement
[Equation]
where (cid:15) =
max x | E π (cid:48)
[ A π ( x, a )] | , A π ( x, a ) =
Q ( x, a ) − V ( x ) is the advantage function, η ( π ) denotes the expected sum of reward under the policy π ,
[Equation]
and L π old is the local approximation of η with the state visitation frequency under the old policy.
From the deﬁnition of distributional Bellman optimality operator in equation 5, one can see that the lower bound in equation 23 also holds when π (cid:48) is greedy with respect to the expectation of the value distribution, i.e., E x (cid:48) ∼ P
[ Z ( x (cid:48) , a (cid:48) )] .
Thus the improvement of the distributional Bellman update is guaranteed in expectation under conservative policy iteration, and the value functions are guaranteed to converge in distribution to a ﬁxed point by γ -contraction.
Schulman et al.
(2015) takes this further, suggesting an algorithm called trust region policy optimization (TRPO), which extends conservative policy iteration to a general stochastic policy by replacing α with Kullback-Leibler (KL) divergence between two policies,
[Equation] [Equation]
Then, the newly formed objective is to maximize the following, which is a form of constraint optimization with penalty:
[Equation]
where r ( π ) refers to the ratio r ( π )
=
π ( a t | x t ) ˜
π ( a t | x t ) .
However, in practice, choosing a ﬁxed penalty coefﬁcient β is difﬁcult and thus Schulman et al.
(2015) uses hard constraint instead of the penalty.
[Equation]
Schulman et al.
(2017) simpliﬁes the loss function even further in proximal policy optimization (PPO) by replacing KL divergence with ratio clipping between the old and the new policy with the following:
[Equation]
Thus, by using PPO as the baseline, we aim to optimize the value function via unique point convergence of distributional Bellman operator for a policy being approximately updated under the principle of conservative policy.
B E XPECTATION VALUE OF Z ( λ ) t
Continuing from equation 12, let us deﬁne a random variable that has a cumulative distribution function of E [ ˜ F Z ] as Z ( λ ) t .
Then, its cumulative distribution function is given by
[Equation]
If we assume that the support of Z ( λ ) t is deﬁned in the extended real line [ −∞ , ∞ ] ,
[Equation]
[Equation]
Thus we can arrive at the desired expression of E [ Z ( λ ) t ] =
G ( λ ) t .
C D ISTRIBUTIONAL B ELLMAN OPERATOR AS A CONTRACTION IN C RAM ´ ER METRIC SPACE
The Cram´er distance possesses the following characteristics (detailed derivation of each can be found in (Bellemare et al., 2017b)):
[Equation]
X, Y ) .
Using the above characteristics
[Equation]
the Bellman operator in l p divergence is l p
[Equation]
T π Z 1 ( x, a )
[Equation]
T π Z 2 ( x, a ))
[Equation]
l p (
[Equation]
( x
[Equation]
a )
[Equation]
1 ( x
[Equation]
a )
[Equation]
( x
[Equation]
a )
[Equation]
π
[Equation]
2 ( x
[Equation]
a )) ≤
[Equation]
a
[Equation]
≤
[Equation]
x
[Equation]
Substituting the result into the deﬁnition of the maximal form of the Cram´er distance yields
[Equation]
Thus the distributional Bellman operator is a | γ | 1 /p -contraction mapping in the Cram´er metric space, which was also proven in Rowland et al.
Similar characteristics as in equation 35 can be derived for the energy distance
[Equation]
showing that the distributional Bellman operator is a γ -contractor in energy distance
[Equation]
D L OSS F UNCTIONS
As in other policy gradient methods, our value distribution approximator models the distribution of the value, V ( x t ) , not the state-action value Q ( x t , a t ) , and denote it as Z θ ( x t ) parametrized with θ , whose cumulative distribution function is deﬁned as
[Equation] [Equation]
Below, we provide the complete loss function of value distribution approximation for each of the cases used in experiments (Section 5).
D. I MPLICIT Q UANTILE + H UBER QUANTILE (IQAC)
For the value loss of IQAC, we follow the general ﬂow of Huber quantile loss described in Dabney et al.
(2018b).
For two random samples τ
[Equation]
where Z ( λ ) t is generated via SR( λ ) and Z ( x ; τ ) =
F − 1 Z ( τ ) is realization of Z ( X ) given X = x and τ .
Then, the full loss function of value distribution is given by
[Equation]
[Equation]
where N and N (cid:48) are number of samples of τ, τ (cid:48) , respectively, and ρ is the Huber quantile loss
[Equation]
, with
[Equation] [Equation]
D. I MPLICIT Q UANTILE + E NERGY D ISTANCE (IQAC-E)
Here, we replace the Huber quantile loss in equation 42 with sample-based approximation of energy distance deﬁned in equation
[Equation]
[Equation]
D. G AUSSIAN M IXTURE + E NERGY D ISTANCE (GMAC)
Unlike the two previous losses, which use samples at τ generated by the implicit quantile network
Z θ ( x t ; τ ) , here we discuss a case in which the distribution is k -component Gaussian mixture parameterized with ( µ k , σ 2 k , w k ) .
Using the expectation of a folded normal distribution
[Equation]
we deﬁne δ between two Gaussian distributions as
[Equation]
Let Z θ ( x ) and Z ( λ ) t be Gaussian mixtures parameterized with ( µ θi , σ 2 θi , w θi ) , ( µ λj , σ 2 λj , w λj ) , respectively.
Then, the loss function for the value head is given by
[Equation]
E P SEUDOCODE OF GMAC
samples discounted
Compute parameters σ w i λ
1 } by
Compute advantage estimates ˆ A t et al
V end
Gather the data from E environments
Update policy using the clipped surrogate loss:
[Equation]
via stochastic gradient ascent.
Update value function using the energy distance between Gaussian mixtures (Equation 19):
[Equation]
=
[Equation]
via stochastic gradient descent.
end
The clipping function g ( (cid:15), A ) shown in the algorithm is deﬁned as follows:
[Equation]
Note that expectation of each loss is taken over each the collection of trajectories and environments.
F I MPLEMENTATION D ETAILS
For producing a categorical distribution, a softmax layer was added to the output of the network.
For producing a Gaussian mixture distribution, the mean of each Gaussian is simply the output of the network, the variance is kept positive by running the output through a softplus layer, and the weights of each Gaussian is produced through the softmax layer.
Since our proposed method takes an archi-
[Table]
tecture which only changes the value head of the original PPO network
[Table]
we base our hyperparameter settings from the original paper (Schulman et al., 2017).
We performed a hyperparameter search on a subset of variables: optimizers= { Adam, RMSprop } , learning rate= { 2.5e-4, 1.0e-4 } , number of epochs= { 4, 10 } , batch size= { 256, 512 }
[Table]
and number of environments= { 16, 32, 64, 128 } overatari tasks of Breakout, Gravitar
[Table]
and Seaquest
[Table]
for which there was no degrade in the performance of PPO
[Table]
[Table] [Table]
[Figure]
[Table]
[Figure]
G M ORE E XPERIMENTAL R ESULTS G.
[Figure]
[Figure]
Here we provide more details on the ﬁve-state MDP presented in Figure 2.
For each cases in the ﬁgure, 8 paramters/mixtures are used to epirically represent the distribution.
For the cases with the label ”na¨ıve”, the network outputs (quantiles, expectiles, etc.) are used to create the plot.
On the other hand, the cases with ”imputation” labels apply appropriate imputation strategy to the statistics to produce samples which are then used to plot the distribution. In all cases, the network is trained on full batches of all states for 8 different empirical samples with Adam optimizer and learning rate of 1e-3. Sample based energy-distance was used to calculate the distance from the true distribution for all cases. The tabular setting of ﬁve-state MDP in Figure 5 uses same exact experimental set-up as the one with neural network.
The only difference is that the statistics are from evenly spaced quantile/expectile targets τ ∈ [0 , 1] .
[Figure]
G. L EARNING CURVES
[Figure] [Figure]
[Table]
ChopperCommand
[Table]
CrazyClimber
[Table]
DemonAttack
[Table]
DoubleDunk
[Table]
FishingDerby
[Table]
Frostbite
[Table]
Hero
[Table]
IceHockey
[Table]
Jamesbond
[Table]
JourneyEscape
[Table]
Kangaroo
[Table]
Solaris 1,236.
2,579.20 SpaceInvaders 148.
2,228.30 StarGunner 664.
104,188.00 Tennis -23.
-5.90 TimePilot 3,568.
13,227.00 Tutankham 11, 167.
209.82 UpNDown 533.
129,243.70 Venture 0.
1,181.00 VideoPinball 16,256.
55,272.82 WizardOfWar 563.
11,388.00 YarsRevenge 3,092.
103,895.05 Zaxxon 32.