A D EEPER L OOK AT D ISCOUNTING M ISMATCH IN A CTOR -C RITIC A LGORITHMS
A BSTRACT
We investigate the discounting mismatch in actor-critic algorithm implementations from a representation learning perspective.
Theoretically, actor-critic algorithms usually have discounting for both actor and critic, i.e. , there is a γ t term in the actor update for the transition observed at time t in a trajectory and the critic is a discounted value function.
Practitioners, however, usually ignore the discounting ( γ t ) for the actor while using a discounted critic.
We investigate this mismatch in two scenarios.
In the ﬁrst scenario, we consider optimizing an undiscounted objective ( γ = 1) where γ t disappears naturally (1 t = 1) .
We then propose to interpret the discounting in critic in terms of a bias-variance-representation tradeoff and provide supporting empirical results.
In the second scenario, we consider optimizing a discounted objective ( γ < 1 ) and propose to interpret the omission of the discounting in the actor update from an auxiliary task perspective and provide supporting empirical results.
I NTRODUCTION
Actor-critic algorithms have enjoyed great success both theoretically (Williams, 1992;
Sutton et al., 2000; Konda, 2002; Schulman et al., 2015a) and empirically (Mnih et al., 2016; Silver et al., 2016;
Schulman et al., 2017; OpenAI, 2018).
There is, however, a longstanding gap between the theory behind actor-critic algorithms and how practitioners implement them.
Let γ, γ A , and γ C be the discount factors for deﬁning the objective, updating the actor, and updating the critic respectively.
Theoretically, no matter whether γ = 1 or γ < 1 , we should always use γ A = γ C = γ (Sutton et al., 2000; Schulman et al., 2015a) or at least keep γ A = γ C if Blackwell optimality (Veinott, 1969;
Weitzman, 2001) 1 is considered.
Practitioners, however, usually use γ A = 1 and γ C < 1 in their implementations (Dhariwal et al., 2017; Caspi et al., 2017; Zhang, 2018; Kostrikov, 2018; Achiam, 2018; Liang et al., 2018; Stooke & Abbeel, 2019).
Although this mismatch and its theoretical disadvantage have been recognized by Thomas (2014); Nota & Thomas (2020), whether and why it yields beneﬁts in practice has not been systematically studied.
In this paper, we empirically investigate this mismatch from a representation learning perspective.
We consider two scenarios separately.
Scenario 1: The true objective is undiscounted ( γ = 1 ) .
The theory prescribes to use γ A
= γ C = γ = 1 .
Practitioners, however, usually use γ A = γ = 1 but γ C < 1 , introducing bias .
We explain this mismatch with the following hypothesis:
Hypothesis 1.
γ C < 1 optimizes a bias-variance-representation trade-off.
It is easy to see that γ C < 1 reduces the variance in bootstrapping targets.
Besides this, we further provide empirical evidence showing that when γ C < 1 , it may become easier to ﬁnd a good representation compared to γ C = 1 .
Consequently, although using γ C < 1 introduces bias, it can facilitate representation learning.
For our empirical study, we make use of recently introduced techniques, such ﬁxed horizon temporal different learning (De Asis et al., 2019) and distributional reinforcement learning (Bellemare et al., 2017) to disentangle the various effects the discount factor has on the learning process.
Scenario 2: The true objective function is discounted ( γ < 1 ) .
Theoretically, there is a γ t term for the actor update on a transition observed at time t in a trajectory (Sutton et al., 2000; Schulman et al., 2015a).
Practitioners, however, usually ignore this term while using a discounted critic, i.e. , γ A = 1 and γ C
= γ < 1 are used.
We explain this mismatch with the following hypothesis:
Hypothesis 2.
Using γ C = γ < 1 and γ A = 1 is effectively similar to using γ C = γ A
= γ < 1 plus an auxiliary loss that sometimes facilitates representation learning.
Our empirical study involves implementing the auxiliary task explicitly by using an additional policy for optimizing the difference term between the loss of γ A = 1 and the loss of γ A < 1 .
We also design new benchmarking environments where the sign of the reward function is ﬂipped after a certain time step such that later transitions differ from earlier ones.
In that setting, γ A = 1 becomes harmful.
B ACKGROUND
[Table] deﬁne [Table]
Markov Decision Processes: We consider an inﬁnite horizon MDP with a ﬁnite state space S , a ﬁnite action space A , a bounded reward function r : S
→ R , a transition kernel p : S × S ×
A → [0 , 1] , an initial state distribution µ 0 , and a discount factor γ ∈ [0 , 1] .
The initial state S 0 is sampled from µ 0 .
At time step t , an agent in state
S t takes action A t ∼ π ( ·| S t ) , where π : A×S → [0 , 1] is the policy it follows.
The agent then gets a reward R t +1 .
= r ( S t ) and proceeds to the next state S t +1 ∼ p ( ·| S t , A t ) .
The return of the policy π at time step t is deﬁned as G t .
= (cid:80) ∞
i =1 γ i − 1 R t
+ i , which allows us to deﬁne the state value function v γπ ( S ) .
=
E [ G t | S t = s ] and the state-action value function q γπ ( s, a ) .
=
E [ G t | S t = s, A t = a ] .
We consider episodic tasks where we assume there is an absorbing state s ∞ ∈ S such that r ( s ∞ )
=
and p ( s ∞ | s ∞ , a ) = 1 holds for any a ∈ A .
When γ < 1 , v γπ and q γπ are always well deﬁned.
When γ = 1 , to ensure v γπ and q γπ are well deﬁned, we further assume ﬁnite expected episode length.
Let T πs be a random variable denoting the ﬁrst time step that an agent hits s ∞ when following π given S
= s .
We assume T max .
= sup π ∈ Π max s E [ T πs ] < ∞ , where π is parameterized by θ and Π is the corresponding function class.
Similar assumptions are also used in stochastic shortest path problems ( e.g. , Section 2.2 of Bertsekas & Tsitsiklis (1996)).
In our experiments, all the environments have a hard time limit of 1000 , i.e., T max = 1000 .
This is standard practice, classic RL environments also have an upper limit on their episode lengths ( e.g.
27k in Bellemare et al.
(2013, ALE)).
Following Pardo et al.
(2018), we add the (normalized) time step t in the state to keep the environment Markovian.
We measure the performance of a policy π with J γ ( π ) .
=
E S 0 ∼ µ
[ v γπ ( S 0 )] .
Vanilla Policy Gradient: Sutton et al.
(2000) compute ∇ θ J γ ( π ) as
[Equation]
(1) where d γπ ( s ) .
= (cid:80) ∞ t =0 γ t Pr( S t = s | µ 0 , p, π ) for γ < 1 and d γπ ( s ) .
=
E [ (cid:80) T πS 0 t =
Pr( S t = s | S 0 , p, π )] for γ = 1 . {+
+}
Note d γ π remains well-deﬁned for γ = 1 when T max < ∞ .
In order to optimize the policy performance J γ ( π ) , one can follow (1) and, at time step t , update θ t as
[Equation]
where α is a learning rate.
If we replace q γ C π with a learned value function, the update rule (2) becomes an actor-critic algorithm, where the actor refers to π and the critic refers to the learned approximation of q γ C π .
In practice, an estimate for v γ C π instead of q γ C π is usually learned.
Theoretically, we should have γ A
= γ C = γ .
Practitioners, however, usually ignore the γ t A term in (2), and use γ C < γ A = 1 .
What this update truly optimizes remains an open problem (Nota & Thomas, 2020).
TRPO and PPO: To improve the stability of actor-critic algorithms, Schulman et al.
(2015a)
propose Trust Region Policy Optimization (TRPO), based on the performance improvement lemma:
Lemma 1.
(Theorem 1 in Schulman et al.
(2015a))
For γ < 1 and any two policies π and π (cid:48) ,
[Equation]
where Adv γπ ( s, a ) .
=
E s (cid:48) ∼ p ( ·| s,a ) [ r ( s ) + γv γπ ( s (cid:48) ) − v γπ ( s )] is the advantage, (cid:15) ( π, π (cid:48) ) .
= max s D KL ( π ( ·| s ) ||
π (cid:48) ( ·| s )) , and D KL refers to the KL divergence.
To facilitate our empirical study, we ﬁrst make a theoretical contribution by extending Lemma 1 to the undiscounted setting.
We have the following lemma:
Lemma 2.
Assuming T max < ∞ , for γ = 1 and any two policies π and π (cid:48) ,
[Equation]
The proof of Lemma 2 is provided in the appendix.
A practical implementation of Lemmas 1 andis to compute a new policy θ via gradient ascent on the clipped objective:
[Equation]
where S t and A t are sampled from π θ old , and clip ( x ) .
= max(min( x, 1 + (cid:15) ) , 1 − (cid:15) ) with (cid:15) a hyperparameter.
Theoretically, we should have γ A = γ C , but practical algorithms like Proximal Policy
Optimization (Schulman et al., 2017, PPO) usually use γ C < γ A = 1 .
Policy Evaluation: We now introduce several policy evaluation techniques we use in our empirical study.
Let ˆ v be our estimate of v γπ .
At time step t , Temporal Difference learning (TD, Sutton (1988)) updates ˆ v as ˆ v ( S t ) ← ˆ v ( S t ) + α ( R t +1 + γ ˆ v ( S t +1 )
− ˆ v ( S t )) .
Instead of the inﬁnite horizon discounted return G t , De Asis et al.
(2019) propose to consider the H -step return G Ht .
= (cid:80)
Hi =1 R t
+ i .
Correspondingly, the H -step value function is deﬁned as v Hπ ( s ) .
=
E [ G Ht | S t =
s ] .
We let ˆ v H be our estimate of v
Hπ .
At time step t , De Asis et al.
(2019) use the following update rule to learn ˆ v H :
[Equation]
where ˆ v 0 ( s ) .
= 0 .
In other words, to learn ˆ v H , we need to learn { ˆ v i } i =1 ,...,H simultaneously.
De Asis et al.
(2019) call (4) Fixed Horizon Temporal Difference learning (FHTD).
As G t is a random variable, Bellemare et al.
(2017) propose to learn its full distribution instead of its expectation only, yielding the Distributional Reinforcement Learning (RL) paradigm.
They use a categorical distribution with 51 atoms uniformly distributed in [ − V max , V max ] to approximate the distribution of G t , where V max is a hyperparameter.
In this paper, we refer to the corresponding policy evaluation algorithm as C51 .
Methodology:
We consider MuJoCo robot simulation tasks from OpenAI gym
(Brockman et al., 2016) as our benchmark.
Given its popularity in understanding deep RL algorithms (Henderson et al., 2017; Ilyas et al., 2018; Engstrom et al., 2019; Andrychowicz et al., 2020) and designing new deep RL algorithms (Fujimoto et al., 2018; Haarnoja et al., 2018), we believe our empirical results are relevant to most practitioners.
We choose PPO, a simple yet effective and widely used algorithm, as the representative actor-critic algorithm for our empirical study.
PPO is usually equipped with generalized advantage estimation (Schulman et al., 2015b, GAE), which has a tunable hyperparameter ˆ γ .
The roles of γ and ˆ γ are similar.
To reduce its confounding effect, we do not use GAE in our experiments, i.e. , the advantage estimation for our actor is simply the TD error R t +1 + γ C ˆ v ( S t +1 )
− ˆ v ( S t ) .
The PPO pseudocode we follow is provided in Alg.
1 in the appendix
and we refer to it as the default PPO implementation.
We use the standard architecture and optimizer across all tasks, in particular, the actor and the critic do not share layers.
We conduct a thorough learning rate search in Ant for each algorithmic conﬁguration ( i.e. , a curve in a ﬁgure) and then use the same learning rate for all other tasks.
When using FHTD and C51, we also include H and V max in the grid search.
All details are provided in the appendix.
We report the average episode return of the ten most recent episodes against the number of interactions with the environment.
Curves are averages over ten independent runs with shaded regions indicating standard errors.
O PTIMIZING THE U NDISCOUNTED O BJECTIVE (S CENARIO 1)
When our goal is to optimize the undiscounted objective J γ =1 ( π ) , one theoretically grounded option is to use γ A
= γ C = γ = 1 .
By using γ A = 1 and γ C < 1 , practitioners introduce bias .
We ﬁrst empirically conﬁrm that introducing bias in this way indeed has empirical advantages.
A simple ﬁrst hypothesis is that γ C < 1 leads to lower variance in Monte Carlo return bootstrapping targets than γ C = 1 , it thus optimizes a bias-variance trade-off. However
[Figure] we further show that there are empirical advantages from γ C < 1 that cannot uniquely be explained by this bias-variance trade-off, indicating that there are additional factors beyond variance.
We then show empirical evidence identifying representation learning as an additional factor, leading to the bias-variance-representation trade-off from Hypothesis 1.
All the experiments in this section use γ A = [Figure]
[Figure] [Figure] Bias-variance trade-off: To investigate the advantages of using γ C < 1 , we ﬁrst test default PPO with γ C ∈ { 0 .
We ﬁnd that the best discount factor is always with γ C <and that γ C = 1 usually leads to a performance drop (Figure 1).
In default PPO, although the advantage is computed as the one-step TD error, the update target for updating the critic ˆ v ( S t ) is almost always a Monte Carlo return.
As there is no γ t A term in the actor update, we should theoretically use γ C = γ A
= 1 when computing the Monte Carlo return, which usually leads to high variance.
Consequently, a simple hypothesis for the empirical advantages of using γ C < 1 is a bias-variance trade-off.
We ﬁnd, however, that there is more at play.
Beyond bias-variance trade-off: To reduce the effect of γ C in controlling the variance, we benchmark PPO-TD (Algorithm 2 in the appendix).
PPO-TD is the same as default PPO except that the critic is updated with one-step TD, i.e. , the update target for ˆ v ( S t ) is now R t +1 + γ C ˆ v ( S t +1 ) .
Although Figure 2 shows that PPO-TD ( γ C = 1) outperforms PPO ( γ C = 1) by a large margin, indicating bias-variance may be at play, Figure 3 suggests that for PPO-TD as well, γ C < 1 is still preferable to γ C = 1 .
To further study this phenomenon, we benchmark PPO-TD-Ex (Algorithm 3 in the appendix), in which we provide N extra transitions to the critic by sampling multiple actions at any single state and using an averaged bootstrapping target.
The update target for ˆ v ( S t ) in PPO-TDEx is 1 N +1 (cid:80) Ni =
R it +1 + γ C ˆ v ( S it +1 ) .
Here R 0 t
+
and S
t +1 refer to the original reward and successor state.
To get R it +
and S it +1 for i ∈ { 1 , . .
, N } , we ﬁrst sample an action A it from the sampling policy, then reset the environment to S t , and ﬁnally execute A it to get R it +1 and S it +1 .
Importantly, we do not count those N extra transitions in the x -axis when plotting.
The advantage for the actor update in PPO-TD-Ex is estimated with R 0 t +
+ ˆ v ( S 0 t +1 ) − ˆ v ( S t )
regardless of γ C to further control the inﬂuence of variance.
The critic ˆ v is not trained on the extra successor states { S it +1 } i =1 ,...,N . [Figure] [Figure] So the quality of the prediction ˆ v ( S it +1 ) depends mainly on the generalization of ˆ v .
Intuitively, if ˆ v generalizes well [Figure] providing proper amount of transitions this way should improve or maintain the overall performance as they help reduce variance.
As shown by Figure 4, PPO-TD-Ex ( γ C = 0 . 99 ) roughly follows this intuition.
However, surprisingly, providing extra data to PPO-TD-Ex ( γ C = 1 ) leads to a signiﬁcant performance drop (Figure 5).
This drop suggests that the larger variance from the randomness of S t +1 is not the only issue when using γ C = 1 to train the critic.
The quality of the estimate ˆ v , at least in terms of making prediction on untrained states { S it +1 } 1 ,...,N , is lower when γ C = 1 is used than γ C < 1 .
In other words, the generalization of ˆ v is poor when γ C = 1 .
The curves for PPO-TD-Ex ( γ C = 0 . 995 ) are a mixture of γ C = 0 .
99 and γ C =
and are provided in Figurein the appendix.
In the undiscounted setting, we should theoretically have R t +
+ ˆ v ( S t +1 )
as the update target for the critic.
When γ C < 1 is used instead, the update target becomes R t +1 + γ C ˆ v ( S t +1 ) and the variance resulting from the randomness of S t +1 becomes less pronounced.
So here, γ C trades off bias with variance, similar to that in Monte Carlo return bootstrapping targets in default PPO.
We refer to this effect of γ C as variance control .
However, γ C can also affect the difﬁculty of learning a good estimate ˆ v for v γ C π ; we refer to this effect of γ C as learnability control (Lehnert et al., 2018;
Laroche & van Seijen, 2018; Romoff et al., 2019).
Inspired by the poor generalization of ˆ v when γ C = 1 , we investigate learnability control mainly from the representation learning perspective.
By representation learning, we refer to learning the bottom layers (backbone) of a neural network.
The last layer of the neural network is then interpreted as a linear function approximator whose features are the output of the backbone.
This interpretation of representation learning is widely used in the
RL community, see e.g. Jaderberg et al.
; Chung et al.
; Veeriah et al.
Bias-representation trade-off: To separate variance control and learnability control, ideally we should investigate the update target R t +1 + γ C , 1 ˆ v ( S t +1 ) , where ˆ v is trained to approximate v γ C , 2 π and γ C , 2 < γ C , 1 = 1 .
Learning an estimate ˆ v for v γ C , 2 π , however, implies to use the update target
R t +1 + γ C , 2 ˆ v ( S t +1 ) : the two effects of γ C , 2 then get mixed again.
To solve this dilemma, we consider the update target R t +
+ ˆ v H − 1 ( S t +1 ) , where ˆ v H − 1 ( S t +1 ) is trained to approximate v H − 1 π , i.e. , we use FHTD to train the critic in PPO, which we refer to as PPO-FHTD
(Algorithm 4 in the appendix).
PPO-FHTD implements γ C , 1 = 1 directly, and manipulating H changes the horizon of the policy evaluation problem, which is also one of the effects of manipulating γ C , 2 .
We test two parameterizations for PPO-FHTD to investigate representation learning.
In the ﬁrst parameterization, to learn v Hπ , we parameterize { v iπ } i =1 ,...,H as H different heads over the same representation layer (backbone).
In the second parameterization, we always learn { v i π } i =1 ,...,as 1024 different heads over the same representation layer, whatever H we are interested in.
To approximate v Hπ , we then simply use the output of the H -th head.
A diagram (Figure 13) in the appendix further illustrates the difference between the two parameterizations.
Figure 6 shows that by tuning H for FHTD, PPO-FHTD with the ﬁrst parameterization matches or exceeds the performance of PPO-TD ( γ C < 1) in most tasks, and that the best H is always [Figure] [Figure] smaller than 1024 .
Theoretically, as long as we use an H ≥ T max = 1000 , we always have v Hπ ( s ) ≡ v γ =1 π ( s ) .
Figure 6 shows that the performance of PPO-FHTD ( H = 1024) is very close to PPO-TD ( γ C = 1) , indicating that learning { v iπ } i =1 ,..., 1023 is not an additional overhead for the network in terms of learning v H =1024 π , i.e. , increasing H does not pose additional challenges in terms of network capacity.
However, Figure 7 suggests that for the second parameterization,
H =is almost always among the best choices of H .
Comparing Figures 6 and 7, we conclude that in the tested domains, learning v Hπ with different H requires different representations.
This suggests that we can interpret the results in Figure 6 as a bias-representation trade-off .
Using a larger H is less biased but representation learning may become harder due to the longer policy evaluation horizon.
Consequently, an intermediate H achieves the best performance in Figure 6.
As reducing H cannot bring in advantages in representation learning under the second parameterization, the less biased H , i.e. , the larger H , usually performs better in Figure 7.
Overall, γ C optimizes a bias-representation trade-off by changing the policy evaluation horizon H .
[Figure]
We further conjecture that representation learning may be harder for a longer horizon because good representations can become rarer.
We provide a simulated example to support this.
Consider policy evaluation on the simple Markov Reward Process (MRP) from Figure 8.
We assume the reward for each transition is ﬁxed and is randomly generated in [0 , 1] .
Let x s ∈ R K be the feature vector for a state s ; we set its i -th component as x s [ i ] .
= tanh( ξ ) , where ξ is a random variable uniformly distributed in [ − 2 , − 2] .
We chose this feature setup as we use tanh as the activation function in our PPO.
We use X ∈ R N × K to denote the feature matrix.
To create state aliasing (McCallum, 1997), which is common under function approximation, we ﬁrst randomly split the N states into S 1 and S 2 such that |S 1 | = αN and |S 2 |
=
(1 − α ) N , where α is the proportion of states to be aliased.
Then for every s ∈ S 1 , we randomly select an ˆ s ∈ S 2 and set x s ← x ˆ s .
Finally, we add Gaussian noise N (0 , 0 .
1 2 ) to each element of X .
We use N = 100 and K = 30 in our simulation and report the normalized representation error (NRE) as a function of γ .
For a feature matrix X , the NRE is computed analytically as
NRE (
[Equation]
, where v γ is the analytically computed true value function of the MRP.
We report the results in Figure 9, where each data point is averaged over 10 4 randomly generated feature matrices ( X ) and reward functions.
In this MRP, the average representation error becomes larger as γ increases, which suggests that learning a good representation under a large γ and state aliasing may be harder than with a smaller γ .
We report the unnormalized representation error in Figure 17 in the appendix, where the trend is much clearer.
Overall, though we do not claim that there is a monotonic relationship between the discount factor and the difﬁculty of representation learning, our empirical study does suggest that representation learning is a key factor at play in the misuse of the discounting in actor-critic algorithms, beyond NRE 40.0% aliased 0. 50.0% aliased 0. 60.0% aliased 0.
70.0% aliased 0. 80.0% aliased 0. 90.0% aliased the widely recognized bias-variance trade-off.
In the appendix, we provide additional experiments involving distributional RL to further support the bias-variance-representation trade-off hypothesis, under the assumption that the beneﬁts of distributional RL comes mainly from the improved representation learning (Bellemare et al., 2017; Munos, 2018;
Petroski Such et al., 2019)
[Figure]
O PTIMIZING THE D ISCOUNTED O BJECTIVE (S CENARIO 2)
[Figure] n [Figure] When our goal is to optimize the discounted objective J γ< 1 ( π ) , theoretically we should have the γ t A term in the actor update and use γ C < 1 .
Practitioners, however, usually ignore this γ t A ( i.e. , set γ A = 1 ), introducing bias .
Figure 10 shows that even if we use the discounted return as the performance metric, the biased implementation of PPO still outperforms the theoretically grounded implementation DisPPO in the domains we tested.
Here PPO refers to the default PPO implementation where γ A = 1 , γ C = γ < 1 , and DisPPO (Alg. 6 in the appendix) adds the missing γ t A term in PPO by using γ A = γ C = γ < 1 .
We propose to interpret the empirical advantages of PPO over
DisPPO with Hypothesis 2.
For all experiments in this section, we use γ C = γ < 1 .
An auxiliary task perspective: The biased policy update implementation of (2) ignoring γ t A can be decomposed into two parts as ∆ t = γ t ∆ t + (1 − γ t )∆ t , where ∆ t .
=
q γ C π ( S t , A t ) ∇ θ log π
( A t | S t ) .
We propose to interpret the difference term between the biased implementation ( ∆ t ) and the theoretically grounded implementation ( γ t ∆ t ), i.e. , the (1 − γ t ) q γ C π ( S t , A t ) ∇ θ log π
( A t | S t ) term, as the gradient of an auxiliary objective with a dynamic weighting 1 − γ t .
Let J s,µ ( π ) .
= (cid:80) a π ( a | s )
q γµ ( s, a ) ; we have ∇ θ J s,µ ( π ) | µ =
π = E a ∼ π ( ·| s )
[ q γ π ( s, a ) ∇ θ log π ( a | s )] .
This objective changes every time step (through µ ).
Inspired by the decomposition, we augment PPO with this auxiliary task, yielding
AuxPPO (Algorithm 7 and Figure 13 in the appendix).
In AuxPPO, we have two policies π and π (cid:48) parameterized by θ and θ (cid:48) respectively.
The two policies are two heads over the same neural network backbone, where π is used for interaction with the environment and π (cid:48) is the policy for the auxiliary task.
AuxPPO optimizes θ and θ (cid:48) simultaneously by considering the following joint loss
[Equation]
[-= (cid:80) ∞ t =
 γ t min (cid:110)-]S
[Equation]
, where S t and A t are obtained by executing θ old .
We additionally synchronize θ (cid:48) with θ periodically to avoid an off-policy learning issue.
Flipped rewards: Besides AuxPPO, we also design novel environments with ﬂipped rewards to investigate Hypothesis 2.
Recall we include the time step in the state, this allows us to simply create a new environment by deﬁning a new reward function r (cid:48)
( s, t ) .
= r ( s ) I t ≤ t 0 − r ( s )
I t>t 0 , where I is the indicator function.
During an episode, within the ﬁrst t 0 steps, this new environment is the same as the original one.
After t 0 steps, the sign of the reward is ﬂipped.
We select t 0 such that [Figure] γ t 0 is sufﬁciently small, e.g. , we deﬁne t 0 .
=
min t { γ t < 0 . 05 } .
With this criterion for selecting t 0 , the later transitions ( i.e. , transitions after t 0 steps) have little inﬂuence on the evaluation objective, the discounted return.
Consequently, the later transitions affect the overall learning process mainly through representation learning.
DisPPO rarely makes use of the later transitions due to the γ t A term in the gradient update.
AuxPPO makes use of the later transitions only through representation learning.
PPO exploits the later transitions for representation learning and the later transitions also affect the control policy of PPO directly
[Figure] Results : When we consider the original environments, Figure 11 shows that in 8 out 12 tasks, PPO outperforms DisPPO, even if the performance metric is the discounted episodic return.
In all those 8 tasks, by using the difference term as an auxiliary task, AuxPPO is able to improve upon DisPPO.
In 6 out of those 8 tasks, AuxPPO is able to roughly match the performance of PPO at the end of training.
For γ ∈ { 0 . 93 , 0 . 9 } in Ant , the improvement of AuxPPO is not clear and we conjecture that this is because the learning of the π -head (the control head) in AuxPPO is much slower than the learning of π in PPO due to the γ t C term.
Overall, this suggests that the beneﬁt of PPO over DisPPO comes mainly from representation learning. When we consider the environments with ﬂipped rewards, PPO is outperformed by DisPPO and AuxPPO by a large margin in 11 out of 12 tasks.
The transitions after t 0 steps are not directly relevant when the performance metric is the discounted return.
However, learning on those transitions may still improve representation learning provided that those transitions are similar to the earlier transitions, which is the case in the original environments.
PPO and AuxPPO, therefore, outperform DisPPO.
However, when those transitions are much different from the earlier transitions, which is the case in the environments with ﬂipped rewards, learning to control on them directly becomes distracting.
PPO, therefore, is outperformed by DisPPO.
Different from PPO, AuxPPO does not learn to control on later transitions.
Provided that the network has enough capacity, the control head π θ in AuxPPO will not be affected much by the irrelevant transitions.
The performance of AuxPPO is, therefore, similar to DisPPO.
To summarize, Figure 11 suggests that using γ A = 1 is simply an inductive bias that all transitions are equally important .
When this inductive bias is helpful for learning, γ A = 1 implicitly implements auxiliary tasks thus improving representation learning and the overall performance.
When this inductive bias is detrimental, however, γ A = 1 can lead to signiﬁcant performance drops.
AuxPPO appears to be a safe choice that does not depend much on the correctness of this inductive bias.
R ELATED W ORK
The mismatch in actor-critic algorithm implementations has been previously studied.
Thomas (2014) focuses on the natural policy gradient setting and shows that the biased implementation ignoring γ t A can be interpreted as the gradient of the average reward objective under a strong assumption that the state distribution is independent of the policy.
Nota & Thomas (2020) prove that without this strong assumption, the biased implementation is not the gradient of any stationary objective.
This does not contradict our auxiliary task perspective as our objective J s,µ ( π ) changes at every time step.
Nota & Thomas (2020) further provide a counterexample showing that following the biased gradient can lead to a policy of poor performance w.r.t.
both discounted and undiscounted objectives.
Both Thomas (2014) and Nota & Thomas (2020), however, focus on theoretical disadvantages of the biased gradient and regard ignoring γ t A as the source of the bias.
We instead regard the introduction of γ C < 1 in the critic as the source of the bias in the undiscounted setting and investigate its empirical advantages , which are more relevant to practitioners.
Moreover, our representation learning perspective for investigating this mismatch is to our knowledge novel.
Although we propose the bias-variance-representation trade-off, we do not claim that is all that γ affects.
The discount factor also has many other effects ( e.g. , Sutton (1995); Jiang et al.
Laroche et al.
Laroche & van Seijen (2018); Lehnert et al.
; Fedus et al.
Van Seijen et al.
; Amit et al.
(2020)), which we leave for future work.
In Scenario 1, using γ C < 1 helps reduce the variance. Variance reduction in RL itself is an active research area (see, e.g., Papini et al. Xu et al.
; Yuan et al.
Investigating those variance reduction techniques with γ C = 1 is a possibility for future work. Recently, Bengio et al. (2020) study the effect of the bootstrapping parameter λ in TD( λ ) in generalization. Our work studies the effect of the discount factor γ in representation learning in the context of the misuse of the discounting in actor-critic algorithms, sharing a similar spirit of Bengio et al.
C ONCLUSION
In this paper, we investigate the longstanding mismatch between theorists and practitioners in actorcritic algorithms from a representation learning perspective.
Although the theoretical understanding of policy gradient algorithms have recently been signiﬁcantly advanced (Agarwal et al., 2019; Wu et al., 2020), this mismatch has drawn little attention.
We hope our empirical study can help practitioners understand actor-critic algorithms better and therefore design more efﬁcient actor-critic algorithms in the setting of deep RL, where representation learning emerges as a major consideration.
We hope our empirical study can draw more attention to the mismatch, which could enable the community to ﬁnally close this longstanding gap.
A P ROOF OF L EMMA
Proof.
The proof is based on Appendix B in Schulman et al.
(2015a), where perturbation theory is used to prove the performance improvement bound (Lemma 1).
To simplify notation, we use a vector and a function interchangeably, i.e. , we also use r and µ 0 to denote the reward vector and the initial distribution vector.
J ( π ) and d π ( s ) are shorthand for J γ ( π ) and d γπ ( s ) with γ = 1 .
All vectors are column vectors.
Let S + be the set of states excluding s ∞ , i.e. , S + .
= S / { s ∞ } , we deﬁne P π ∈ R |S + |×|S + |
such that P π ( s, s (cid:48) ) .
= (cid:80) a π ( a | s ) p ( s (cid:48) | s, a ) .
Let G .
= (cid:80) ∞ t =0 P tπ .
According to standard Markov chain theories, G ( s, s (cid:48) ) is the expected number of times that s (cid:48) is visited before s ∞ is hit given S
= s .
T max < ∞ implies that G is well-deﬁned and we have G
=
( I − P π ) − 1 .
Moreover, T max < ∞ also implies ∀ s, (cid:80) s (cid:48) G ( s, s (cid:48) )
≤ T max , i.e. , || G || ∞ ≤ T max .
We have J ( π ) =
µ (cid:62)
Gr .
Let G (cid:48) .
= ( I − P π (cid:48) ) − 1 , we have
[Equation]
Let ∆ .
= P π (cid:48) − P π , we have
[Equation]
Left multiply by G (cid:48) and right multiply by G ,
[Equation]
Expanding
[Equation]
RHS recursively)
[Equation]
So we have
[Equation]
It is easy to see µ (cid:62)
G = d (cid:62) π and Gr = v π .
So µ
[Equation]
[Equation]
a | s ) − π
[Equation]
a | s ))
f ( s ) = 0 holds for any f that dependes only on s ) =
[Equation]
a π
[Equation]
by Bellman equation
[Equation]
We now bound µ (cid:62)
G (cid:48)
∆ G ∆ Gr .
First, |
[Equation]
| = |
[Equation]
where D TV is the total variation distance.
So
[Equation]
Moreover, for any vector x ,
[Equation]
which completes the proof.
Note this perturbation-based proof of Lemma 2 holds only for r : S → R .
For r : S ×
A → R , we can turn to the coupling-based proof as Schulman et al.
(2015a), which, however, complicates the presentation and deviates from the main purpose of this paper.
We, therefore, leave it for future work.
B E XPERIMENT D ETAILS B. M ETHODOLOGY
We use HalfCheetah , Walker , Hopper , Ant , Humanoid , and HumanoidStandup as our benchmarks.
We exclude other tasks as we ﬁnd PPO plateaus quickly there.
The tasks we consider have a hard time limit of 1000.
Following Pardo et al.
(2018), we add time step information into the state, i.e. , there is an additional scalar t/ 1000 in the observation vector.
Following Achiam (2018), we estimate the KL divergence between the current policy θ and the sampling policy θ old when optimizing the loss (3).
When the estimated KL divergence is greater than a threshold, we stop updating the actor and update only the critic with current data.
We use Adam (Kingma & Ba, 2014) as the optimizer and perform grid search for the initial learning rates of Adam optimizers.
Let α A and α C .
= βα
A be the learning rates for the actor and critic respectively.
For each algorithmic conﬁguration ( i.e. , a curve in a ﬁgure), we tune α A ∈ { 0 . 125 , 0 . 25 , 0 .
5 , 1 , 2 } × 3 · 10 −and β ∈ { 1 , 3 } with grid search in Ant with 3 independent runs maximizing the average return of the last 100 training episodes.
In particular, α A = 3 · 10 − 4 and β = 3 is roughly the default learning rates for the PPO implementation in Achiam (2018).
We then run this algorithmic conﬁguration with the best α A and α C in all tasks.
Overall, we ﬁnd after removing GAE, smaller learning rates are preferred.
When we use FHTD, we additionally consider
H ∈ { 16 , 32 , 64 , 128 , 256 , 512 , 1024 } in the grid search.
When we use C51, we additionally consider V max ∈ { 20 , 40 , 80 , 160 , 320 , 640 , 1280 , 2560 , 5120 , 10240 , 81920 , 163840 , 327680 } in the grid search.
We use PPO-TD with γ C = 0 .
99 as an example to study how the best hyperparameter conﬁguration in Ant transfers to other games. As shown in Figure 12, the best learning rates of Ant ( α A = 3 · 10 − 4 and β = 3 ) yields reasonably good performance in all the other games except
Humanoid . In the paper, we do not draw a conclusion from a single task. So an outlier is unlikely to affect the overall conclusion.
In the discounted setting, we consider only Ant , HalfCheetah and their variants.
For Walker2d , Hopper , and Humanoid , we ﬁnd the average episode length of all algorithms are smaller than t 0 , i.e. , the ﬂipped reward rarely takes effects.
For HumanoidStandup , the scale of the reward is too large.
To summarize, other four environments are not well-suited for the purpose of our empirical study.
Moreover, in the discounted setting, we performed the grid search of the learning rates for both Ant and HalfCheetah .
[Figure]
B. A LGORITHM D ETAILS
The pseudocode of all implemented algorithms are provide in Algorithms 1 - 7 with their architectures illustrated in Figure 13.
For hyperparameters that are not included in the grid search, we use the same value as Dhariwal et al.
; Achiam (2018).
In particular, for the rollout length, we set
K = 2048 .
For the optimization epochs, we set K opt = 320 .
For the minibatch size, we set B = 64 .
For the maximum KL divergence, we set KL target = 0 .
We clip π θ ( a | s ) π θold ( a | s ) into [ − 0 .
We use N s = 51 supports for PPO-C51.
We use two-hidden-layer neural networks for function approximation.
Each hidden layer hashidden units and a tanh activation function.
The output layer of the actor network has a tanh activation function and is interpreted as the mean of an isotropic Gaussian distribution, whose standard derivation is a global state-independent variable as suggested by Schulman et al.
(2015a).
Algorithm Input: θ, ψ : parameters of π, ˆ v α A , α C : Initial learning rates of the Adam optimizers for θ, ψ K, K opt , B : rollout length, number of optimization epochs, and minibatch size KL target : maximum KL divergence threshold
S 0 ∼ µwhile True do
Initialize a buffer M θ old ← θ for i = 0 , . .
, K − 1 do
A i ∼ π θ old ( ·| S i )
Execute A i , get R
i +1 ,
S i +1 if S i +1 is a terminal state then m i ← 0 , S
i +1 ∼ µelse
m i ←end end
G K ← ˆ v ( S K ) for i =
K − 1 , . .
. , 0 do
G i ←
R i +1 +
γ C m i
G i +
Adv i ←
R i +
+ γ C
m i ˆ v ψ ( S i +1 )
− ˆ v ψ ( S i )
Store ( S i , A i , G i , Adv i ) in M end
Normalize Adv i in M as Adv i ← Adv i − mean ( { Adv i } ) std
( { Adv i } ) for o = 1 , . . .
, K opt do
Sample a minibatch { ( S i , A i , G i , Adv i ) } i =1 ,...,B from M
L ( ψ ) ← 12 B (cid:80) Bi
=
(ˆ v ψ ( S i ) − G i ) 2 /* No gradient through G i */
L ( θ ) ← 1 B (cid:80) B i =1 min { π θ
( A i | S i ) π θold
( A i | S i )
Adv i , clip ( π θ ( A i | S i ) π θold ( A i | S i ) )
Adv i }
Perform one gradient update to ψ minimizing L ( ψ ) with Adam if 1 B (cid:80) Bi
=1 log π θ old ( A i | S i ) − log π θ ( A i | S i ) <
KL target then
Perform one gradient update to θ maximizing L ( θ ) with Adam end end end
Input: while True do
Initialize a buffer M θ old ← θ for i = 0 , . .
, K − 1 do
A i ∼ π θ old ( ·| S i )
Execute A i , get R
i +1 ,
S i +1 if S i +1 is a terminal state then m i ← 0 , S
i +1 ∼ µelse m i ←end endfor
i =
K − 1 , . .
. , 0 do
Adv i ←
R i +
+ γ C
m i ˆ v ψ ( S i +1 )
− ˆ v ψ ( S i ) S (cid:48)
i ← S
i +1 , r i ←
R i +1 Store
( S i , A i , m i , r i , S (cid:48)
i , Adv i ) in M end
Normalize Adv i in M as Adv i ← Adv i − mean ( { Adv i } ) std
( { Adv i } ) for o = 1 , . . .
, K opt do
Sample a minibatch { ( S i , A i , m i , r i , S (cid:48)
i , Adv i ) } i =1 ,...,B from M y
i ← r i + γ C m i ˆ v ψ ( S (cid:48) i )
L ( ψ ) ← 12 B (cid:80) Bi
=
(ˆ v ψ ( S i ) − y i ) 2 /
* No gradient through y i */
L ( θ ) ← 1 B (cid:80) B i =1 min { π θ
( A i | S i ) π θold
( A i | S i )
Adv i , clip ( π θ ( A i | S i ) π θold ( A i | S i ) )
Adv i }
Perform one gradient update to ψ minimizing L ( ψ ) with Adam if 1 B (cid:80) Bi
=1 log π θ old ( A i | S i ) − log π θ ( A i | S i ) <
KL target then
Perform one gradient update to θ maximizing L ( θ ) with Adam end end end
Algorithm
Input: θ α r
S 0 ∼ µwhile True do
Initialize a buffer M θ old ← θ for i = 0 , . .
, K − 1 do for j = 0 , . . .
, N do
A ji ∼ π θ old ( ·| S i ) , R ji +1 ← r ( S i , A ji ) , S ji +1 ∼ p ( ·| S
i , A ji ) if S ji +1 is a terminal state then m ji ← 0 , S ji +1 ∼ µelse
m ji ←end end
S i +1 ← S
i + endfor i =
K − 1 , . .
. , 0 do
Adv i ← R
i +1 + γ C m
i ˆ v ψ ( S 0 i +1 ) − ˆ v ψ ( S 0 i ) for j = 0 , . . .
, N do
S (cid:48)
ji ← S ji + end
Store ( { S ji , A ji , m ji , r ji , S (cid:48) ji } j =0 ,...,N , Adv i ) in M end
Normalize Adv i in M as Adv i ← Adv i − mean ( { Adv i } ) std
( { Adv i } ) for o = 1 , . . .
, K opt do
Sample a minibatch { ( { S ji , A ji , m ji , r ji , S (cid:48) ji } j =0 ,...,N , Adv i ) } i =1 ,...,B from M y
i ← 1 N +1 (cid:80) Nj =
r ji + γ C m ji ˆ v ψ ( S (cid:48) ji )
L ( ψ ) ← 12 B (cid:80) Bi
=
(ˆ v ψ ( S 0 i ) − y i ) 2 /* No gradient through y i */
L ( θ ) ← 1 B (cid:80) Bi
=1 min { π θ
( A 0 i | S 0 i )
π θold ( A 0 i | S 0 i )
Adv i , clip ( π θ ( A 0 i | S 0 i )
π θold ( A 0 i | S 0 i ) )
Adv i }
Perform one gradient update to ψ minimizing L ( ψ ) with Adam if 1 B (cid:80) Bi
=1 log π θ old ( A 0 i | S 0 i ) − log π θ ( A 0 i | S 0 i ) <
KL target then
Perform one gradient update to θ maximizing L ( θ ) with Adam end end end
Input: θ, ψ : parameters of π, { ˆ v j } j =1 ,...,H α A , α C
, ψ B
S 0 ∼ µwhile True do
Initialize a buffer M θ old ← θ for i = 0 , . .
, K − 1 do
A i ∼ π θ old ( ·| S i )
Execute A i , get R
i +1 ,
S i +1 if S i +1 is a terminal state then m i ← 0 , S
i +1 ∼ µelse m i ←end endfor
i =
K − 1 , . .
. , 0 do
Adv i ←
R i +
+ m i
ˆ v
Hψ ( S i +1 )
− ˆ v
Hψ ( S i )
S (cid:48)
i ← S
i +1 , r i ← R i
+
Store ( S i , A i , m i , r i , S (cid:48)
i , Adv i ) in M end
Normalize Adv i in M as Adv i ← Adv i − mean ( { Adv i } ) std
( { Adv i } ) for o = 1 , . . .
, K opt do
Sample a minibatch { ( S i , A i , m i , r i , S (cid:48)
i , Adv i ) } i =1 ,...,B from M for j = 1 , . .
, H do y ji ← r i + m i ˆ v j − 1 ψ ( S (cid:48) i )) /
* ˆ v 0 ( S (cid:48) i ) ≡*/ end
L ( ψ ) ← 1 2 B (cid:80) B i =1 (cid:80)
H j =1 (ˆ v jψ
( S i ) − y ji ) 2 /* No gradient through y ji */
L ( θ ) ← 1 B (cid:80) Bi
=1 min { π θ
( A i | S i ) π θold
( A i | S i )
Adv i , clip ( π θ ( A i | S i ) π θold ( A i | S i ) )
Adv i }
Perform one gradient update to ψ minimizing L ( ψ ) with Adam if 1 B (cid:80) Bi
=1 log π θ old ( A i | S i ) − log π θ ( A i | S i ) <
KL target then
Perform one gradient update to θ maximizing L ( θ ) with Adam end end end Input:
∆ z .
= 2 V max N s − 1 , { z j .
=
− V max
+ ( j − 1)∆ z : j = 1 , . . .
N supports
S 0 ∼ µwhile True do
Initialize a buffer M θ old ← θ for i = 0 , . .
, K − 1 do
A i ∼ π θ old ( ·| S i )
Execute A i , get R
i +1 ,
S i +1 if S i +1 is a terminal state then m i ← 0 , S
i +1 ∼ µelse m i ←end endfor
i =
K − 1 , . .
. , 0 do
Adv i ←
R i +
+ m i γ C (cid:80) N s j =1 ˆ v jψ
( S i +1 )
z j − (cid:80) N s j =1 ˆ v jψ
( S i )
z
j
S (cid:48)
i ← S
i +1 , r i ← R i
+
Store ( S i , A i , m i , r i , S (cid:48)
i , Adv i ) in M end
Normalize Adv i in M as Adv i ← Adv i − mean ( { Adv i } ) std
( { Adv i } ) for o = 1 , . . .
, K opt do
Sample a minibatch { ( S i , A i , m i , r i , S (cid:48)
i , Adv i ) } i =1 ,...,B from M for i = 1 , . . .
, B do for j = 1 , . . .
, N s do z ij ← r i
+ m i
γ C z j end end for j = 1 , . . .
, N s do y ij ← (cid:80) N s k =1 [1 − | [ z ij ] V max − V max − z
j | ∆ z ] 10 ˆ v kψ ( S (cid:48)
i ) /
* [ x ] ul .
= min(max( x, l ) , u ) */ end
L ( ψ ) ← 1 B (cid:80) Bi
=1 (cid:80) N s j =1 − y ij log ˆ v jψ
( S i ) /
* No gradient through y ij */
L ( θ ) ← 1 B (cid:80) Bi
=1 min { π θ
( A i | S i ) π θold
( A i | S i )
Adv i , clip ( π θ ( A i | S i ) π θold ( A i | S i ) )
Adv i }
Perform one gradient update to ψ minimizing L ( ψ ) with Adam if 1 B (cid:80) Bi
=1 log π θ old ( A i | S i ) − log π θ ( A i | S i ) <
KL target then
Perform one gradient update to θ maximizing L ( θ ) with Adam end end end
Algorithm
Input: θ, ψ : parameters of π, ˆ v α A , α C : Initial learning rates of the Adam optimizers for θ, ψ
K, K opt , B : rollout length, number of optimization epochs, and minibatch size
KL target : maximum KL divergence threshold
S 0 ∼ µ 0 , t ←while True do
Initialize a buffer M θ old ← θ for i = 0 , . .
, K − 1 do
A i ∼ π θ old ( ·| S i ) , t i ← t
Execute A i , get R
i +1 ,
S i +1 if S i +1 is a terminal state then m i ← 0 , S i +1 ∼ µ 0 , t ←else m i ← 1 , t ← t +end end
G K ← ˆ v ( S K ) for i =
K − 1 , . .
. , 0 do
G i ←
R i +1 +
γ C m i
G i +
Adv i ←
R i +
+ γ C
m i ˆ v ψ ( S i +1 )
− ˆ v ψ ( S i )
Store ( S i , A i , G i , Adv i , t i ) in M end
Normalize Adv i in M as Adv i ← Adv i − mean ( { Adv i } ) std
( { Adv i } ) for o = 1 , . . .
, K opt do
Sample a minibatch { ( S i , A i , G i , Adv i , t i ) } i =1 ,...,B from M
L ( ψ ) ← 12 B (cid:80) Bi
=
(ˆ v ψ ( S i ) − G i ) 2 /* No gradient through G i */
L ( θ ) ← 1 B (cid:80) B i =1 γ t i A min { π θ
( A i | S i ) π θold
( A i | S i )
Adv i , clip ( π θ ( A i | S i ) π θold ( A i | S i ) )
Adv i }
Perform one gradient update to ψ minimizing L ( ψ ) with Adam if 1 B (cid:80) Bi
=1 log π θ old ( A i | S i ) − log π θ ( A i | S i ) <
KL target then
Perform one gradient update to θ maximizing L ( θ ) with Adam end end end
Input: θ, θ (cid:48) , ψ : parameters of π, π (cid:48) , ˆ v α A , α C
Initial learning rates the for θ, ψ
, B
S 0 ∼ µ , t ←while True do
Initialize a buffer M θ old ← θ, θ (cid:48) ← θ for i = 0 , . .
, K − 1 do
A i ∼ π θ old ( ·| S i ) , t i ← t
Execute A i , get R
i +1 ,
S i +1 if S i +1 is a terminal state then m i ← 0 , S i +1 ∼ µ 0 , t ←else m i ← 1 , t ← t +end end
G K ← ˆ v ( S K ) for i =
K − 1 , . .
. , 0 do
G i ←
R i +1 +
γ C m i
G i +
Adv i ←
R i +
+ γ C
m i ˆ v ψ ( S i +1 )
− ˆ v ψ ( S i )
Store ( S i , A i , G i , Adv i , t i ) in M end
Normalize Adv i in M as Adv i ← Adv i − mean ( { Adv i } ) std
( { Adv i } ) for o = 1 , . . .
, K opt do
Sample a minibatch { ( S i , A i , G i , Adv i , t i ) } i =1 ,...,B from M
L ( ψ ) ← 12 B (cid:80) Bi
=
(ˆ v ψ ( S i ) − G i ) 2 /* No gradient through G i */
L ( θ, θ (cid:48) ) ← 1 B (cid:80) Bi
=1 γ t i C min { π θ ( A i | S i )
π θ old ( A i | S i )
Adv i , clip ( π θ ( A i | S i )
π θ old ( A i | S i ) )
Adv i } + (1 − γ t i C ) min
{ π θ (cid:48)
( A i | S i ) π θold
( A i | S i )
Adv i , clip ( π θ (cid:48)
( A i | S i ) π θold
( A i | S i ) )
Adv i }
Perform one gradient update to ψ minimizing L ( ψ ) with Adam if 1 B (cid:80) Bi
=1 log π θ old ( A i | S i ) − log π θ ( A i | S i ) <
KL target then
Perform one gradient update to θ, θ (cid:48) maximizing L ( θ, θ (cid:48) ) with Adam end end end [Figure]
C A DDITIONAL E XPERIMENTAL R ESULTS C. D ISTRIBUTIONAL RL
Hypothesis 1 and the previous empirical study suggest that representation learning may be the main bottleneck of PPO-TD ( γ C = 1 ).
To further support this, we benchmark PPO-C51 ( γ C = 1) (Algorithm 5 in the appendix), where the critic of PPO is trained with C51.
C51 is usually considered to improve representation learning by implicitly providing auxiliary tasks (Bellemare et al., 2017;
Munos, 2018; Petroski Such et al., 2019). Figure 14 shows that training the critic with C51 indeed leads to a performance improvement and PPO-C51 ( γ C = 1) sometimes outperforms PPO-TD ( γ C < 1 ) by a large margin. Figure 15 further shows that when V max is optimized for PPO-C51, the beneﬁt for using γ C < 1 in PPO-C51 is less pronounced than that in PPO-TD, indicating the role of γ C < 1 and distributional learning may overlap. Figures 6, 7, & 9, suggest that the overlapping is representation learning.
C. O THER C OMPLEMENTARY R ESULTS
Figure 16 shows how PPO-TD-Ex ( γ C = 0 . 995 ) reacts to the increase of N . Figure 17 shows the unnormalized representation error in the MRP experiment. Figure 18 shows the average episode length for the Ant environment in the discounted setting.
For HalfCheetah , it is always 1000.
D L ARGER V ERSION OF F IGURES
[Figure]
un
[Figure]
n
[Figure]
[Figure]
un d i s c o un t e d r e t u r n
[Figure]
[Figure]
un
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
un
[Figure]
u
[Figure]
n
[Figure] [Figure] un [Figure]
n
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]