A SYM Q: A SYMMETRIC Q- LOSS TO MITIGATE OVERESTIMATION BIAS IN OFF - POLICY REINFORCEMENT LEARNING
A BSTRACT
It is well known that off-policy deep reinforcement learning algorithms suffer from overestimation bias in value function approximation.
Existing methods to reduce overestimation bias often utilize multiple value function estimators.
Consequently, these methods have a larger time and memory consumption.
In this work, we propose a new class of policy evaluation algorithms dubbed, AsymQ , that use asymmetric loss functions to train the Q-value network.
Departing from the symmetric loss functions such as mean squared error (MSE) and Huber loss on the Temporal difference (TD) error, we adopt asymmetric loss functions of the TD-error to impose a higher penalty on overestimation error.
We present one such AsymQ loss called Softmax MSE (SMSE) that can be implemented with minimal modifications to the standard policy evaluation.
Empirically, we show that using SMSE loss helps reduce estimation bias, and subsequently improves policy performance when combined with standard reinforcement learning algorithms.
With SMSE, even the Deep Deterministic Policy Gradients (DDPG) algorithm can achieve performance comparable to that of state-of-the-art methods such as the Twin-Delayed DDPG (TD3) and Soft Actor Critic (SAC) on challenging environments in the OpenAI Gym MuJoCo benchmark.
We additionally demonstrate that the proposed SMSE loss can also boost the performance of Deep
Q learning (DQN) in Atari games with discrete action spaces
I NTRODUCTION
Learning an accurate value function in Deep Reinforcement Learning (DRL) is crucial; the value function of a policy is not only important for policy improvement (Sutton & Barto, 2018) but also useful for many downstream applications such as risk-aware planning (Kochenderfer, 2015) and goal-based reachability planning (Nasiriany et al., 2019).
However, most off-policy DRL algorithms are accompanied by estimation bias in policy evaluation and how to remove this estimation bias has been a long-standing challenge.
In this work, we revisit the estimation bias problem in offpolicy DRL from a new perspective and propose a lightweight modification of the standard policy evaluation algorithm to mitigate estimation bias.
Value estimation bias in DRL Thrun & Schwartz (1993) firstly shows that maximization of a noisy value estimation consistently induces overestimation bias in Q-learning, where the learned value function overestimates the learned policy, i.e., the prediction from the learned value function is higher than the ground truth value of the policy.
Several methods have been proposed to reduce estimation bias in policy evaluation and policy improvement.
Hasselt (2010); Van Hasselt et al.
(2016) propose double Q-learning, which trains two independent estimators to suppress overestimation.
In the continuous state-action space setting, Fujimoto et al.
(2018) shows the existence of overestimation in popular Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015) and proposes Twin Delayed Deep Deterministic Policy Gradient (TD3) to alleviate the overestimation issue, which utilizes the minimum value estimation from two critic networks to fit Q value function.
However, these approaches that involve multiple value function estimators together with a minimum operator, may succumb to underestimation bias (Ciosek et al., 2019; Pan et al., 2020), where the value prediction of the learned function is lower than the real policy performance.
Lyu et al. Wu et al.
Wei et al. Kuznetsov et al.
; Chen et al. ; Liang et al.
Lee et al.
(2021) propose new critic update schemes to reduce the estimation bias.
However, all these methods require multiple actors or an ensemble of critics and usually involve other convoluted tricks that often incur additional computation and memory overheads to improve policy performance.
In this work, we explore an efficient approach that reduces value estimation bias without incurring extra computational costs.
Loss function in policy evaluation The policy evaluation in DRL is typically based on the Bellman update equation (Bellman, 1966) where the discrepancy between the predicted and target values are minimized.
The target value is the combination of the immediate reward and subsequent value function prediction on future states.
There are two main components in policy evaluation based on Bellman temporal difference (TD) learning, the fitting target that acts as supervised signals to train neural networks and the loss function that serves as an objective metric.
Most of the existing work in reducing estimation bias focuses on the first component, and typically try to construct a more robust target value, such as a lagged target network (Mnih et al., 2015) and ensemble multiple value networks (Fujimoto et al., 2018).
The choice of the loss function used for value-function fitting somehow receives much less attention.
In practice, most RL algorithms choose symmetric mean square error (MSE) or Huber loss (Patterson et al., 2022) as a metric in fitting value functions.
Once zero MSE loss is reached for each state, the prediction of the value network can match policy performance exactly.
In this work, we propose a novel approach based on modifying the loss function to control estimation bias in DRL.
We discover that the optimization landscape used to train value networks, governed by the loss functions, plays a crucial role in policy evaluation.
A proper choice of the value-fitting loss function can effectively control estimation bias.
In particular, we show asymmetric functions can be used for policy evaluation to reduce estimation bias and propose a class of algorithms called AsymQ .
We specifically evaluate one family of AsymQ loss functions called SoftMax MSE (SMSE) to learn the Q value function in both continuous and discrete action environments, but we validate the benefit of using other AsymQ loss functions as well.
We find asymmetric loss functions can inject inductive bias to the learning process and thereby control estimation bias present in DRL policy evaluation updates.
We highlight the intuition behind our approach in Fig 1, where asymmetry can assign different weights for both overestimated and underestimated states and help alleviate bias in policy evaluation.
Notably, compared with existing methods (Hasselt, 2010; Fujimoto et al., 2018), our approach needs only one critic and actor and negligible overhead computational and memory cost.
We summarize our contributions as follows: (1) We show asymmetric loss functions can be used for learning the value function, and introduce a simple asymmetric loss for policy evaluation, namely,
Softmax MSE (SMSE) parameterized by a temperature parameter, that can be easily combined with existing RL algorithms.
(2) We further propose an auto-tuning algorithm for the temperature to reduce the burden of tuning parameters for our proposed approach.
(3) We show that SMSE can significantly reduce estimation bias and improve the performance of popular baseline algorithms such as DDPG on MuJuCo environments and DQN on Atari games.
To the best of our knowledge, this is the first algorithm that achieves such competitive performance without the need for other tricks such as multiple critic networks and crafted exploration methods that often incur extra computational and memory costs.
P RELIMINARIES : V ALUE - BASED D EEP R EINFORCEMENT L EARNING
The interaction of the RL agent with the environment can be formalized as a Markov Decision
Process (MDP).
The MDP is represented as a tuple ( S , A , p, r, γ ) , where S , A represents the set of states and actions respectively, p represents the transition probabilities, r ( s , a ) the reward function, and γ represents the discount factor.
The goal of an RL agent is to learn a behavior policy π ϕ ( a t | s t ) such that the expected return J ( π ϕ ) =
E [ (cid:80) ∞ t =0 γ t r ( s t , a t ) | π ϕ ] is maximized.
In RL the value function is usually learned based on temporal difference (TD) learning, an update scheme based on the Bellman equation Sutton & Barto (2018); Bellman (1966).
The Bellman equation formulates the value of a state-action pair ( s , a ) in terms of the value of subsequent state-action pairs ( s ′ , a ′ ) :
[Equation]
where Q π ( s , a )
=
E [ (cid:80) ∞ t =0 γ t r ( s t , a t ) | s , a ] , denotes the expected return when performing action a in state s at t = 0 and following policy π for all the subsequent states.
Q π is also known as the value function or the critic.
A policy can be represented and learned explicitly as done in actor-critic methods or inferred implicitly from the value function in methods such as Q-learning.
In continuous control, the parameterized policy π ϕ can be updated through the deterministic policy gradient algorithm Silver et al.
[Equation]
In DRL, the value function is estimated by a neural network, which we denote as Q θ ( s , a ) parameterized by network parameters θ .
Based on temporal difference learning, existing works update θ by minimizing the mean squared TD-error, θ ← θ − α ∇ θ E s , a , [ δ ( s , a ) 2 ] where the TD-error δ ( s , a ) is defined by
[Equation]
Here Q θ ′ is usually chosen as a frozen target network.
The weights θ ′ are usually updated in an exponential moving average approach, i.e., θ ′ ← τθ
+ (1 − τ ) θ ′ for some 0 < τ < 1 .
We start our investigation by revisiting estimation bias and necessary conditions for fitting the Q value function in Section 3, and show there exist many asymmetric loss functions that also satisfy these conditions.
We demonstrate the advantage of using asymmetric loss functions by examining a specific family of AsymQ loss functions that we term as SoftmaxMSE (SMSE) in Section 4 and provide more empirical results and ablation studies in Section 5.
E STIMATION BIAS AND ASYMMETRIC LOSS FUNCTION
Estimation bias and supervised target
The overestimation bias in DRL is believed to be due to its greedy update and the noise present in training.
The overestimation issue was initially described in Thrun & Schwartz (1993), where the value estimation is updated with a greedy target y
= r
+ γ max a ′ Q ( s ′ , a ′ ) .
When there exists noise or approximation error ϵ in the Q value function, the maximum over it along with the error will generally be greater than the true maximum, namely, E ϵ [max a ′ ( Q ( s ′ , a ′ ) + ϵ )] ≥ max a ′ Q ( s ′ , a ′ ) .
Similar results are also found in the continuous setting.
We briefly summarize these results in the following lemma (Fujimoto et al., 2018)
(more details are provided in Appendix A).
Lemma 1.
For a pair of actor-critic { π ϕ , Q θ } , assume one step update based on policy gradient results in a new policy π ¯ ϕ .
Under mild assumptions, the value estimate will be overestimated:
E [ Q θ ( s , π ¯ ϕ ( s ))] ≥ E [ Q
[Equation]
( s , π ¯ ϕ (
[Equation]
The overestimation bias turns out to be a serious problem in DRL.
The bias may accumulate over iterations and result in an inaccurate value function.
Thus, an inaccurate value function will exacerbate sub-optimal actions and lead to poor policies.
To alleviate overestimation bias, a popular approach is to construct robust supervised signals from multiple critics to fit the Q function.
For instance, TD3 takes a minimum of two critics to update the target in TD-error as
[Equation]
which can effectively suppress overestimation.
However, this pessimistic approach often induces underestimation bias (Ciosek et al., 2019; Pan et al., 2020) as we demonstrate in Fig 3.
A key challenge in tackling this issue is that we do not know during training if a particular estimate is overestimated as computing the ground truth Q-value is expensive, therefore, we have to rely on proxy
TD error signals during training.
If the supervised target value is close to ground truth values then overestimated state-action pairs have negative TD errors, while underestimated pairs are associated with positive TD errors.
Estimation bias and loss function Another important component of policy evaluation is the loss landscape.
MSE of TD errors is the default option for most existing works in learning value functions.
Based on Bellman update formula (Bellman, 1966), zero TD-error, i.e., δ ( s , a ) = 0 for every state-action pair, indicates that the value network can predict value function perfectly.
For MSE Q loss, besides its simple formula, the global minima will coincide with a well-fitted value function that has zero TD error.
The magnitude of MSE fitting loss quantifies the discrepancy between the learned Q value function and the value prediction target, such as Eq (4).
However, the constructed supervised target usually depends on bootstrapping as in Eq (4) and the loss is not the distance from the ground truth value function of the policy.
In fact, due to nonconvexity and the limited capacity of neural networks, zero TD error is almost impossible to reach.
In practice, finite or even one update step of stochastic gradient descent is employed to optimize value networks after each policy update in DRL.
However, symmetric losses like MSE or Huber are not the only options for optimizing the value network so that zero TD error is reached at its global minimum.
In this work, we investigate loss functions that are asymmetric.
Compared with MSE loss, in which overestimated states with negative TD error are indistinguishable from underestimated states with positive TD error, asymmetric loss functions can create different loss landscapes for state-action pairs with positive and negative TD errors.
Due to its asymmetry with respect to state-action pairs with positive and negative TD-error δ , we coin our new losses as asymmetric Q (AsymQ) loss functions.
Among many choices for AsymQ loss functions, we start our investigation with a simple function that is obtained by making a small modification to the standard MSE function.
From MSE L MSE ( { δ i } ) =
(cid:80) Bi
=1 1 B δ
i
, we introduce asymmetric loss whose global minimum coincides with zero TD-error
L SMSE ( { δ i } , T ) =
B (cid:88) i =
s.g.
[ Softmax ( {− δ i T }
[Equation]
where the operator s.g.
[ · ] denotes the stop gradient operator during backpropagation.
The design of this asymmetric loss has several merits.
First, it parameterizes a family of loss functions including
MSE as a special case when T → ∞ .
Second, compared with standard MSE loss, it introduces minimum overhead computation and can be interpreted as weighted MSE loss.
Instead of assigning the same weights for all samples, Eq (5) determines weights based on the TD-error distribution of samples in one batch.
Since the weights in Eq (5) are obtained from Softmax-operator on TD-errors, we coin the asymmetric loss as Softmax MSE (SMSE) .
Although Eq (5) depends on the samples in a batch, i.e., a particular sample’s weight with the same
TD-error can vary across batches, we can still compare samples in batches and their underlying base functions.
MSE loss function is based on quadratic function L MSE ( { δ i } )
∝ (cid:80) Bi
=1 δ 2 i , while the loss function for SMSE is (the detailed derivation is included in Appendix A):
[Equation]
This loss landscape is depicted in Fig 2.
Note the choice of SMSE is mainly due to its simplicity and versatility as we can control the loss asymmetry and loss landscape by the sign and magnitude of temperature.
While in this paper we primarily explore the SMSE loss function, we investigate some additional choices of AsymQ loss functions in Section 5.
Leveraging the asymmetric loss landscape of SMSE, we can effectively encode inductive bias for policy evaluation.
For the inherent overestimation bias present in algorithms such as DQN and
DDPG, SMSE with positive temperature will penalize states with negative TD error.
We illustrate this point with a toy example in Fig 2, where we simulate value function updates with varying
SMSE landscapes with all simulations starting with a batch where the TD-errors of all samples are set to a constant negative value by fixing appropriate target values.
We study how the TD errors of sampled-batch evolve over updates and find that the loss landscape changes the value function fitting dynamics significantly.
Samples in the batch shift from negative TD-error to positive TD-error as we lower the temperature.
With the temperature in the right range, estimation bias can be lowered quickly in our synthetic simulation of value function update.
We demonstrate that this indeed scales to mitigating estimation bias during RL training on challenging continuous control benchmarks in the next section.
[Figure]
has a different and asymmetric loss landscape as in Fig 2a.
It reduces to MSE when T = ∞ .
Fig 2b,2c simulate SGD trajectories of TD-error on SMSE loss landscape for 10 steps where sampled state-action pairs have a negative TD-error (note that negative
TD-error corresponds to overestimation).
We further visualize the final histogram of TD-error for different temperatures in Fig 2d.
The shaded regions depict histograms for the final TD-error results of each temperature with solid- lines indicating the fitted Gaussian distributions. The intuition found in the toy example can be generalized to more general RL envs, where asymmetric Q loss function can control bias preference of policy evaluation and alleviate inherited overestimation in DDPG and DQN.  A SYMMETRIC Q L OSS : CONTROL ESTIMATION BIAS
In this section, we show that the small change in value-fitting loss has a large impact on reinforcement learning and that different loss landscapes will lead to different estimation biases.
Our simple asymmetric loss can drastically boost policy learning in existing RL methods without involving other computationally expensive tricks.
We experiment with our algorithm on the suite of MuJoCo continuous control tasks (Todorov et al., 2012), interfaced through OpenAI Gym (Brockman et al., 2016).
4. C ONTROL ESTIMATION BIAS
We visualize the estimation bias of DDPG, TD3, and DDPG+SMSE (ours) for the OpenAI gym environments Walker2d-v3 and Ant-v3 in Fig 3.
The estimation bias is measured by the average difference between Q network prediction and ground truth value for 10,000 randomly sampled actionstate pairs.
More details regarding how the estimation bias is computed can be found in Appendix B.
Clearly, DDPG suffers significant overestimation, which is expected due to inherited maximum bias (see lemma 1).
In contrast, TD3 has a serious underestimation issue, a phenomenon that was also reported in Pan et al.
; Lyu et al. (2022).
As different temperatures parameterize different loss landscapes, we conduct an ablation study on estimation bias with different temperatures.
Lower temperatures put more penalty on action-state pairs whose predictions are larger than their supervised labels and more relaxed for pairs with positive TD-error.
As it validates in Fig 3c and 3d, positive temperatures can relieve overestimation issues in DDPG and extremely small positive temperatures even lead to slight underestimation.
The results suggest we can to some extent control estimation bias through the parameterized loss function, therefore these asymmetric loss functions can be used to introduce inductive bias for learning value functions.
When the temperature is within a proper range, we can significantly reduce estimation bias and learn accurate value functions.
[Figure]
0.25 of the standard deviation of the estimation bias.
4. B OOST POLICY PERFORMANCE
We expect RL algorithms to benefit from more accurate value functions.
We investigate the policy performance of DDPG with our SMSE loss.
Fig 4a and 4b show the effects of different temperature.
As T decreases from positive infinity, the more accurate value function helps boost policy performance.
At the same time, we also find learning at extremely small temperatures may deteriorate policy performance.
The issue may be explained by the following factors.
A lower temperature makes learning more pessimistic as learning may ignore samples with positive TD error and penalizes both true positive and false positive overestimation.
As an extreme case, as T → 0 , the weight from softmax in Eq (5) of SMSE will concentrate on samples with the smallest TD-error, which succumb to outliers and noise in the training.
To illustrate this effect, we introduce the metric effective batch ratio (EBR),
[Equation]
where { w i } quantifies the contribution weights for each sample in one batch.
Note that for MSE (equivalently SMSE with T = ∞ ) w i = 1 B and EBR =
so each sample contributes equally to the value function update.
On the other extreme, when T → 0 , the weights { w i } = Softmax ( {− δ i T } ) for SMSE concentrate on one sample and EBR = 1 B .
In general, EBR can measure the effective percentage of samples that contribute to the value function update.
EBR is inspired by effective sample size in statistics literature (Murphy, 2012).
The statistics also indicate the variance of { w i } in one batch.
We visualize EBR for different temperatures in Fig 4c during training.
As expected, EBR decreases as we decrease temperature.
We additionally adopt clipped weights in our practical implementation (Algorithm 1) to prevent very small EBR and improve training stability. [Figure] fixed temperature
4. A UTOMATIC TEMPERATURE ADJUSTMENT FOR SMSE
The experiments on estimation bias demonstrate the importance of choosing the proper temperature for various environments, and it can be expensive to do an exhaustive grid search for every environment, so auto-tuning of this parameter should greatly reduce the burden of adopting our method.
Though there exist many metrics we can employ to perform automatic temperature adjustments (Haarnoja et al., 2018), we present one auto-tuning algorithm tailored for SMSE DDPG with the
EBR metric.
In our experiments with DDPG, we find the optimal fixed temperatures for different environments can vary from 3 to 50 for the proposed SMSE loss.
An ideal temperature should balance the tradeoff between the estimation bias and variance, where a lower temperature puts more penalty on overestimation than underestimation to counteract inherent bias while higher temperatures make the weights uniform which is preferred to resist noise and reduce variance during learning.
Here we introduce one simple approach to adjust the temperature based on EBR.
Empirically, we find experiments with competitive performance usually have EBR in a fixed interval, which is around [0 .
Therefore, a simple tuning scheme can be proposed based on EBR.
We increase the temperature when EBR is too small and decrease the hyperparameter when EBR is too large over the expected interval.
We summarize the final algorithm in Algorithm 1 (More discussions and details of hyperparameters are included in Section 5 and Appendix B).
We term the auto-tuned loss as SMSE*, and illustrate the learning curve and the auto-tuned temperature for DDPG-SMSE* in
Ant-v3 in Fig 5 and Fig 4d respectively.
Algorithm
Input: β up , β down : interval for desired EBR, β multiplier : temperature multiplier
Sample mini-batch of B transitions ( s i , a i , r i , s ′ i ) from replay-buffer B
TD-error δ i
=
r i + γQ θ ′ ( s ′ i , π ϕ ′
( s ′ i ))
− Q
θ ( s i , a i )
Weights { w i } =
Softmax ( clip ( −{ δ i } T , − ln 2 , ln 2))
Loss L θ =
B (cid:88) i = s.g. ( w i )
δ 2 i // s.g. : stop gradient
Update θ ← θ − α ∇ θ L
Every d iterations
Calculate EBR based on Eq (7)
T ← T × β multiplier if EBR > β up
T ← T/β multiplier if EBR < β down
E XPERIMENTS
We present more experiments to answer the following questions: (1) How is the performance of
DDPG+SMSE compared with the existing algorithms with multiple critics?
(2) Can our auto-tuning algorithm help find the proper temperature to reduce estimation bias?
(3) Can other asymmetric loss functions apart from SMSE improve policy learning?
(4) Can our approach work for environments with discrete action space?
[Figure]
Benchmark We demonstrate the effectiveness of our approach in the six MuJoCo environments.
We compare our approach with popular baselines, DDPG, TD3, and Soft Actor-Critic with automating entropy adjustment (Haarnoja et al., 2018)
(SAC).
For SAC, we include two variants, SAC-D which uses two critics, and the Clipped Double Q-Learning trick in Eq (4), and SAC-S which only has one critic.
For reproducibility, our implementation of selected baselines and our algorithm is based on CleanRL (Huang et al., 2021).
We use default hyperparameters unless stated otherwise for all experiments.
We apply the SMSE trick in DDPG, denoted as DDPG+SMSE.
Our results are presented in Tab 1 with learning curves in Fig 5.
For each environment, we select the temperature by sweeping temperature over values { 3 , 5 , 10 , 20 , 50 } .
We include more details of our implementation and hyperparameters for both our algorithms and compared baselines in Appendix B.
( RL with a single critic )
The experiment results show our algorithm can significantly improve DDPG by simply replacing MSE loss with the asymmetric SMSE loss for fitting Q functions.
Our algorithm achieves performance comparable with the SOTA algorithms with multiple critics.
It indicates our algorithm can save half of the computational resources (see Appendix for update time comparisons) for policy evaluation while retaining policy performance, in some cases even outperforming existing algorithms in challenging environments like Ant-v3.
( Automating temperature adjustment )
We also include learning curves obtained by our proposed automating temperature adjustment, denoted as
DDPG+SMSE*.
Our algorithm can find the proper temperature for most environments.
( Compositionality with other techniques )
We also apply the SMSE trick in SAC-S, denoted as SAC-S+SMSE.
The asymmetric loss can also benefit the off-policy algorithm with a stochastic actor.
( Other asymmetric loss function )
Our asymmetric algorithm is not limited to a specific loss.
We experiment with two new asymmetric losses:
[Equation]
Though these new asymmetric losses have different landscapes, they also penalize negative TDerror more and positive TD-error less.
Tab 1 shows DDPG with these new asymmetric losses can also boost policy learning of DDPG in challenging environments.
Interestingly, we find the fixed temperature T = 1 . 5 for ˆ L 2 works for all environments.
[Table]
5. DQN EXPERIMENTS
We demonstrate that our proposed SMSE loss can also be beneficial for discrete action scenarios by demonstrating it’s effectiveness when combined with the DQN algorithmMnih et al. (2015).
We evaluate our proposed method on the MinAtar benchmark Young & Tian (2019) and some
Atari games Bellemare et al.
(2013) comparing it against both baseline DQN and DoubleDQN updateVan Hasselt et al.
with all the variants operating with a common set of hyperparameters.
In Fig 6 we report the best-performing learning curves for some of the games in benchmarks obtained by a grid search in (0.5, 1, 5, 10, 20, 100) for the temperature hyperparameter in DQN+SMSE keeping the clip parameter at ln 2 .
Results for more games in the benchmarks are available in Appendix B.7.
C ONCLUSIONS AND LIMITATIONS
We introduce a lightweight approach to control estimation bias for policy evaluation in DRL.
Instead of constructing robust supervised targets at the cost of multiple value networks, we find the fitting loss function in policy evaluation has a surprising effect on controlling estimation bias.
We are able to reduce estimation bias and boost policy performance by simply changing MSE loss to an asymmetric loss in DDPG without increasing the computation cost by much.
We point out several limitations of our approach here.
Though we propose SMSE which includes MSE as a special case and an automatic temperature adjustment algorithm for SMSE, how to design a proper or even optimal loss function is out of the scope of our work.
In fact, We actually find SMSE is not so robust in DQN setting, several fail cases and possible explanations are included in
Appendix B.
Our results provide one approach to inject inductive bias in DRL via loss functions for policy evaluation.
It remains an open problem to adjust and customize the loss landscape automatically for different environments.
R EPRODUCIBILITY S TATEMENT
The detailed discussion on assumptions and proofs are included in Appendix A. For experiments, hyperparameters and other implementation details of Algorithm 1 and compared algorithms are included in the main paper and
Appendix B. This should provide sufficient information for the reader of interest to reproduce our results.
A P ROOF A. LEMMA
The proof is heavily based on conclusions from Section 4.1 and Appendix B from Fujimoto et al.
We summarize the finding below.
Given a pair of actor ( π ϕ , Q θ ) , we can define two new actors from the policy gradient algorithm
[Equation]
We note ¯ ϕ is based on estimated value function Q θ while ˆ ϕ is for ground truth value function Q π .
Q π is only for analysis purposes and not available during training.
We assume Z 1 and Z 2 are chosen to normalize the gradient.
Due to gradient ascend, there exists ϵ 1 such that if α ≤ ϵ 1 the estimated value of π ¯ ϕ will be bounded below by the estimated value of π ˆ ϕ :
[Equation]
Similarly, there exists ϵ 2 such that if α ≤ ϵ 2 then the true value of π ¯ ϕ will be bounded above by the true value of π ˆ ϕ :
[Equation]
If in expectation the Q θ is at least as large as the Q π for policy π ˆ ϕ E
[ Q θ ( s , ϕ ˆ ϕ ( s )] ≥
E [ Q π ( s , ϕ ˆ ϕ ( s )] , then Eq (11) and (12) indicate that policy ¯ ϕ will also be overestimated if step size α ≤ min( ϵ 1 , ϵ 2 ) :
[Equation]
The above conclusion is possible when we have the normalized gradient during gradient ascent.
For the unnormalized gradient, a stronger assumption on Q π and Q θ to make Eq (13) hold (Fujimoto et al., 2018).
Although Eq (13) is made on one step policy gradient, the overestimation may accumulate as we train policy for more steps.
A. E Q (6)
To prove Eq (6) equivalent to Eq (5) up to normalization constants, we show their gradients with respect to δ are equivalent.
We denote loss in Eq (5) as L 1 and Eq (6) as L 2 .
Therefore, the gradients can be obtained
[Equation]
where (cid:80) Bi exp( − δ i T ) is a normalization constants shared for every sample δ i in one batch.
On the other hand,
[Equation]
Based on Eq (14) and (15), we can conclude that gradient during one step gradient descent for Land L 2 are equivalent up to normalization constants.
B E XPERIMENT DETAILS AND DISCUSSIONS B. I MPLEMENTATION OF SMSE FOR POLICY EVALUATION
For Softmax MSE loss, we include the key modification based on PyTorch (Paszke et al., 2019) in Fig 7.
We clip the weight logits for numerical stability.
For all experiments, we find clipping range [ − log 2 , log 2] works best for various experiments.
log 2 is chosen to guarantee the ratio between the largest weight and the smallest weight is less than 4.
Without clipping, our Softmax
MSE may assign unreasonable large and small weights for samples in one batch and result in small
EBR which leads to very noisy updates and could destabilize the training.
[Figure]
For DDPG, SAC-D, and TD3, our implementation is based on CleanRL (Huang et al., 2021) 1 , which provides a single-file implementation for each algorithm.
We add an extra evaluating step that tests each RL agent in the same environments with 10 different random seeds.
All learning curves and reward return numbers are based on experiments in testing environments despite we find that episode return curves during training are similar to testing ones.
For DDPG, we use the implementation from Fujimoto et al.
(2018) instead of the original DDPG (Lillicrap et al., 2015), which includes several tricks that improve DDPG performance.
Our SAC-D implementation matches the performance of the official codebase 2 .
For a fair comparison, we use the same learning rate 3 × 10 −and start-timesteps 25000 for all experiments.
SAC-S is based on SAC-D but it only uses one critic instead of taking the same minimum value of two independent critics.
Unless stated otherwise, the hyperparameters and implementation details follow the default choice in CleanRL.
B. H YPERPARAMETERS FOR SMSE
For SMSE algorithm with fixed temperatures for policy evaluation, we sweep the temperature options across environments.
For each environment, we report the performance with the best temperature and present the best performing temperature parameter in Tab 2.
For weights’ clip value, we try ( − ln 2 , ln 2) , ( − ln 3 , ln 3) , ( − ln 5 , ln 5) and ( − ln 10 , ln 10) , and find ( − ln 2 , ln 2) and ( − ln 3 , ln 3) can learn good policies most of the time, therefore we use ( − ln 2 , ln 2) by default across environments.
In Fig 8, we compare policy performance and EBR for different clipping ranges for a fixed temperature T = 3 in Walker2d-V3.
The experiment shows that clipping helps stabilize training and learn better policies.
We also note that clipping plays a less important role when we use higher temperatures in SMSE.
For auto-tuned temperature SMSE*, we use interval (0 . 95 , 0 . 98) for desired EBR range in Algorithm 1.
The range is selected based on EBR curves for different environments.
We have not further swept EBR options and better performance may be possible with fine-tuned hyperparameters.
We believe that the above hyperparameters might not be optimal for SMSE and better hyperparameters can be obtained through a more extensive search.
However, our results still demonstrate a surprising performance boost obtained by simply replacing MSE with SMSE, warranting further investigation of asymmetric loss landscapes for value function learning.
Sensitivity of β multiplier : For Ant-v3 we try three different β multiplier parameters and report the performance curves for DDPG+SMSE* in Fig 9.
We see that the performance is robust to the choice of this hyperparameter when contrasted with the difference in performance observed while tuning the temperature parameter directly.
B. O THER ASYMMETRIC LOSS
We also conduct some initial experiments with more asymmetric loss functions Eq (8).
It is worth noting the main difference between ˆ L 1 and SMSE is that SMSE has a stop gradient operator in calculating SoftMax.
Although they appear to be very similar, ˆ L 1 still presents a different loss landscape and has different gradient descent dynamics.
ˆ L 2 is another even simpler asymmetric when using SMSE with a lower temperature by avoiding very small EBR Without clipping SMSE may assign large weights for overestimated samples and ignore underestimated samples [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
loss function, which can be interpreted as a combination of two quadratic functions.
We initially parameterize it as
[Equation]
We perform a grid-search for T over { 1 . 2 , 1 . 5 , 2 . 0 } , and find that T = 1 .
5 works well for all environments.
We emphasize that both functions put more penalization on state-action pairs whose
TD-Error is negative.
B. E STIMATION BIAS FOR F IG
For each environment, we pick 20 checkpoints uniformly from the train steps to test RL agents’ performance.
For each checkpoint, we evaluate the learned Q values on 10,000 state-action pairs that are uniformly sampled from the replay buffer.
The predicted Q-value is obtained by simply feed-forwarding the state-action pair into the Q-network from the checkpoint.
The ground truth Q π is estimated in Monte-Carlo fashion, where we execute the policy checkpoint and collect rewards along trajectories to estimate the average discounted return J ( π ) , the trajectories are clipped atsteps and the average over 3 such trajectories are treated as the ground truth value, we use onlytrajectories because the variance observed across trajectories were low as the policy is deterministic.
We plot the mean and standard derivation of estimation bias over 10,000 state-action pairs in Fig 3.
The pseudo-code for this procedure is provided in Algorithm 2.
Let π , Q π be the policy and value function to evaluate.
Explore environment with π + N (0 , ϵ ) and store N transitions to replay-buffer B
Sample M state-action pairs { ( s i , a i ) } from B
Let Q PRED = (cid:2)
(cid:3) 1 × M and Q TRUE
= (cid:2) 0 (cid:3) 1 × M for i = 1 to M do
Q PRED [i] =
Q π ( s i , a i )
Q TRUE
[i] = Monte-Carlo-estimate ( discounted return ( s i , a i , π )) end for return mean and standard deviation of (cid:0) Q PRED - Q TRUE (cid:1) B. U PDATE TIME ANALYSIS
We benchmark the average update time for the algorithms DDPG, TD3, SAC-D, SAC-S, DDPG +
SMSE, and SAC-S + SMSE for training on the Ant-v3 environment. All algorithms are run with the same batch-size hyperparameter on an Intel i9-9900 16 core + RTX 2080 Ti desktop.
From Fig 10, we observe that our SMSE loss function adds very minimal cost (0.2 ms/step) to the update time on-top of DDPG but still achieves performance comparable to the state-of-the-art algorithms.
B. E NVIRONMENTS WITH DISCRETE ACTIONS
The DQN implementation for the Atari benchmark is based on the Tianshou(Weng et al., 2021) package that leverages envpool(Weng et al., 2022) implementations of the Atari environments to speed up the run-time.
The architecture and pre-processing of frames are similar to the ones found in the original DQN paper(Mnih et al., 2015).
The hyperparameters used to come from the default parameters present in the Tianshou code-base, we just experimented with different temperature parameters of the additionally introduced SMSE loss, and temperatures from the set (0.5, 1, 5, 10, 20, 100) were tried.
In Tab 3 we outline the best performing run’s temperature and final performances for Atari games and contrast it against the baseline DQN (T = ∞ ) and DoubleDQN.
The results reported are the mean and standard deviation of returns over 5 random seeds. Environment
[Table]
For the MinAtar (Young & Tian, 2019) games benchmark, we use the CleanRL (Huang et al., 2021) implementation of DQN and leverage the hyperparameters suggested in the MinAtar paper along with additionally limiting the environment steps for the MinAtar/Seaquest-v0 environment to 10, steps as done in (Lan et al., 2020).
Similar to the Atari games benchmark we try different temperature parameters from the set (0.5, 1, 5, 10, 20, 100) and report the best performing run in Tab 4, where the result represents the mean and standard deviation over 5 random seeds.
[Table]
While the performance improvement is observed on some of the games in both the Atari and MinAtar benchmarks, it still fails to show major improvements for a good fraction of the environments [Table] unlike the continuous control MuJoCo benchmark.
This could be due to several reasons, one major factor is that estimation bias in DQN is different from that of DDPG [Table] and the loss developed and
[Table] tested extensively in DDPG doesn’t simply transfer and work for all DQN environments. It is to be noted that even DoubleDQN updates sometimes result in lower performance compared to DQN so it can be expected that just reducing estimation bias does not always necessarily translate to higher performance.
Additionally, we leverage the hyper-parameters suggested in popular open-source packages and do not perform any tuning apart from a small grid search on the SMSE-temperature parameter.
An extensive hyper-parameter search along with the exploration of new asymmetric loss functions for environments with discrete action spaces is an exciting direction for future work.
B. SMSE WITH ENSEMBLE Q- NETWORKS
Arguably one of the most effective approaches to reducing the bias and having stable value-function targets is the usage of an ensemble of critics (Kuznetsov et al., 2020;
Chen et al., 2021; Liang et al., 2022; Lee et al., 2021;
Lyu et al., 2022; Wu et al., 2020; Wei et al., 2022).
In this section, we compare SMSE with state-of-the-art algorithms based on an ensemble of critics.
We find our approach can achieve competitive results in a much more computational efficiency approach.
Besides, we surprisingly find our asymmetric loss is orthogonal to the ensemble method to some extends.
SMSE vs REDQ (Chen et al., 2021)
We first compare our work with an ensemble Q algorithm called Randomized Ensembled Double Q-Learning (REDQ)
(Chen et al., 2021).
REDQ leverages an ensemble of Q-networks with additional tricks such as high update-to-data ratio and targetminimization from a subset of these Q-networks to demonstrate strong performance in a sampleefficient manner.
In Tab 5 we compare the best performance (mean across 3 seeds) of REDQ and our proposed SAC-S+SMSE on some Mujoco environments and report the number of environment steps and wall-clock time duration to achieve the same.
For this comparison, we use the same underlying hyperparameters such as architecture, learning rate, etc., with REDQ specifically usingnetworks, a random subset size of 2 for target construction, and an update-to-data (UTD) ratio of 20.
The performance of SAC-S+SMSE is reported for the best temperature parameter selected by the procedure outlined in Appendix B.2 as done in the main paper.
REDQ indeed achieves a very high performance, which can be attributed to stable targets obtained from ensemble networks and the high UTD ratio.
While a sample-efficient performance is achieved, the run-time and memory consumed by such ensemble techniques are high.
If we use N networks, compare M networks to provide a target estimate, and perform G updates per step (UTD ratio) - we can approximate the FLOPs used in REDQ for the network size used in our experiments as ≈ G × ((2 × FwdPass × M ) + N × ( FwdPass + (2 × BckwdPass ))) .
The FLOPs count of SAC-S+SMSE can estimated with G = 1 , M = 1 , and N = 1 .
We obtain the FLOPS for FwdPass from fvcore library and approximate the cost of BckwdPass as 3 × FwdPass.
SMSE improves Ensemble To experiment if we can incorporate an ensemble of networks into the SMSE framework, we improve on the ensemble averaging target (AVG) baseline also used in (Chen et al., 2021), by plugging in our proposed SMSE loss to update the Q-networks from targets predisposed to overestimation bias.
In Fig 11 we compare the impact of using SMSE loss instead of the baseline MSE loss.
We see that the proposed loss function significantly improves the AVG baseline.
To further see if our method helps reduce estimation bias we compare REDQ, AVG, and AVG+SMSE on Hopper-v3 and present the results in Fig 12.
Average bias is the mean of the differences between the predicted Q-values and the estimated true Q-values (estimated by a
Monte Carlo simulation) over state-action pair samples.
The normalized bias divides the bias by the estimated true Q-value to visualize the scale of the bias in terms of the value function.
We observe that the proposed SMSE loss can effectively reduce the bias in the action-value function by keeping the average bias close to 0 and providing competitive performance improvements.
These findings indicate that SMSE is better than the default MSE loss, and the improvement is orthogonal to other methods proposed to address the estimation bias problem in RL.
We are optimistic that further investigation into Asymmetric loss functions can benefit many algorithms and downstream applications that rely on value functions for further optimization and planning.