Paper: Mitigating Estimation Errors By Twin TD-Regularized Actor and Critic for Deep Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
The employed baselines use implementations that have not been optimized for the DeepMind Control suite (nor for the sample-efficiency setting considered as the authors only train for 1000 steps). This makes the reported results notably inferior than what considered standard for what these algorithms can achieve (e.g. see the learning curves in [1]).

Author response:
>The employed baselines use implementations that have not been optimized for the DeepMind Control suite

The question is fair, BUT we have taken this issue into consideration in our implementations when reporting results. Please read our implementation details of the paper (Appendix C) and Common points 3 more carefully. 
Specifically, 

1. We use default environment setting (1000 steps horizon), which is well accepted as shown in previous papers [1,2,3]. 

2. The length of training at 1e6 maximum steps is also well accepted in the community [2,3]. 

3. we use the same common hyperparameters for all algorithms. This choice of hyperparameters is well accepted as shown in previous papers [1,2]. 

As for the reference provided by the reviewer, it uses CUSTOM hyperparameters, while our selected set of hyperparameters are i) consistently used throughout our evaluations, and ii) also used in seminal works such as original TD3 and SAC.

Reviewer comment (Theory):
The main novelty of the paper is the TDR procedure which foregoes dual TD-learning and chooses the target Q-function based on the lower TD error. However, I believe the paper does not convey a solid intuition as to why this heuristic is superior than standard minimisation of the target Qs. Hence, I am worried the proposed methodology might simply trade-off slightly improved sample-efficiency for stability and convergence (in harder environments/longer training regimes) due to 1) employing n-step returns 2) lowering the pessimism to counteract the actor's Q-maximization. In connection to Point 1 above I would really like to see experimental evidence showing how the proposed algorithms fare in harder environments and for longer training horizons, to empirically validate their efficacy.

Author response:
>Novolty and LNSS issue

Please refer to our Common Point 2 for novelty of TDR.

LNSS is not among our claims for novelty in this study. Just like n-step to D4PG, LNSS is a useful add-on.  If the reviewer is interested in LNSS, please refer to their paper [4]. 

As for the strategy of "lowering the pessimism to counteract the actor's Q-maximization," this approach is central to the functionality of TD3. In TD3, the method involves directly minimizing the Q value, which serves to mitigate the pessimism that arises as a byproduct of the actor's maximization of the Q value. However, as we've identified, this direct minimization approach can lead to other issues, as highlighted by the reviewer. Please refer to our Common Point 2, Theorem 1 (specifically case 1 and case 2) where by using TDR, we effectively addressed this issue by selecting a critic value that is associated with a lesser TD error.

Reviewer comment (Experiments):
The authors claim several times that the dual TD-learning (target Q-function minimization from TD3) "promotes a new problem of underestimation, which usually occurs during the early stage of learning, or when subjected to corrupted reward feedback or inaccurate states" (page 4). Yet this claim is not supported by any empirical or theoretical evidence in the text.

Author response:
>"promotes a new problem of underestimation, which usually occurs during the early stage of learning, or when subjected to corrupted reward feedback or inaccurate states" (page 4). Yet this claim is not supported by any empirical or theoretical evidence in the text.

Please refer to our Common Point 3. Consider at very early stage of training with $Q = 0$ as a common initialization. If the reward noise is negative, by inspecting the bellman equation, the estimation error will most likely result in an underestimation error. Similarly for inaccurate state and action due to  $s_{noise}$ and $a_{noise}$, it is possible to have $\mathbb{E} [Q(s_{noise},a_{noise})] < \mathbb{E} [Q(s_{true},a_{true}]$ which will result in underestimation. This is why we need to study estimation error under noisy reward, state and action rather than a noise free environment. This also shows the significance and novelty of our work.

Reviewer comment (Writing):
The authors incorporate what they refer to as 'Long N-step surrogate state' (LNSS), which appears to be a simple modification to n-step returns that takes the discounted average rather than the discounted sum of the rewards. While they remark several times this is a core component of their methodology, they never explain this simple procedure in the main text and relegate its description to a Section in the Appendix I found unnecessarily convoluted. Am I missing something about this method?

Author response:
>Novolty and LNSS issue

Please refer to our Common Point 2 for novelty of TDR.

LNSS is not among our claims for novelty in this study. Just like n-step to D4PG, LNSS is a useful add-on.  If the reviewer is interested in LNSS, please refer to their paper [4]. 

As for the strategy of "lowering the pessimism to counteract the actor's Q-maximization," this approach is central to the functionality of TD3. In TD3, the method involves directly minimizing the Q value, which serves to mitigate the pessimism that arises as a byproduct of the actor's maximization of the Q value. However, as we've identified, this direct minimization approach can lead to other issues, as highlighted by the reviewer. Please refer to our Common Point 2, Theorem 1 (specifically case 1 and case 2) where by using TDR, we effectively addressed this issue by selecting a critic value that is associated with a lesser TD error.

Reviewer comment (Theory):
**[Problematic claim in the introduction]** The authors claimed that "The clipped double Q-trick is designed to solve the overestimation caused by the max operator." Actually, the max operator is not the culprit for the overestimation; we would encounter overestimation even using the bellman evaluation operator. The culprit is function approximation errors.

Author response:
We would like to make the following two points.

1) The notion of overestimation error has long been established and frequently used in  previous seminal works such as those we cited. For example, in Lan 2020 [2], published at ICLR, the first sentence of the second paragraph under section 3 says "The overestimation bias occurs since the target $max_{a' \in A} Q(s_{t+1},a')$ is used in the Q-learning update." TD3 [4] states  in the introduction that "Overestimation bias is a property of Q-learning in which the maximization of a noisy value estimate induces a consistent overestimation". In Thrun \& schwartz 1993 [3], under section 2, it says that "The max operator (in Q learning), however, always picks the largest value, making it particularly sensitive to overestimations."  

2) About the reviewer's comment that "we would encounter overestimation even using the bellman evaluation operator. The culprit is function approximation errors", please refer to our Common Point 3 where we discussed causes/sources of estimation error, which is NOT ONLY just the function approximation error. It can also come from noisy reward, state, and action. This is the very reason of our experiment design, and is considered one of our major contributions to studying estimation error at the root cause.

Reviewer comment (Experiments):
**[Unreliable Baseline Performance]** The author's report of SAC and TD3 performances is not consistent with what is reported in other papers.  
For example:  
* AcrobotSwingup task: The author reported SAC performance is ~4@1M steps. However, referring to the TD-MPC[1] paper (Figure 3), SAC achieves ~100@500k steps. This performance is much higher than the authors' own dTDR algorithm.  
* Quadruped Walk task: The authors report SAC performance of ~196@1M steps. However, referring to the BAC[2] paper (Figure 32), SAC achieves ~600@400k steps.

Author response:
Incorrect assessment. Please refer to our Common Point 3.

Reviewer comment (Experiments):
**[Need More Experimental Results]** The evaluation of just 6 medium and easy DMControl tasks is not enough to support the effectiveness of the proposed practical mechanism. More experimental results on MuJoCo and DMControl tasks should be added.

Author response:
The reviewer may have missed reading SOTA developments and results on DRL. Please refer to our Common Points 3 and 4. Additionally, please note that Mujoco [1] is indeed a physical simulation engine.  But the DeepMind Control Suite (DMC) has established its position in DRL as providing benchmark environments, which are built upon the Mujoco simulator.

Reviewer comment (Experiments):
**[Arbitrary Claim on baseline D4PG]** The claim that D4PG is SOTA is overly arbitrary in the abstract. There are different SOTA methods for different tasks, and this claim is disputed. For example, in BAC[2], the method outperforms D4PG on some tasks. Additionally, RND[3] and REDQ achieve SOTA on some tasks.

Author response:
First, on using SOTA baseline, please carefully review our Common Point 4.

Additionally, our selection of off-policy baselines, as substantiated by extensive results based on the Deepmind Control Suite [5,6,7], represents the state-of-the-art (SOTA). Specifically, D4PG is widely recognized for its superior performance in continuous control problems, a fact underscored by its impressive scores, frequently surpassing 900 on a scale of 1000. This level of achievement has led to D4PG's extensive use in numerous significant and recent publications as a baseline method for comparison [8,9,10,11].

In regards to a comparison with BAC, please read our Common Point 3. Next, we noted that BAC is currently under review for ICLR 2024 and is yet to be officially published. Our criteria for choosing baselines prioritizes algorithms that are not only published but have also gained widespread acceptance in the research community. This approach ensures that our comparisons are made against benchmarks that are established and validated in the field, and the results can be repeated. 
One more important detail that we'd like to point out - BAC only considers adding a single noise to the action. This does not allow a sufficient study of the estimation error issue. 


In response to the recommendation to include RND in our comparison, it's important to note that RND primarily focuses on environments such as Montezuma’s Revenge, which is not a continuous control problem. As for REDQ, its evaluation predominantly relies on the OpenAI Gym benchmarks. According to  benchmark studies such as Pardo 2020 [6], it has reached a consensus that  methods such as SAC and PPO, which may perform well in Gym environments, show less impressive results in DMC benchmarks. As such, when consider using the more established DMC benchmarks.  As previously mentioned, D4PG stands out as a well-recognized SOTA method within the community.

Reviewer comment (Novelty):
**[Add related Works]** BAC is a paper dedicated to underestimation and overestimation, reporting good performance and conducting many experiments on a wide range of benchmark tasks. It is suggested to add BAC to the related work discussion, and possibly conduct experimental comparisons if the BAC authors provide code or experimental data.

Author response:
In regards to a comparison with BAC, please read our Common Point 3. Next, we noted that BAC is currently under review for ICLR 2024 and is yet to be officially published. Our criteria for choosing baselines prioritizes algorithms that are not only published but have also gained widespread acceptance in the research community. This approach ensures that our comparisons are made against benchmarks that are established and validated in the field, and the results can be repeated. 
One more important detail that we'd like to point out - BAC only considers adding a single noise to the action. This does not allow a sufficient study of the estimation error issue.

Reviewer comment (Presentation):
**[Move Table 1 to Appendix]** Figure 2 and Table 1 present results from the same set of experiments. There is no need to occupy half a page in the text with Table 1; it could be moved to the Appendix.

Author response:
We thank the reviewer for the suggestion. But this table is highly informative, important, and comprehensive as it compliments the learning curves and other plots by providing quantitative performances upon learning convergence.  
This approach is standard and has been a common practice in the community. It can be seen in almost all seminal papers such as  TD3 and SAC.

Reviewer comment (Experiments):
**[Better not to use the limited results to answer Q1-Q6 questions repeatedly]** The Experiment Section begins by proposing to answer six distinct questions. However, for several of these questions, there is only 1–2 supporting evidence points repeatedly drawn from the main experiments. Readers are required to make indirect associations to find this evidence. For clarity and focus, it is advisable to reduce the number of main questions to 2–3, move the less important ones to the Appendix, and conduct more experiments to support them, rather than repeatedly using limited results to answer many questions.

Author response:
We thank the reviewer for the feedback but we respectfully disagree. 
In quantitative analysis of experimental results employing both horizontal and vertical comparison analyses is necessary and essential. This is a standard technique in scientific data analysis on different performance measures. They answer different questions. Additionally, our presentation of using both horizontal and vertical comparisons actually helps guide readers to read the results and verify our claims.

Reviewer comment (Experiments):
The comparative experimental results are not sufficiently reliable. None of the baselines are designed for addressing the estimation error problem.

Author response:
We thank the reviewer for reading our paper and for providing comments.
However, we respectfully disagree with the reviewer on this point. 
Actually, our experiments were designed for the purpose of addressing estimation error. 
Please refer to our Common Points 2 and 3.

Furthermore, please note that the issue of estimation error is of great importance regardless if the baseline algorithms are designed for this purpose or not. Refer to Common Point 3, estimation error poses a great challenge for almost all DRL algorithms. As demonstrated in our results (Figures 4, 5 and Table 6 in the updated paper pdf during rebuttal), if we deliberately introduce those factors that may cause  estimation error (such as adding noises to $s, a, r$, all baseline algorithms show a significant performance drop from that without noise using the same benchmark environments. However, by using TDR, the lost performance were all  compensated for to a large degree, and the baseline methods with TDR still produce stable results. This clearly shows our contribution and the novelty of TDR.

Reviewer comment (Experiments):
The paper emphasizes that the Twin TD-Regularized method can effectively control errors, but the effectiveness of bias control has not been demonstrated in the experiment section.

Author response:
We adhere to the same performance measures used in prior works such as TD3 and SAC, both of which aim to address estimation error. Please refer to our Figures 2,4,5, under both noisy and noise free case, TDR boosts average reward of baseline methods.

We also remind the reviewer to read our Common Points 3 where we clearly explained why by adding multiple sources of noise, our baseline results no long replicate those published results (because they did not use noise).

Reviewer comment (Theory):
The first Q-network is traditionally chosen to compute the actor TD regularization term. Will it have a negative impact on the selection of Q-network target values in the proposed method?

Author response:
Please refer to Common Point 2 where as we discussed that, following the TD3 convention, we use the first critic to compute actor.  This is because theoretically $Q_{\theta_1}$ and $Q_{\theta_2}$  converge to the same result, making it redundant if we go between the two.

For impact on the selection of Q-network value please refer to common point 2.

Reviewer comment (Experiments):
It is very sensitive to parameters for DRL algorithms, and using different parameters can have different effects. Will the parameters of the comparison algorithms in the experiment be consistent with the original paper (the best situation), and will their performance be worse than the original paper?

Author response:
Please refer to our Common Point 3. If we evaluate under the Noise-Free conditions of the original papers, that defeats the purpose of this paper which aims at adequately addressing estimation errors that may come from multiple sources ($s, a, r$, and $\theta$). In real-world applications, these factors are prevalent and ubiquitous. 

Nonetheless, in the rebuttal, we have provided new data to further strengthen the novelty and significance of TDR. Please refer to Common Point 1 for our evaluations of baselines without Noise.

Reviewer comment (Experiments):
The paper only demonstrates that TDR can improve the performance of baselines. You should compare some SOTA bias control algorithms in recent years.

Author response:
It's important to note that TD3 and SAC, the two baselines we've compared TDR against, are not only well-accepted in the field but also have demonstrated efficacy in directly addressing overestimation error, a key aspect of bias control. Additionally these two methods have been used as SOTA baselines in most recent top conference papers [4-7]. 

Our evaluations focus on showing the effectiveness of TDR within the context of general off-policy methods. By applying TDR to these widely recognized and established baseline methods (TD3, SAC, D4PG), we aim to demonstrate its capability to enhance performance in more realistic (aka noisy) environments.

Reviewer comment (Experiments):
The ablation studies are not quite thorough. You should provide more experimental results about approximate estimation error for clearly suggesting the benefits of different components of the algorithm.

Author response:
We respectfully disagree with the reviewer and here is why.

In our evaluations, we follow the same established protocol in widely accepted and sound research works. We recommend the reviewer  read TD3 and D4PG more carefully to find out. TD3's core innovation lies in its Clipped Double $Q$ Network, accompanied by additional features such as Delayed Policy Update and Target Policy Smoothing Regularization. In their ablation study, the authors of TD3 assessed the impact of each individual component by removing it from the TD3 framework.

Likewise, D4PG's primary advancement is the integration of distributional learning into DDPG, enhanced by additional elements like Prioritized Experience Replay (PER) and n-step updates. The effectiveness of each component in D4PG was evaluated by incrementally adding them to DDPG.

In a similar vein, our approach with TDR involves components like TD regularized critic (TD critic), TD regularized actor (TD actor), and an additional enhancement, namely LNSS. We analyze the contribution of each of these elements by incorporating them into baseline algorithms, thereby assessing their individual impact on performance. This approach allows us to understand the distinct value added by each component in the TDR framework. 

In terms of ..."You should provide more experimental results about approximate estimation error...", in addition to our evaluations provided in the paper, please also review our newly added data (Figures 4,5 and Table 6) that summarize TDR evaluations without noise. That actually is an evalution (and performance comparison) for approximation errors.

Reviewer comment (Novelty):
The main weaknesses of the paper are the novelty of the approach and the significance of experimental results. It seems an A+B+C work, which modifies the previous TD-regularized AC method (Parisi et al., 2019) marginally, and then combines distributional learning and N-step learning into the proposed approach.

Author response:
We thank the reviewer for reading our paper and for providing commons. 
However, we respectfully disagree with the reviewer. The TDR method is NOT an A+B+C work. Please refer to our Common Point 2, especially the one on Novelty of TDR. 

As a side note, LNSS is just a useful add-on, playing a role similar to PER in D4PG [3]. We never claimed any credit for LNSS as a (key) contribution of this work.

Reviewer comment (Experiments):
The experimental part is also a bit weird, where the replicated baselines seem worse than original papers.

Author response:
We thank the reviewer for noticing this. Please refer to Common Point 3.

Reviewer comment (Presentation):
In subsection 3.1 "DOUBLE Q IN ACTOR-CRITIC METHOD", which is under the method section (Sec 3), the contents are actually a summary and description of the literature instead of the proposed method, which can confuse the readers.

Author response:
Yes, and No. Section 3.1 is not only a  background but also an introduction of necessary notations and problem formulation. This is very common, and necessary, in RL papers. Please refer to TD3, D4PG, SAC [1,2,3].

Reviewer comment (Theory):
Regarding Equation (7), TDR always uses a target value associated with a smaller target TD value (regardless of the error sign) between the two. The reviewer questions how this sample-based TD error can serve as a sound measure for reducing the estimation error, since estimation error is measured by comparing the estimated value with an expectation of the true value. The reviewer suggests this may lead to large variance. In addition, there is a typo where “overesdiation” should be “overestimation”.

Author response:
This sample based approximation is a common approach in DRL. DPG [6] uses sampled  TD error to approximate true gradient. DQN, DDPG, SAC, D4PG [2,3,4,5] use collected samples to approximate $Q$ values. TD3 [1] uses the Q value samples from two target $Q$ networks to address overestimation bias. In a similar vein, our TDR method uses TD error samples from two target networks to approximate both overestimation and underestimation errors.

Regarding the variance, "the variance can grow rapidly with each update if the (estimation) error from each update is not tamed." as state by TD3 authors (refer to TD3 [1] section 5). Based on Theorem 1 and discussions of Common Point 2, we show that by using TDR, RL agents will result in better target values with less estimation error compared to previous clipped double $Q$ network in TD3. Therefore, following the same analysis as TD3, TDR will have the same or even more variance reduction.

Reviewer comment (Writing):
The statement "Our TD-regularized actor network directly penalizes the actor’s learning objective whenever there is a critic estimation error" is considered weird. The reviewer suggests it would be better stated as "whenever the critic estimation error is non-zero" and notes that it is unavoidable to have estimation errors in practice when learning these objectives.

Author response:
We thank the reviewer for rephrasing the sentence for us. Actually there is no difference between how we and the reviewer said it. Nonetheless, we can update the phrase if the paper gets accepted.