Paper: Why not both? Combining Bellman losses in deep reinforcement learning (ICLR-2024)

Reviewer comment (Theory):
The authors ought to present the theoretical analysis more formally and organize them into theorems like many FQE papers do.

Author response:
We have now phrased the result as a proposition in the revised paper. Thank you for your feedback.

Reviewer comment (Reproducibility):
The authors ought to list a detailed hyperparameter setup table in the appendix for clarity.

Author response:
We have put a hyper-parameter table in the appendix. Thank you for your feedback.

Reviewer comment (Reproducibility):
The hyperparameter $\lambda$ seems to be important and the most critical part of the proposed method. While I do not see enough discussions on this hyperparameter. The authors use different $\lambda$ for different tasks or when combined with different algorithms. It is important to give practical guidance on how to determine this parameter.

Author response:
We have added a sensitivity analysis on $\lambda$ for the SAC + Mujoco experiment and we describe how we ended up choosing $\lambda = 0.1$ for that particular experiment which hopefully provides some helpful insight.  Because FQE diverges relatively quickly in that setting with high update rates of the target, it ended up relatively easy to set a sensible parameter for that experiment. We discuss this in more detail in this new section of the  appendix.

Reviewer comment (Experiments):
Are there any way of tuning this parameter (i.e., $\lambda$) automatically? How does this hyperparameter affect the performance? How sensitive is the method to this parameter? Have you try some other datasets other than MuJoCo? Can your conclusion still hold?

Author response:
We have launched some additional experiments on LunardLander-v2 and MinAtar tasks on DQN. Please refer to the new appendix for more details. These preliminary results indicate that the BE term can also be efficient in stabilizing DQN in high target network update regimes.

Reviewer comment (Experiments):
The authors only combine their method with SAC, then can your method benefit more advanced algorithms like TQC [1], REDQ [2]?

Author response:
There has been a large amount of prior work aimed at improving the sample efficiency of deep RL algorithms. While it is difficult to make definite statements about how this particular auxiliary loss would combine with existing ones, we would like to point out that the mechanisms by which sample efficiency in our case is improved is very unique. Indeed, to the best of our knowledge this regime of high update rate of the target network remains rarely studied as a way of speeding up deep RL. We have shown that it can be impactful on Mujoco tasks and with the new baseline added in the revision, that other modeling-based auxiliary losses are not as effective as the proposed BE loss in preventing divergence of FQE in this regime. Because the properties of our approach are rather unique, we believe there is a good chance that it is complementary with existing methods. Of course successfully integrating these different methods is not trivial but can be an interesting future direction.

Reviewer comment (Experiments):
To my understanding, a hyperparameter (lambda in algorithm 2) is used to balance how much auxiliary loss to use, can you provide more ablation on how the algorithm's behavior and how the accuracy of its Q estimates change as lambda changes?

Author response:
Thank you for your feedback. We have run additional experiments on the SAC + Mujoco setting with 4 additional values of $\lambda$ on the previous 4 Mujoco tasks and averaged over 20 seeds. Results and discussion can be found in the appendix. Please let us know if this addresses your concerns.

Reviewer comment (Experiments):
In some recent works it has been shown that techniques that provide more accurate Q estimates can be especially helpful when the algorithm is taking more updates per data point collected. Will the proposed method also benefit from this setting?

Author response:
The way this paper aims at improving sample efficiency is by recomputing the target more often. This leads to more frequent applications of the Bellman operator and lets the rewards propagate faster through the chain of state-action pairs. This shows well in the policy evaluation experiment where we can compute the distance to the true Q function and has tangible benefit when solving RL tasks as shown in the SAC + Mujoco experiments. To the best of our knowledge, no other work has investigated improving sample efficiency by making FQE more robust to faster target value recomputations. We think because its properties are unique, it is more likely that our work can integrate well with prior improvements of SAC/deep RL such as increasing the Q-function ensemble size or performing more gradient steps.

Reviewer comment (Evaluation):
How much computation overhead does the proposed method have? Would like to see a table comparing wall clock time between it and SAC baseline.

Author response:
In the SAC + Mujoco experiment, the execution time of DouBel was about ~1.2 times  the execution time of vanilla SAC, while DouBel with bias correction was ~1.5 times the execution time of vanilla SAC, because of the extra V-value network. We have added this to the appendix, thank you for pointing it out.

Reviewer comment (Evaluation):
Will the proposed method also lead to better long-term performance?

Author response:
We have not observed better long-term performance in the Mujoco task. We think the main benefits of our methods are in the initial speed-up provided by a more frequent application of the Bellman operator. That is why our results focus on the 0-500K range for the SAC + Mujoco task.

Reviewer comment (Experiments):
Given the same computation budget, will the proposed method be more efficient compared to methods with other auxiliary losses or with ensemble-based bias reduction?

Author response:
Methods such as AQE can be about 200 times slower than SAC, because it has a 10 times larger ensemble size and uses 20 times more gradient steps per sample. Under a fixed computational budget this might not even be more efficient than vanilla SAC. Of course this all depends on the environment and if the cost of interaction is very high (e.g. robotics) it might still be beneficial to focus on sample efficiency first and foremost. As we have shown in the new baseline requested by Reviewer JxN6, auxiliary losses such as those predicting next state features do not stabilize FQE in the high update rate of the target regime, and hence the advantages of our methods are unique. Because of that we believe it can complement quite well prior work towards the goal of developing increasingly more sample efficient deep RL algorithms. Of course merging all these approaches is not necessarily trivial but can be an interesting future direction.

Reviewer comment (Experiments):
Will it be beneficial to combine the proposed method with other auxiliary losses or other bias reduction techniques, or that does not make sense?

Author response:
Methods such as AQE can be about 200 times slower than SAC, because it has a 10 times larger ensemble size and uses 20 times more gradient steps per sample. Under a fixed computational budget this might not even be more efficient than vanilla SAC. Of course this all depends on the environment and if the cost of interaction is very high (e.g. robotics) it might still be beneficial to focus on sample efficiency first and foremost. As we have shown in the new baseline requested by Reviewer JxN6, auxiliary losses such as those predicting next state features do not stabilize FQE in the high update rate of the target regime, and hence the advantages of our methods are unique. Because of that we believe it can complement quite well prior work towards the goal of developing increasingly more sample efficient deep RL algorithms. Of course merging all these approaches is not necessarily trivial but can be an interesting future direction.

Reviewer comment (Writing):
The presentation related to the theoretical results is, in general, clear, apart from equation 13: what are $m_r$ and $M_{\Phi}$?

Author response:
> 1) What are $m_r$ and $M_\Phi$

$m_r$ and $M_\Phi$ define linear models in $Phi$ that minimize the squared error to the reward $\cal R$ and the next state features $\Phi’$.  We have expanded the explanation in page 5. We hope it is clearer now.

Reviewer comment (Writing):
In the paper, it is claimed that the proposed method allows for more aggressive target network update rates. However, from the text, I could not understand what exactly that means: is it the case for DouBel(20) or DouBel(1). This generated some confusion throughout the text. I suggest to add an explanation in the text on how target networks are used and why they are important in this setting.

Author response:
> 2) Clarifying what are (aggressive) target network update rates

We have updated Sec. 2.2 to clarify what target update rates mean. Please see the bolded text. 

As for what aggressive update rates are, it depends greatly on the task. For example, tuned hyper-parameters in stable-baseline 3 suggest an update rate of 10 for CartPole but 1000 for Atari. 

In our SAC + Mujoco experiment, we consider the update rate of 10 as aggressive since vanilla SAC, without the proposed regularizer, ‘diverges’ in many cases. 

For the policy evaluation task, at the fastest update rate of 1, FQE(1) does not diverge but adding our regularizer has a big impact on the distance to the true Q function, whereas for an update rate of 20, DouBel(20) and FQE(20) are about the same in performance and convergence profile.

Reviewer comment (Experiments):
The paper lacks a simple experiment (e.g., with finite state and action spaces) where the theoretical results can be shown to hold true without approximating the solution of LSTD. It would be beneficial to demonstrate how the auxiliary loss aids such settings before introducing approximations.

Author response:
> 3) Adding a simple experiment

We experimented with small scale tabular environments with rich observations (i.e. the MDP is finite and small but the learner observes high dimensional vector valued states).  However, in these simple MDPs, Neural FQE was converging very fast to the true Q function even with a target update rate of 1. These simple MDPs were thus not able to illustrate the challenges of policy evaluation with neural function approximators that one can experience on more complex MDPs.

Reviewer comment (Experiments):
From Table 1, it seems that both DouBel(20) and FQE(20) have much lower loss than DouBel(1) and FQE(1). I would expect to see for some of the datasets presented a plot showing the final distance to the true Q function as a function of the number of gradient steps before the target network is updated (e.g., from 1 to 40). Is the loss in DouBel always lower than the loss in FQE, or is there a trade-off?

Author response:
> 4) DouBel(20) and FQE(20) have much lower loss than DouBel(1) and FQE(1)

Yes, experiments in 6.1 show that if the computational budget allows it, the slower update rate of 20 is better. However, the learning curve illustrates well how slow policy iteration can be in this regime. Because the targets are updated less frequently, the Bellman operator is applied less often and it takes more time to reach a fixed point. To give an idea of the scale of the X axis for the policy evaluation experiments: we perform up to 10K gradient steps, but an algorithm such as PPO on Mujoco performs typically only 320 gradient steps per iteration for its policy evaluation! Even after 10K gradient steps, there are still four Mujoco datasets where DouBel(1) outperforms DouBel(20). Given how computationally expensive deep RL can be, we believe it is still important to consider improving neural FQE through this faster update of the targets. One can always revert to slower update rates once the error reduces enough. The claim of the paper still holds that these faster update rate regimes do not seem helpful with vanilla FQE but become interesting when equipped with our proposed regularizer.

Reviewer comment (Theory):
It is not clear to me how the theoretical results imply that the algorithm can have more aggressive target network update rates. I understand that the use of the auxiliary loss can decrease the error in the Q function. However, could the author clarify why this matters when having more target network updates?

Author response:
> 6) Relation between theoretical results and more aggressive update rates

The theoretical results relate the usual auxiliary losses used in prior work based on modeling the reward and next state features with the proposed loss $L(\Phi)$. It is not about the learning dynamics of neural FQE. A hand-wavy argument could be that Bellman error minimization is stable because it is a true gradient descent method (unlike FQE), and adding an explicit BE term seems to be effective at avoiding gradient updates of the feature function that would blow-up the Bellman error. However, the only validation of this property in the paper is empirical.

Reviewer comment (Presentation):
What is the difference between Figures 6, 7, and 8?

Author response:
> 7) Difference between Figures 6, 7, and 8 

As can be seen in Table 1, the policy evaluation experiments are done on three datasets for each of the four Mujoco tasks corresponding to different performance levels of the data generating policy. Figures 6, 7, 8 show the learning curves on four Mujoco tasks for each of the three performance levels.

Reviewer comment (Evaluation):
Why is there an initial divergence in Figure 4 for SAC, and how does it relate to the theoretical results?

Author response:
> 8) Initial divergence in Figure 4 for SAC and relation to theoretical results 

Note that in Figure 4, it is not SAC with default hyper-parameters but with a more aggressive target network update rate every 10 gradient steps. We doubt there are theoretical results that would explain why it happens only in the beginning since there are counter examples where it happens later in the learning cycle (e.g. Figure 4, right). If the question was why it happens in general, to the best of our knowledge there are only convergence results for neural FQE with very specific (shallow, 1 hidden layer) neural architectures [1] that do not apply here, and as far as we know there are no theoretical results on why FQE would necessarily converge to begin with.

[1] Gaur et al.; On the Global Convergence of Fitted Q-Iteration with Two-layer Neural
Network Parametrization; ICML23