Paper: Iterated Deep $Q$-Network: Efficient Learning of Bellman Iterations for Deep Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
The technical analysis lacks strong theoretical justifications besides rephrasing a theorem from another paper and some not fully clear interpretations of this.

Author response:
We argue that the theoretical analysis is valid and supports our claim that iDQN has a theoretical benefit over DQN. In the revised submission, we further clarify Section $5$ in order to ease the understanding. We stress the point that using an already-published theorem does not make the theoretical analysis weaker. In Section $5$, we now propose two different points of view. In the first one, we consider iDQN without target networks and demonstrate that each approximation error is minimized with $K$ times more gradient steps than with DQN. In the second point of view, we justify why the following approximation errors are controllable when using iDQN as opposed to DQN, which uses a naive approach.

Reviewer comment (Evaluation):
Concerning the computation requirement, one gradient descent step seems to become at least slightly more computational expensive as compared to DQN. In the paper it is mentioned at the very end of the discussion section that "with K=5 (it) only requires 1 to 2 times more time to run". Can this quantification be more accurate or why would it "sometimes" same compute time and "sometimes" double the compute time?

Author response:
We face the problem that an algorithm takes a different amount of time to run depending on:
- the type of GPU used to run the algorithm. Our cluster has different types of GPUs, which makes the comparison hard to make (see table underneath).
- the number of seeds that are run in parallel.
- the Atari game. Some games take longer to run simply because the emulator takes longer to generate the next step.
- the performance. Better algorithms are more likely to generate longer trajectories. This also has an impact on the running time.

Here are some examples to illustrate our point:

| Game         | Algorithm | Time  | # seeds | GPU  | 
| :----------- | :-------: | :---: | :-----: | :--: |
| Alien        | IQN       |  63h  |   2     |  B   |
| Alien        | iDQN K=5  |  60h  |   2     |  B   |
| Asterix      | DQN       |  42h  |   2     |  B   |
| Asterix      | iDQN K=4  |  46h  |   2     |  B   |
| Breakout     | DQN       |  31h  |   2     |  B   |
| Breakout     | iDQN K=5  |  54h  |   2     |  B   |
| Enduro       | iDQN K=5  |  25h  |   1     |  A   |
| Enduro       | iDQN K=5  |  50h  |   1     |  B   |
| Enduro       | iDQN K=5  |  58h  |   2     |  B   |
| VideoPinball | iDQN K=5  |  25h  |   1     |  A   |
| VideoPinball | iDQN K=5  |  49h  |   1     |  B   |
| VideoPinball | iDQN K=5  |  67h  |   2     |  B   |

A: RTX 4090
B: RTX 2080Ti, RTX 3080Ti, RTX 3090, or A5000 GPU

For those reasons, we changed the sentence in the paper to say that the running time of iDQN is comparable to the one of IQN on the same hardware for the same Atari game.

Reviewer comment (Evaluation):
The ablation study and discussion section provide an interesting discussion on the hyperparameters and answer many of the questions one could have. However, I still have questions about the interpretations done for these hyperparameters. Why do you claim that you provide a "thorough understanding of their effects" given that I don't see very clear backup for the claims such as "Problems in which the environment is highly stochastic will require more gradient steps to learn a Bellman iteration hence the need to decrease the rolling step frequency" and "highly stochastic problems will benefit from having a small target update frequency since the positions of the online networks are more likely to be noisy". I would suggest explaining in more details how these claims are made and if there is no fully clear data for these interpretations, I would suggest being a little more cautious.

Author response:
We agree with the Reviewer. We further clarified those claims with $2$ new ablation studies on the $2$ hyperparameters in Figure $9$ middle and right of the revised submission.

Reviewer comment (Reproducibility):
Is an open-source implementation of the code made available? I do not see any GitHub link.

Author response:
We invite the Reviewer to have a look at our code in the appendix. We will turn the GitHub public upon acceptance.

We apologize for our mistake. The code is not in the Appendix, but in the uploaded supplementary material.

Reviewer comment (Theory):
Although intuitively sound, the main idea lacks sufficient theoretical support. The only theorem in the paper is quoted from [1] for approximate value iteration, which provides a bound on the error $\||Q^* - Q^{\pi_K}\||$ in terms of a weighted sum of previous error components $\||\Gamma^* Q_{k-1} - Q_k\||$. The authors claim that since one gradient step of iDQN affects multiple of these components at a time, then "iDQN can lower the approximation error bound more than DQN.” However, I do not think this argument is true. This bound is based on the accuracy of previous Q-functions relative to their respective Bellman updates after learning. Just because a gradient step for iDQN affects more than one Q-function at a time does not mean the error bound will automatically be lower. For example, if one of iDQN’s heads suddenly changes, the errors for the downstream heads could suddenly increase because the Bellman operator’s projected location would also change, requiring the other heads to adjust accordingly.

Author response:
We respectfully disagree about this. We believe that there is a misunderstanding and hope that Section $5$ of the new submission clarifies it. We think that the example brought by the Reviewer is misleading for the general understanding of the theoretical benefit of iDQN. We insist on the fact that it is crucial to compare the behavior of iDQN with the behavior of DQN in the example given by the Reviewer. In the revised submission, we now tackle this specific case in Figure $6$ and the text explaining this figure. The approximation errors corresponding to the downstream networks can indeed vary along the training, especially when the target networks of iDQN get updated. Nonetheless, we show that those approximation errors are bounded by a controllable term as opposed to DQN, which only takes the last online network as a starting point to learn the approximation error of the subsequent step. In simpler words, while DQN has a naive approach to minimize the approximation errors corresponding to the subsequent steps, iDQN proposes a way to pre-train the next online $Q$-function.

Reviewer comment (Experiments):
There is weak empirical evidence that learning many Bellman iterations in parallel is feasible and significantly improves value estimation. Because each head must bootstrap from the previous one, I would think that maybe only two heads at most could be reliably trained simultaneously; any heads afterwards would begin bootstrapping from extremely biased estimates and quality would degrade significantly. This appears to be the case in Figure 5, where even using 10 heads does not greatly reduce the value error compared to 1 head in a low-dimensional problem.

Author response:
We believe that this comment is linked with the first point and hope that the answer helps clarify the concern. We now have updated the submission and provided significant additional evidence that learning many Bellman iterations brings a boost in performance. Indeed, Figure $7b$ of the revised submission shows that iDQN + $3$-step return is performing on the level of advanced distributional methods while not learning the distribution of the return. Figure $8$ illustrates that the combination of iDQN with IQN is outperforming iDQN and IQN. 

Concerning the experiment on Car-On-Hill, we provide the following table showing the distance of the current estimate $V^{\pi_k}$ to the optimal $Q$-function $|| V^* - V^{\pi_k} ||$:

| K  | k=15         | k=20         | 
| :- | :----------: | :----------: |
| 1  | 3.91 +- 0.87 | 2.77 +- 0.87 |
| 2  | 4.02 +- 0.64 | 2.82 +- 0.75 |
| 4  | 4.15 +- 0.70 | 2.32 +- 0.45 |
| 5  | 3.64 +- 0.73 | 2.24 +- 0.36 |
| 10 | 2.50 +- 0.42 | 2.03 +- 0.09 |
| 20 | 2.25 +- 0.24 | 2.02 +- 0.11 |

When $K$ increases, we observe a clear decrease of the considered distance.

Reviewer comment (Experiments):
The separate network heads for iDQN are split immediately after the convolutional layers, and not immediately before the final linear layer as I would have expected. Because the vast majority of weights in the DQN conv net are contained in the dense layers, this adds an enormous number of extra parameters to iDQN, which might be improving its performance. It also makes the proposed method very expensive.

Author response:
The cost of splitting the heads after the last convolutional layer can be quantified by the spatial complexity and the temporal complexity. Regarding the spatial complexity, we provide a thorough analysis in Section $C$ of the appendix. This analysis shows that the increase in spatial complexity is negligible compared to the memory required for storing the replay buffer. For the temporal complexity, in theory, it is possible to fully parallelize the additional computations required by iDQN such that it is reduced to the temporal complexity of DQN. In practice, we observe an overhead comparable to IQN. We believe that the approach proposed by iDQN is still worth it, considering the theoretical and experimental advantages that it brings. 

We want to point out a misconception brought by the review. The boost in performance given by iDQN over DQN is not a direct consequence of the fact that iDQN is using more parameters. Indeed, in iDQN, the actions are sampled from a single network having the same amount of parameters than a DQN agent. It is the way the networks are trained that brings a boost in performance.

Reviewer comment (Experiments):
The target-network update frequency is faster for iDQN than the baseline DQN, which I think makes the empirical comparison unfair. This could be contributing to the apparent performance increase in addition to the extra parameters.

Author response:
We provide a new ablation study in Figure $8$ right of the revised submission in which we set the rolling step frequency of iDQN to be equal to the target update frequency of DQN ($8000$). In this experiment, we observe that iDQN greatly outperforms DQN. This is due to the fact that iDQN performs $4$ times more gradient steps per approximation error than DQN while having the same overall number of gradient steps.

Reviewer comment (Novelty):
The paper would benefit from a stronger discussion of $n$-step returns, as the proposed method is more related to $n$-step methods than is currently appreciated. An $n$-step return can be seen as a stochastic approximation to $n$ iterations of the (on-policy) Bellman operator. Thus, $n$-step returns are an alternative way to achieve a similar effect as the proposed algorithm. The paper currently cites TD($\lambda$) [2] for $n$-step returns, but it should cite [3] instead—see the bibliographical/historical remarks at the end of chapter 7 of [4] for related references.

Author response:
We kindly ask the Reviewer to have a look at the new submission in which new experiments with $n$-step return have been added. We agree with the Reviewer that an $n$-step return can be seen as a stochastic approximation to $n$ iterations of the (on-policy) Bellman operator; however, these $n$ iterations are directly included in the reward instead of being learned in a chained manner like iDQN proposes. 

Thank you for noticing this mistake. We now cite the suggested work for $n$-step return.

Reviewer comment (Writing):
I had trouble understanding the intuition the author's are trying to convey when they refer to "learning the Bellman iteration". This is a central concept in various discussions but I'm still unclear about what they mean. Fortunately, the authors also provide more formal descriptions of their method so I am fairly confident I understand what their method is doing, at least mechanistically.

Author response:
We believe that understanding what we call "learning the Bellman iteration" is a major point for the general comprehension of our work. Therefore, we hope that the following clarification will be useful to clear all doubts. The optimal $Q$-function is the fixed point of the Bellman operator. DQN uses the fact that the Bellman operator is a contracting operator. Thus, iterating multiple times over this Bellman operator can lead any random $Q$-function to the desired fixed point. Theoretically speaking, DQN applies the Bellman operator when computing the target, as explained in Section $3$. Due to the use of function approximation, DQN needs to first learn the target before applying the Bellman operator a second time. This step of learning the target is what we call learning the Bellman iteration. 

Nevertheless, we take the Reviewer's opinion into account and, in the revision, we now use the term: “minimizing the approximation errors” in Section $5$.

Reviewer comment (Theory):
The theoretical analysis seems very informal and I don't believe we can say much from it. The comparison with the loss bound that the authors make implies that we are talking about the same $Q_k$'s but their definitions differ greatly between the action-values of each step of approximate value iteration and those defined by the proposed method.

Author response:
We believe that there is a misunderstanding here since the $Q$-functions used in iDQN are the same as the ones used in the theorem. We stress that DQN and iDQN are performing approximate value iteration. As written in Section $5$, the loss of DQN $(r + \gamma \max_{a'} Q_{k-1}(s', a') - Q_k(s, a))$ is an unbiased estimator of the approximation error $||\Gamma Q_{k-1} - Q_k||_{2, \nu}^2$ and the same goes for iDQN. We further clarify that part in Section $5$. We would appreciate some feedback from the Reviewer to know if this has improved the understandability of the section. We hope that this explanation togethet with the clarification brought in the answer of point $1.$ help resolve this misunderstanding.

Reviewer comment (Experiments):
It's not clear whether the comparison to DQN is fair when normalizing for "gradient steps". If DQN is twice as fast, could I not do twice as many gradient steps? The question this brings is if this isn't more an observation of update frequency, e.g., doing several updates per environment step, which we know can improve performance with regards to sample efficiency. I'd be interested in hearing the author's thoughts on this.

Author response:
We point out that the comparison to DQN is not only made for the same amount of gradient steps but also for the same amount of samples. The latter is the usual way of comparing algorithms for the Atari benchmark. In order the respond to the last part of the comment, we added an experiment in Figure $8$ left of the revised submission. In this experiment, we set the rolling step frequency to be equal to the target update frequency of DQN ($8000$). In that case, both algorithms have access to the same amount of Bellman iterations. In the same figure, we added an experiment in which DQN performs $4$ times more gradient steps than DQN (Adam), as asked by the Reviewer. We see that the performances drop while iDQN greatly outperforms both versions of DQN.

Reviewer comment (Experiments):
The ablation study is quite limited. I understand that these experiments are computationally expensive but some understanding of the behavior of this method and its hyperparameter could still be found with smaller scale experiments in less costly domains. Several of my questions (below) could have been answered with a more complete ablation study.

Author response:
We understand the request. We invite the Reviewer to check the revised submission in which we added several ablation studies corresponding to each point raised by the review. A description of the new ablation studies is provided in the general response.