Paper: CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity (ICLR-2024)

Reviewer comment (Experiments):
The concept should be sufficiently general to combine with other baseline algorithms than SAC, so evaluation across multiple algorithms would further broaden applicability [i.e. are the results specific to SAC variations?].

Author response:
This is an interesting observation. We based CrossQ on SAC because our baselines REDQ and DroQ are also based on SAC. However, our concepts are directly applicable to other off-policy TD algorithms. For demonstration, **we added a TD3 [1] baseline**, as well as a version of CrossQ based on TD3 to Figure 5. This combination shows the same effects in terms of increased sample efficiency, confirming that our proposed design choices regarding batch normalization and target networks can be applied more broadly and are not specific to SAC and the max-entropy RL setting

[1] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, 2018.

Reviewer comment (Experiments):
Generally, a more diverse set of environments would further strengthen the evaluation – this could include other domains (DeepMind Control Suite / MetaWorld / …) or visual control tasks [i.e. are the results specific to OpenAI Gym domains?].

Author response:
To align our statements with the experiments, we have narrowed the claims within the paper by specifying more exactly that our results are restricted to continuous-control tasks with state observations. In our original submission, we included the same benchmark suite as the REDQ and DroQ papers, and even reported experiments on 6 environments rather than 4 as in the previous papers. In our updated submission, we double our number by including **6 additional environments on the DeepMind Control Suite**, revealing the same trend in the improved sample efficiency provided by CrossQ. However, note that we had a limited time during the rebuttal to tune CrossQ as well as the baselines on those environments, and better results may be achievable till the camera-ready submission deadline.

We note that a thorough evaluation of CrossQ on vision-based tasks is in general beyond the scope of the present paper and will constitute a separate publication. In the current paper, we focus on establishing CrossQ as a state-of-the-art algorithm for systems with state-based observations, closely following the setting of REDQ and DroQ. **We foresee an extension of CrossQ to vision-based tasks as the next step for future work.**

Reviewer comment (Novelty):
The paper claims it “show[s] the first successful application of BatchNorm in off-policy Deep RL.” However, it appears that DDPG [1] also used batch normalization 7 years ago in the exact same setting: off-policy deep RL for continuous control in MuJoCo. This means the titular contribution of the paper is not original.

Author response:
It is true that the DDPG paper reported experiments with batch normalization. However, we would like to emphasize that ours is arguably the first “successful” application of BatchNorm in Deep RL. There are two crucial points:
1. In the DDPG paper, **BatchNorm does not significantly improve** the performance, and in the **absence of the target networks even actively harms the performance** (see Fig. 2 in https://arxiv.org/abs/1509.02971). Our paper is the first to use BatchNorm to effectively accelerate Deep RL training by an order of magnitude, similar to supervised learning.
2. The DDPG paper’s **use of BN is limited**: it relies on a unique architectural choice where **actions are fused late** into the critic, and **applies BN only to early layers** which appear before this fusion.
    > “In the low-dimensional case, we [i.e., the DDPG authors] **used batch normalization on the state input** and [...] all layers of the Q network **prior to the action input**.” (Lillicrap et. al 2016).

This is a smart way to **avoid dealing with the on- and off-policy action distribution mismatch in BN** — one simply does not use BN with action-informed features — but is unfortunately only applicable to this specific late-fusion architecture, which is not commonly used. 

Moreover, we found several online discussions that claim that BatchNorm actively harms training [1][2], and we could not find any open source implementation of DDPG using BatchNorm. Therefore, we argue that our CrossQ algorithm is effectively the first method that provides a working open source implementation of batch normalization in Deep RL that results in improved learning performance.

**We added a clarification about the difference of our approach compared to DDPG in Sec. 3.2 “Design Choice 2: Using Batch Normalization”.**

[1] https://github.com/keras-team/keras-io/issues/198

[2] https://www.alexirpan.com/2017/04/26/perils-batch-norm.html

Reviewer comment (Experiments):
There is a lot of hyperparameter tuning that might be unfairly benefiting CrossQ. For instance, the network width for CrossQ is increased from 256 to 2048 units, an 8x increase. Adam’s momentum parameter was also reduced from 0.9 to 0.5. As far as I can tell, these hyperparameters were not tested for the baselines, so CrossQ has been over-optimized for MuJoCo. A fairer comparison would be testing the methods with the same network sizes and then selecting the best Adam hyperparameters for each of them.

Author response:
We would like to note that **CrossQ performs well even with a smaller critic network and Adam’s momentum 0.9**, as shown in Fig. 9 by the “CrossQ (small)” line. Nevertheless, **we added the requested ablations**, running DroQ and REDQ with increased critic networks and smaller Adam’s momentum in Fig. 14 and 15. As can be seen, **DroQ and REDQ perform the same or worse with these hyperparameters**. Therefore, this ablation confirms that these parameters alone are not sufficient to improve upon the baselines, and the other design choices of CrossQ such as BatchNorm and removal of the target network are also important.

Reviewer comment (Evaluation):
I feel that the claim of “state-of-the-art” performance is a bit strong given that only 6 MuJoCo environments were tested, and CrossQ is a clear improvement only in 2 of them. The method has not been evaluated in other continuous-control problems nor against other potential baselines, so I would not necessarily agree that it is state-of-the-art. Nevertheless, the empirical results are still great.

Author response:
We have **toned down the claims** within the paper by specifying more exactly that our claims are **restricted to continuous-control tasks with fully observed state**. In our original submission, we followed the prior work, REDQ and DroQ, and used the same benchmark suite, but we even extended the experiments to 6 environments rather than 4 as in the previous papers. In our updated submission, **we included experiments on DeepMind Control Suite**, which show the same trend in the improved sample efficiency provided by CrossQ. However, note that we had a limited time during the rebuttal to tune CrossQ on those environments, and better results may be achievable till the camera-ready submission deadline.
We added:
1. evaluations on **6 additional Deep Mind Control environments**, 
2. a **TD3 baseline** (Fig. 5),
3. additional **ablations** (Fig. 9, 14, 15),
4. a version of **CrossQ based on TD3** (Fig. 5). 

We hope this demonstrates the more general applicability of our proposed method.

Reviewer comment (Writing):
Inconsistent use of hyphens should be fixed. For example, “Q-function” and “Q function” both appear in the paper.

Author response:
We thank the reviewer for pointing these out and have corrected them in the updated version of the paper.

Reviewer comment (Writing):
Possible typo in Figure 2’s caption: “a **single double** Q function network”?

Author response:
We thank the reviewer for pointing these out and have corrected them in the updated version of the paper.

Reviewer comment (Theory):
As Equation 1 is currently written, it is not correct to refer to the Polyak averaging parameter $\tau$ as “momentum” since $\tau=1$ corresponds to a moving average with no inertia.

Author response:
We thank the reviewer for pointing these out and have corrected them in the updated version of the paper.

Reviewer comment (Presentation):
References are not consistently formatted, e.g., some conference names are capitalized but others are not.

Author response:
We thank the reviewer for pointing these out and have corrected them in the updated version of the paper.

Reviewer comment (Theory):
While the empirical results are compelling, the paper lacks theoretical analysis or justification for the success of CrossQ. Providing insights into why the proposed modifications lead to improved performance would enhance the paper's depth and contribute to a more comprehensive understanding of the algorithm.

Author response:
We added **further insights** into why the proposed modifications lead to improved performance in several places in the paper:
1. we added Fig. 4 that show the difference in the action distributions from the replay buffer and from the policy, highlighting the **importance of using batch normalization** in the proposed manner;
2. in Fig. 9, we added an ablation experiment running SAC with both BatchNorm and Target Networks (as suggested by reviewer 7EH3), showing that our design choice of **removing the target network is important** for the superior performance of the proposed CrossQ algorithm;
3. we added Fig. 14 and 15, providing ablations on DroQ and REDQ with elements of CrossQ such as wider critic networks and $\beta\_1 = 0.5$, showing that these changes alone are insufficient to boost their performance, and **it is really the combination of the components in CrossQ** that is responsible for the strong learning performance.

Thus, we tried our best to **provide intuitions and establish general facts** about the proposed design choices. Nevertheless, we admit that a deeper theoretical investigation would be very illuminating, despite being a difficult task. We hope that our paper may stimulate interest in studying the effects of different design choices in Deep RL algorithms.

Reviewer comment (Experiments):
The experiments are only conducted in one continuous control domain (MuJoCo). It is dangerous to draw conclusion of sample efficiency only from one experimental domain.

Author response:
We followed the prior work, REDQ and DroQ, and used the same benchmark suite. In fact, even in our original submission we extended the experiments from 4 to 6 environments, adding the hard HumanoidStandup task. Following the reviewer's suggestions, in our updated submission **we have doubled the number of environments** by adding **6 DeepMind Control Suite tasks**. We observe the same trend of improved sample efficiency provided by CrossQ (see Fig. 10). We hope that this addresses the reviewer's concern about the applicability of CrossQ to other environments.

Reviewer comment (Experiments):
This paper does not include image-based settings. There are tons of efficient RL works recently on image-based settings and some of the techniques (state augmentation [1, 2] and auxiliary losses [3, 4]) can be also applied to state-based settings. It is worth discussing the recent progress of efficient RL in the related work section or adding baselines for comparison.

Author response:
We agree that a thorough evaluation of CrossQ on vision-based tasks would be very interesting; however, we consider this extension as being **beyond the scope of the present paper**. Efficient image-based RL requires multiple ingredients, as noted by the reviewer, and a careful study of CrossQ in tandem with these would constitute a separate publication.

In the current paper, we focus on establishing CrossQ as a state-of-the-art algorithm for continuous control from state observations, closely following the setting of REDQ and DroQ. We do agree that much like our use of BatchNorm, the techniques mentioned by the reviewer are also strong tools for boosting RL efficiency; **we have added the suggested references** and highlighted the extension of CrossQ to **vision-based tasks as the next step for future work**.

Reviewer comment (Evaluation):
Can you elaborate on the specific scenarios or domains where CrossQ might outperform other algorithms significantly? Are there any limitations or challenges in which CrossQ may not be the most suitable choice?

Author response:
This is a very good question!

In general, we believe **CrossQ should outperform algorithms which use low UTD ratios** and use no special techniques to accelerate convergence. E.g. CrossQ outperforms SAC and TD3 significantly, but is comparable in efficiency to REDQ which emphasizes making the most out of the available data.

In addition to the potential limitations we’ve already noted in the conclusion section, we believe CrossQ may inherit any challenges that BatchNorm introduces. For example, models with BatchNorm are known [1] to struggle in **training with very small minibatch sizes**. BatchNorm uses minibatch statistics to normalize data, and while minibatch noise is regarded as a regularizer, it has also been implicated in sometimes harming optimization stability [2]. BatchNorm-equipped training is also sensitive to non-i.i.d batches, and could possibly struggle with non-stationary data sampling schemes such as **Prioritized Experience Replay** [2]. Such limitations in BatchNorm have been addressed with very simple fixes like Batch Renormalization [4], which might also benefit CrossQ in the future.

[1] https://arxiv.org/abs/1708.02002

[2] https://proceedings.mlr.press/v89/lian19a.html

[3] https://github.com/yhenon/pytorch-retinanet/issues/24 

[4] https://arxiv.org/abs/1702.03275

Reviewer comment (Theory):
The paper focuses on empirical results; however, are there any insights or intuitions on why the chosen modifications (removal of target networks, Batch Normalization, wider networks) lead to the observed improvements? Providing additional context on the underlying mechanisms would enhance the paper's impact.

Author response:
We added **further insights** into why the proposed modifications lead to improved performance in several places in the paper:
1. we added Fig. 4 that show the difference in the action distributions from the replay buffer and from the policy, highlighting the **importance of using batch normalization** in the proposed manner;
2. in Fig. 9, we added an ablation experiment running SAC with both BatchNorm and Target Networks (as suggested by reviewer 7EH3), showing that our design choice of **removing the target network is important** for the superior performance of the proposed CrossQ algorithm;
3. we added Fig. 14 and 15, providing ablations on DroQ and REDQ with elements of CrossQ such as wider critic networks and $\beta\_1 = 0.5$, showing that these changes alone are insufficient to boost their performance, and **it is really the combination of the components in CrossQ** that is responsible for the strong learning performance.

Thus, we tried our best to **provide intuitions and establish general facts** about the proposed design choices. Nevertheless, we admit that a deeper theoretical investigation would be very illuminating, despite being a difficult task. We hope that our paper may stimulate interest in studying the effects of different design choices in Deep RL algorithms.

Reviewer comment (Experiments):
Using wider critic networks may have played a pivotal role in the performance boost, especially in humanoid-based environments. Can you draw SAC+wider networks to Figure 7?

Author response:
We note that even without the wider critic network, CrossQ delivers a strong performance, as shown in Figure 9 in the updated paper by the magenta curve corresponding to “CrossQ (small)”. We have added the requested SAC+wider network “SAC wide Net” ablation study to Figure 9 (yellow curve). There, we observe that **SAC does not get a significant improvement** simply from adding wider networks, indicating that it is **the combination of CrossQ’s design choices that is important**, and not only the wider critic networks alone.