Paper: Tactics of Robust Deep Reinforcement Learning with Randomized Smoothing (ICLR-2024)

Reviewer comment (Novelty):
This work targets only on ONE random smoothing paper: CROP, which limits this paper’s signification. There is another concurrent work on random smoothing [1] with CROP. No discussion and comparison about [1] makes the contribution of this work unclear.

Author response:
Thank you for pointing out your main concern!

>**Q1:** This work targets only on ONE random smoothing paper: CROP, which limits this paper’s signification. There is another concurrent work on random smoothing [1] with CROP. No discussion and comparison about [1] makes the contribution of this work unclear.

**A1:** In [1], they focused on the issue that the bounds in the Supervised Learning setting cannot directly transfer to the RL setting because of the non-static nature of RL. They provided another proof for the bounds and pointed out that the proof in CROP might have some issues. However, [1] also uses the same naive smoothing during inference, which will lead to the same problem as CROP did. We only compare to CROP since the environments we use are closer to CROP. There will be no difference between CROP and [1] in terms of the practical implementation. Hence, [1] still falls into the issue of low clean reward when the smoothing variance increases. We consider this issue important since the agents with strong guarantees but poorly performed are unacceptable.

[1] Kumar, et al. “Policy Smoothing for Provably Robust Reinforcement Learning”, ICLR 2022

**#Additional experiments performed in rebuttal**

We would like to highlight the additional experiments we performed during the rebuttal. We compare our methods with recent methods including ATLA [2] and WocaR [3]. The DQN results are shown in Table A.1 and A.2. The PPO results are shown in Table B.1 and B.2. Our methods still outperform these methods in most of the settings, which strengthens our contributions.

**Table A.1.** The reward of DS-DQN and WocaR-DQN under PGD attack in the Pong environment
(Pong) eps|0.01|0.02|0.03|0.04|0.05
---|---|---|---|---|---
Our DSDQN (Vanilla)|19.2+-0.75|**18.4+-2.15**|**19.2+-1.17**|**17.2+-2.56**|**18.8+-1.17**
WocaRDQN|**21.0+-0.0**|-21.0+-0.0|-21.0+-0.0|-21.0+-0.0|-21.0+-0.0

**Table A.2.** The reward of DS-DQN and WocaR-DQN under PGD attack in the Freeway environment
(Freeway) eps|0.01|0.02|0.03|0.04|0.05
---|---|---|---|---|---
Our DSDQN (Vanilla)|**32.6+-1.20**|**31.6+-1.50**|**30.0+-1.10**|**28.0+-1.41**|**23.8+-1.17**
WocaRDQN|21.4 +- 1.36|21.8 +- 1.17|21.0 +- 0.89|21.0 +- 0.63|21.0 +- 1.10

**Table B.1.** The reward of AS-PPO, Radial-PPO, ATLA-PPO, and WocaR-PPO under attacks in the Walker environment
(Walker) eps=0.05|clean|random|critic|MAD|Min RS|Optimal attack
---|---|---|---|---|---|---
AS-PPO (Ours)|4969|5039|**5488**|**5290**|**4323**|**4296**
Radial-PPO|**5251**|**5184**|No result|4494|3572|3320
ATLA-PPO|3920|3779|3915|3963|3219|3463
WocaR-PPO|4156|4244|No result|4177|4093|3770

**Table B.2.** The reward of AS-PPO, Radial-PPO, ATLA-PPO, and WocaR-PPO under attacks in the Hopper environment
(Hopper) eps=0.075|clean|random|critic|MAD|Min RS|Optimal attack
---|---|---|---|---|---|---
AS-PPO (Ours)|3667|3546|**3706**|2916|1558|1500
Radial-PPO|**3740**|**3729**|No result|3214|2141|1722
ATLA-PPO|3487|3474|3524|3081|1567|1224
WocaR-PPO|3616|3633|No result|**3541**|**3277**|**2390**

[2] Zhang, et al. “Robust Reinforcement Learning on State Observations with Learned Optimal Adversary”, ICLR 2021

[3] Liang, et al. “Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning”, NeurIPS 2022

**Summary**

In summary, we have:
* In Q1, explain that the issue in CROP actually happens generally and the paper [1] also has the same issue.
* Provide additional experiments and show that our methods still outperform recent methods in most of the settings.

We believe that we have addressed all your concerns. Please let us know if you still have any reservations and we would be happy to address them!

Reviewer comment (Novelty):
This main concern the reviewer have is that this work is completely based on CROP and is a correction of CROP. Whether this paper should be published as a full research paper in ICLR main track is doubtful.

Author response:
Thanks for the reply and sharing your concerns! Sorry for making you feel unclear about our contribution. We feel that the current issue of the CROP framework is significant since a smoothed agent with a high guarantee would still be meaningless given that the clean reward is already largely degraded. Our agents are also the first SOTA with a strong provable guarantee and we believe that our contribution is not minor, which can open a new direction of further strengthening the robustness guarantee for future works. 

Here we give a detailed list of our 4 major contributions to this work to make the understanding easier:

1. **Contributions related to the simple Randomized Smoothing (RS):**
    * We identify the failure mode of the existing smoothed DRL agents, showing that the naively smoothed agents have poor trade-offs on the clean reward and robustness.
   * We point out that the robustness of the CROP agents might be overestimated due to the smoothing strategy and attack they used. We fix this issue by introducing hard RS and a stronger attack.
   * We extend CROP’s proposed robust guarantee for DQN agents to the PPO setting and defined action bound. To our best knowledge, the action bound for PPO has never been derived before. We also do experiments on this bound (see Appendix A.13).
2. **Contributions related to robust DRL agents:**
    * Our agents achieved state-of-the-art results under attack. Following the reviewers’ suggestions, We include new baselines and show that our agent still outperforms the current best WocaR-RL in most of the environments.
    * Our agent is the first state-of-the-art agent with a high robustness guarantee at the same time (certified radius and reward lower bound), while the previous state-of-the-art only evaluated their agents under empirical attack.
3. **Contributions related to Randomized Smoothing (RS) in DRL:**
    * We develop the first robust DRL training algorithms leveraging randomized smoothing for both discrete actions (DS-DQN) and continuous actions (AS-PPO).
    * We show that simply training with RS does not work, and is necessary to use denoised smoothing and adversarial training.
    * We point out that different smoothing strategies can affect the robustness guarantee. (i.e. Our hard RS strategy is not affected by the output range of the Q-network and generally achieves a larger certified radius.)
4. **Contributions related to adversarial attack:**
    * We develop a new attack aiming at attacking smoothed agents. This attack model we proposed can be easily used on any attack based on optimization. We show how our setting can be applied to the PA-AD attack, denoted as S-PA-AD, and lead to a stronger attack for smoothed agents.

Thanks for your valuable feedback! Please let us know if you have further concerns. We are happy to address them!

Reviewer comment (Novelty):
The proposed method and its presentation rely heavily on the existing CROP method, thereby limiting its novelty.

Author response:
**A1:** There is another paper [1] that also studies randomized smoothing in RL. This paper and CROP both do randomized smoothing during the inference, which leads to the problem we discussed. Although we use the CROP paper to identify the problem, this is actually a general issue that happens when introducing randomized smoothing naively.  We consider this issue important since the agents with strong guarantees but poorly performed are unacceptable.

[1] Kumar, et al. “Policy Smoothing for Provably Robust Reinforcement Learning”, ICLR 2022

Reviewer comment (Presentation):
The method's description is dispersed throughout the paper, complicating comprehension, and the approaches for discrete and continuous cases appear to differ.

Author response:
**A2:** Sorry for making it difficult for you to read. Our proposed method of robust DRL agents for both discrete and continuous actions (DS-DQN in sec 4.1 and AS-PPO in sec 4.2) is structured as follows:
1. **Motivating example:** We first point out where the previous works did not work well. Hence, we try to leverage RS in the training and design S-DQN and S-PPO. However, we found that there are many problems with this simple design. That's why we design DS-DQN and AS-PPO.
2. **Training:** This part provides the details of our methods. We also include flow charts in Appendix A.2 and A.3 to give a whole picture of our methods. 
3. **Testing:** The most important thing here is to show our smoothing strategies. We use hard RS for DS-DQN and median smoothing for AS-PPO since there are some issues with the smoothing strategy in CROP.
4. **Attack:** Since all our attacks are based on our new attack, we especially introduce how we evaluate the robust reward of our agents.

We believe structuring the main methods in this order can best describe the process of how we discover a problem, how we solve the problem, and how we evaluate our algorithms. As for the connection between the discrete case and the continuous case, they both show how to effectively leverage randomized smoothing to train a smoothed agent that is provably robust. We feel that it is necessary to show that no matter in the discrete or continuous setting, our framework achieves high clean reward, high robust reward, and strong guarantees simultaneously.

Reviewer comment (Evaluation):
The implications of robust certification remain unclear within the context of the evaluated task.

Author response:
**A3:** The robust certificate means the certified radius and reward lower bound for DQN and the reward lower bound for PPO. We provide the details of how we estimate and compute those bounds in practice in Appendix A9.  As for the DS-DQN, simply removing the noise that is already known cannot improve the robustness. The denoiser should be viewed as a part of the agent, which does not necessarily denoise the state to the original one since we also use the temporal difference loss. The denoiser makes the agents more tolerant against the Gaussian noise and hence the problem of decreasing clean reward can be mitigated. Since DS-DQN is more tolerant against the noise, a much larger smoothing variance is applied to enhance the robustness.

Reviewer comment (Evaluation):
What are the implications of a robust certificate in this context? How is it computed for experimental evaluation, such as for Atari Pong?

Author response:
**A3:** The robust certificate means the certified radius and reward lower bound for DQN and the reward lower bound for PPO. We provide the details of how we estimate and compute those bounds in practice in Appendix A9.  As for the DS-DQN, simply removing the noise that is already known cannot improve the robustness. The denoiser should be viewed as a part of the agent, which does not necessarily denoise the state to the original one since we also use the temporal difference loss. The denoiser makes the agents more tolerant against the Gaussian noise and hence the problem of decreasing clean reward can be mitigated. Since DS-DQN is more tolerant against the noise, a much larger smoothing variance is applied to enhance the robustness.

Reviewer comment (Theory):
In DS-DQN, a random number generated from a Gaussian distribution is added to the state during training. A denoiser network is then employed to reconstruct the original state. Does this imply that, in an ideal scenario, the denoiser network essentially mirrors the Gaussian distribution introduced to the state initially? If so, what is the purpose of training such a denoiser when we already know the noise model added to the input? How does the denoiser contribute to achieving better rewards and enhanced robustness?

Author response:
**A3:** The robust certificate means the certified radius and reward lower bound for DQN and the reward lower bound for PPO. We provide the details of how we estimate and compute those bounds in practice in Appendix A9.  As for the DS-DQN, simply removing the noise that is already known cannot improve the robustness. The denoiser should be viewed as a part of the agent, which does not necessarily denoise the state to the original one since we also use the temporal difference loss. The denoiser makes the agents more tolerant against the Gaussian noise and hence the problem of decreasing clean reward can be mitigated. Since DS-DQN is more tolerant against the noise, a much larger smoothing variance is applied to enhance the robustness.

Reviewer comment (Writing):
Is a denoiser model used for the continuous (AS-PPO) case? What constitutes the smoothing component in this PPO scenario? What does ΔS_t represent in the paragraph following Equation 9?

Author response:
**A4:** We did not use the denoiser for the PPO case since the input state is not an image. PPO agents are much more tolerant to the Gaussian noise and allow us to directly train the agent with randomized smoothing. The $\Delta S_t$ is the perturbation from the adversary that we use while training AS-PPO. In the stage of optimizing the adversary, we intend to find a set of perturbations  ${\Delta S_1, …, \Delta S_T}$ that minimizes the cumulative reward.

Reviewer comment (Evaluation):
I am not clear on why this denoiser method helps in achieving a better trade-off between robustness and reward. A detailed description would assist in understanding the empirical performance.

Author response:
**A5:** The naively smoothing strategy in CROP decreases the clean reward since the agents are not trained to tolerate the large Gaussian noise. The larger the smoothing variance is, the more the clean reward decreases. We introduce a denoiser to make agents more tolerant to the Gaussian noise. Therefore, we can make the smoothing variance larger, which enhances the robustness guarantee, without reducing the clean reward.

Reviewer comment (Experiments):
Are the attack models for discrete (DQN) and continuous (PPO) cases the same?

Author response:
**A6:** There are all L-p norm attacks, but the PPO case is more complicated since there is no Q-value to reduce. For the PPO attacks, please refer to [2] for more details. Our PPO attacks are based on their methods. 

[2] Zhang, et al. "Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations", NeurIPS 2020

Reviewer comment (Presentation):
It would enhance clarity to use pseudocode diagrams to illustrate the algorithms' flow. Visual aids can make the presentation of the algorithms more accessible.

Author response:
**A1:** Please refer to the block diagram in Appendix A2 and A3, which gives a general idea of how the algorithms work.

Reviewer comment (Evaluation):
The paper falls short in the discussion of limitations. A more comprehensive exploration of potential limitations would provide a well-rounded view.

Author response:
**A2:** A limitation of the guarantees can be the curse of dimensionality. Randomized smoothing provides a guarantee under L-2 norm bounded attack, which can generalize to any L-p norm attack. However, in the high dimensional space, there will be very small or no guarantee for L-inf attacks, because the guarantee will shrink by a factor of $\sqrt{d}$ where $d$ is the dimension.

Reviewer comment (Experiments):
It's disappointing that the paper does not include ATLA[1] and WocaR-RL[2] as robust baselines. While the selected baselines do provide a reasonable basis for comparison, it is necessary and valuable to discuss these adversarial robust RL papers (as well as others that are not mentioned) in the related work section. I am concerned that the literature survey of robust RL baselines by the authors might not be comprehensive, especially with regard to recent works. (The authors do cite [1], but why do not discuss ATLA?)

Author response:
**A3:** Thank you for your suggestions! Following your suggestions, we have conducted additional experiments to include more baseline of Radial-RL, ATLA, and WocaR-RL in Tables A.1, A.2,  B.1, and B.2. 

The DQN results are shown in Table A.1 and A.2. We compare our DS-DQN (Vanilla), which only leverages denoised smoothing without further using a robust base model, with WocaR-DQN. The ATLA algorithm can only apply to the PPO setting, and hence no need to include it here. WocaR-DQN performs well only under a small attack budget and is not comparable to our DS-DQN (Vanilla). Although we use a much larger budget to evaluate all the DQN agents. we feel that this is necessary since previous works such as SA-DQN or Radial-DQN already demonstrated strong robustness under a small attack budget.

The PPO results are shown in Table B.1 and B.2. We do an additional comparison with Radial-PPO, ATLA-PPO, and WocaR-PPO in Walker and Hopper environments. Our AS-PPO outperforms these baselines in the Walker environment, while WocaR-PPO and Radial-PPO perform better in the Hopper environment.
 
Nevertheless, We would like to highlight that all these baselines do not come with robustness guarantees like our agents. Our agents are not only empirically robust but also provably robust. It is especially hard to make the provably robust agents also perform well under empirical attack.

**Table A.1.** The reward of DS-DQN and WocaR-DQN under PGD attack in the Pong environment
(Pong) eps|0.01|0.02|0.03|0.04|0.05
---|---|---|---|---|---
Our DSDQN (Vanilla)|19.2+-0.75|**18.4+-2.15**|**19.2+-1.17**|**17.2+-2.56**|**18.8+-1.17**
WocaRDQN|**21.0+-0.0**|-21.0+-0.0|-21.0+-0.0|-21.0+-0.0|-21.0+-0.0

**Table A.2.** The reward of DS-DQN and WocaR-DQN under PGD attack in the Freeway environment
(Freeway) eps|0.01|0.02|0.03|0.04|0.05
---|---|---|---|---|---
Our DSDQN (Vanilla)|**32.6+-1.20**|**31.6+-1.50**|**30.0+-1.10**|**28.0+-1.41**|**23.8+-1.17**
WocaRDQN|21.4 +- 1.36|21.8 +- 1.17|21.0 +- 0.89|21.0 +- 0.63|21.0 +- 1.10

**Table B.1.** The reward of AS-PPO, Radial-PPO, ATLA-PPO, and WocaR-PPO under attacks in the Walker environment
(Walker) eps=0.05|clean|random|critic|MAD|Min RS|Optimal attack
---|---|---|---|---|---|---
AS-PPO (Ours)|4969|5039|**5488**|**5290**|**4323**|**4296**
Radial-PPO|**5251**|**5184**|No result|4494|3572|3320
ATLA-PPO|3920|3779|3915|3963|3219|3463
WocaR-PPO|4156|4244|No result|4177|4093|3770

**Table B.2.** The reward of AS-PPO, Radial-PPO, ATLA-PPO, and WocaR-PPO under attacks in the Hopper environment
(Hopper) eps=0.075|clean|random|critic|MAD|Min RS|Optimal attack
---|---|---|---|---|---|---
AS-PPO (Ours)|3667|3546|**3706**|2916|1558|1500
Radial-PPO|**3740**|**3729**|No result|3214|2141|1722
ATLA-PPO|3487|3474|3524|3081|1567|1224
WocaR-PPO|3616|3633|No result|**3541**|**3277**|**2390**

Reviewer comment (Experiments):
I find the presentation of the experiments to be somewhat confusing. In Table 3, why is the comparison limited to SA-PPO and vanilla PPO, and where are Radial-PPO and other PPO-based baselines? It appears that the proposed method performs better only on larger attack budgets. However, in Table 8, it shows that AS-PPO does not have a significant advantage, or even any advantage, on large budgets.

Author response:
**A5:** We include an additional comparison with Radial-PPO and other works in Table B.1 and B.2. For the performance of our agents, it is true that our agents do not always outperform the previous works under every setting. However, our agents demonstrate a better trade-off between the clean reward, empirical robustness, and robust guarantee. Our agents still have a much better overall performance.

Reviewer comment (Presentation):
In Table 5, the paper fails to clearly demonstrate the advantages of DS-DQN and AS-PPO; the terms "high" or "highest" are quite vague and do not provide an intuitive representation of contributions.

Author response:
**A6:** Table 5 shows that our agents are the first to achieve strong empirical robustness and provable robustness guarantees at the same time. The purpose of Table 5 is to point out that the previous state-of-the-art focused on comparing the empirical robustness, while the part of certifiable agents is still without much exploration.

Reviewer comment (Experiments):
This paper introduces attack methods based on RS. However, the authors do not compare their attack method with the strongest evasion attacks, such as PA-AD[3], nor do they provide a discussion of prior attack methods in the related work.

Author response:
**A7:** Our attack suggests also smoothing the adversary or introducing RS in the PGD attack to enhance the attack against the smoothed agents. Our proposed attack method only makes a difference to the original attack when the victim is a smoothed agent. In fact, the RS setting can be applied to every attack, including PA-AD, to transform the given attack into an attack target at the smoothed agents. Here we give an additional analysis of how our proposed method affects the PA-AD attack in Table C. The method S-PGD and S-PA-AD denotes the smoothed attack which is our proposed method. The results show that in order to effectively reduce the reward of a smoothed agent, one should consider the smoothing variance.

**Table C.** The comparison between the original attack and the attack after smoothing.
DSDQN eps=0.05|No attack|PGD attack|PA-AD|S-PGD attack (Ours)|S-PA-AD (Ours)
---|---|---|---|---|---
Pong|21.0 +- 0.00|19.4 +- 2.33|19.0 +- 2.10|**18.8 +- 1.17**|**-11.8 +- 2.79**
Freeway|33.2 +- 0.40|30.4 +- 1.85|28.4 +- 1.36|**23.8 +- 1.17**|**1.4 +- 0.80**
RoadRunner|35420 +- 5116|16200 +- 1482|41760 +- 5827|**0 +- 0**|**0 +- 0**

Reviewer comment (Experiments):
It would be useful to explain how representative the issues in CROP are. Also, are there recent works that address/avoid these issues already? Fig.1 only compares results with CROP. It would be useful to have results from other methods.

Author response:
**A1:** To our best knowledge, there is another robust RL literature related to randomized smoothing

[1] Kumar, et al. “Policy Smoothing for Provably Robust Reinforcement Learning”, ICLR 2022

CROP proved bounds for randomized smoothing in RL, while [1] focused on showing that the bounds in the Supervised Learning setting cannot directly transfer to RL setting because of the non-static nature of RL. [1] provided another proof for the bounds and pointed out that the proof in CROP might have some issues. These two papers all do simple randomized smoothing during the inference. Hence, the current works of randomized smoothing for RL all fall into the issue of low clean reward when the smoothing variance increases. We consider this issue important since the agents with strong guarantees but poorly performed are unacceptable.

Reviewer comment (Experiments):
The baselines are all before 2022. Are there more recent methods? If so, please compare with them.

Author response:
**A3:** Following your suggestion, we have conducted additional experiments in the rebuttal to provide the comparison of two additional baselines, ATLA [2] and WocaR [3] in Tables A.1, A.2, B.1, and B.2. ATLA [2] trained their agents using adversarial training with the optimal attack they proposed, which is only applicable in the PPO setting. WocaR [3] performed worst-case-aware policy optimization without using an adversary and can be applied to both DQN and PPO settings.

The DQN results are in Table A.1 and A.2. We found that WocaR-DQN can only tolerate a smaller attack budget as the previous works did, which is not comparable to our DS-DQN (Vanilla) without using any robust base agent.

The PPO results are in Table B.1 and B.2. ATLA is generally not comparable to our AS-PPO in both Walker and Hopper environments. WocarR outperforms our AS-PPO in the Hopper environment, while is less robust than AS-PPO under the Walker environment.

We would like to highlight that all these baselines do not come with robustness guarantees like our agents. Our agents are not only empirically robust but also provably robust.

**Table A.1.** The reward of DS-DQN and WocaR-DQN under PGD attack in the Pong environment
(Pong) eps|0.01|0.02|0.03|0.04|0.05
---|---|---|---|---|---
Our DSDQN (Vanilla)|19.2+-0.75|**18.4+-2.15**|**19.2+-1.17**|**17.2+-2.56**|**18.8+-1.17**
WocaRDQN|**21.0+-0.0**|-21.0+-0.0|-21.0+-0.0|-21.0+-0.0|-21.0+-0.0

**Table A.2.** The reward of DS-DQN and WocaR-DQN under PGD attack in the Freeway environment
(Freeway) eps|0.01|0.02|0.03|0.04|0.05
---|---|---|---|---|---
Our DSDQN (Vanilla)|**32.6+-1.20**|**31.6+-1.50**|**30.0+-1.10**|**28.0+-1.41**|**23.8+-1.17**
WocaRDQN|21.4 +- 1.36|21.8 +- 1.17|21.0 +- 0.89|21.0 +- 0.63|21.0 +- 1.10

**Table B.1.** The reward of AS-PPO, ATLA-PPO, and WocaR-PPO under attacks in the Walker environment
(Walker) eps=0.05|clean|random|critic|MAD|Robust Sarsa|Optimal attack
---|---|---|---|---|---|---
AS-PPO (ours)|**4969**|**5039**|**5488**|**5290**|**4323**|**4296**
ATLA-PPO|3920|3779|3915|3963|3219|3463
WocaR-PPO|4156|4244|No result|4177|4093|3770

**Table B.2.** The reward of AS-PPO, ATLA-PPO, and WocaR-PPO under attacks in the Hopper environment
(Hopper) eps=0.075|clean|random|critic|MAD|Robust Sarsa|Optimal attack
---|---|---|---|---|---|---
AS-PPO (ours)|**3667**|3546|**3706**|2916|1558|1500
ATLA-PPO|3487|3474|3524|3081|1567|1224
WocaR-PPO|3616|**3633**|No result|**3541**|**3277**|**2390**

[2] Zhang, et al. “Robust Reinforcement Learning on State Observations with Learned Optimal Adversary”, ICLR 2021

[3] Liang, et al. “Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning”, NeurIPS 2022

Reviewer comment (Experiments):
It would be useful to explain how representative the issues in CROP are. Also, are there recent works that address/avoid these issues already? Fig.1 only compares results with CROP. It would be useful to have results from other methods.

Author response:
**A1:** To our best knowledge, there is another robust RL literature related to randomized smoothing

[1] Kumar, et al. “Policy Smoothing for Provably Robust Reinforcement Learning”, ICLR 2022

CROP proved bounds for randomized smoothing in RL, while [1] focused on showing that the bounds in the Supervised Learning setting cannot directly transfer to RL setting because of the non-static nature of RL. [1] provided another proof for the bounds and pointed out that the proof in CROP might have some issues. These two papers all do simple randomized smoothing during the inference. Hence, the current works of randomized smoothing for RL all fall into the issue of low clean reward when the smoothing variance increases. We consider this issue important since the agents with strong guarantees but poorly performed are unacceptable.

Reviewer comment (Presentation):
One issue I have is that the paper seems to contain two pieces of results, one for discrete and the other for continuous, and they seem to be quite orthogonal. It would be helpful to explain the common components of the schemes.

Author response:
**A2:** In this work, we focused on how to effectively train a smoothed agent that is provably robust. The DQN and PPO parts both leverage randomized smoothing during the training to solve the issue of low clean reward. We feel that it is necessary to show that no matter in the discrete or continuous setting, our framework achieves high clean reward, high robust reward, and strong guarantees simultaneously.

Reviewer comment (Experiments):
The baselines are all before 2022. Are there more recent methods? If so, please compare with them.

Author response:
**A3:** Following your suggestion, we have conducted additional experiments in the rebuttal to provide the comparison of two additional baselines, ATLA [2] and WocaR [3] in Tables A.1, A.2, B.1, and B.2. ATLA [2] trained their agents using adversarial training with the optimal attack they proposed, which is only applicable in the PPO setting. WocaR [3] performed worst-case-aware policy optimization without using an adversary and can be applied to both DQN and PPO settings.

The DQN results are in Table A.1 and A.2. We found that WocaR-DQN can only tolerate a smaller attack budget as the previous works did, which is not comparable to our DS-DQN (Vanilla) without using any robust base agent.

The PPO results are in Table B.1 and B.2. ATLA is generally not comparable to our AS-PPO in both Walker and Hopper environments. WocarR outperforms our AS-PPO in the Hopper environment, while is less robust than AS-PPO under the Walker environment.

We would like to highlight that all these baselines do not come with robustness guarantees like our agents. Our agents are not only empirically robust but also provably robust.

**Table A.1.** The reward of DS-DQN and WocaR-DQN under PGD attack in the Pong environment
(Pong) eps|0.01|0.02|0.03|0.04|0.05
---|---|---|---|---|---
Our DSDQN (Vanilla)|19.2+-0.75|**18.4+-2.15**|**19.2+-1.17**|**17.2+-2.56**|**18.8+-1.17**
WocaRDQN|**21.0+-0.0**|-21.0+-0.0|-21.0+-0.0|-21.0+-0.0|-21.0+-0.0

**Table A.2.** The reward of DS-DQN and WocaR-DQN under PGD attack in the Freeway environment
(Freeway) eps|0.01|0.02|0.03|0.04|0.05
---|---|---|---|---|---
Our DSDQN (Vanilla)|**32.6+-1.20**|**31.6+-1.50**|**30.0+-1.10**|**28.0+-1.41**|**23.8+-1.17**
WocaRDQN|21.4 +- 1.36|21.8 +- 1.17|21.0 +- 0.89|21.0 +- 0.63|21.0 +- 1.10

**Table B.1.** The reward of AS-PPO, ATLA-PPO, and WocaR-PPO under attacks in the Walker environment
(Walker) eps=0.05|clean|random|critic|MAD|Robust Sarsa|Optimal attack
---|---|---|---|---|---|---
AS-PPO (ours)|**4969**|**5039**|**5488**|**5290**|**4323**|**4296**
ATLA-PPO|3920|3779|3915|3963|3219|3463
WocaR-PPO|4156|4244|No result|4177|4093|3770

**Table B.2.** The reward of AS-PPO, ATLA-PPO, and WocaR-PPO under attacks in the Hopper environment
(Hopper) eps=0.075|clean|random|critic|MAD|Robust Sarsa|Optimal attack
---|---|---|---|---|---|---
AS-PPO (ours)|**3667**|3546|**3706**|2916|1558|1500
ATLA-PPO|3487|3474|3524|3081|1567|1224
WocaR-PPO|3616|**3633**|No result|**3541**|**3277**|**2390**

[2] Zhang, et al. “Robust Reinforcement Learning on State Observations with Learned Optimal Adversary”, ICLR 2021

[3] Liang, et al. “Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning”, NeurIPS 2022