Paper: Illusory Attacks: Information-theoretic detectability matters in adversarial attacks (ICLR-2024)

Reviewer comment (Evaluation):
the paper does not seem to give much guidance on how to design better defenses against these attacks/ how can illusory attacks allow us to design more robust RL techniques?

Author response:
Thank you for this remark. As stated in the Conclusion, we believe that a crucial line of defense against illusory attacks is the hardening of observation channels. We include preliminary experiments on this in Section 5 ("Robustification using reality feedback"). These experiments suggest that even the partial hardening of observation channels against adversary interference can drastically improve worst-case performance. Our paper also includes empirical evaluations of attack detectors in novel settings, for example, we embed the SOTA detector from [A] in an information-theoretic sequential hypothesis testing framework.

We now clearly point out three directions toward more robust RL techniques in the conclusion. First, research into better detection methods is required, both under the consideration of information-theoretic optimal detection approaches, as well as practical implementations that allow for real-time application. Second, robustified policies should incorporate active detection mechanisms. This would allow victims to actively seek out environment states in which the presence of adversaries may be more detectable. 
Third, we believe that hardening observation channels is an important part of defending against illusory attacks.

[A] Out-of-Distribution Detection for Reinforcement Learning Agents with Probabilistic Dynamics Models, Haider et al., AAMAS 2023

Reviewer comment (Writing):
I'm a little confused by the definition of adversary score, what exactly is the normalization used to compute this value? The paper states that the value is normalized with respect to the "highest adversarial return in each class, as well as the victim’s expected return in the unattacked environment" but I don't understand what this means.

Author response:
We provide the formula for the computation of the adversary score below. Specifically, we first compute the reduction in victim reward, normalized by the reward in the unattacked environment. Then, we normalize by the highest normalized reduction in victim reward for a specific setting (i.e. a specific environment and specific budget). 

$\textrm{adversary score} = \frac{r_{\textrm{unattacked}}-r_{\textrm{attacked}}}{r_{\textrm{unattacked}}-r_{\textrm{attacked}}(\textrm{lowest for setting})}$

Reviewer comment (Presentation):
Many of the figures have only limited explanation. Figure 1 is not very clear—there is a lot going on, and it could only be fully understood after going through the rest of the paper.

Author response:
We have updated the caption of this figure to make it more accessible and welcome any additional feedback here. We have also included an explanatory paragraph in Section 4.

Reviewer comment (Presentation):
Figure 2 is presented with only high-level explanation, and it is unclear what all the arrows and lines mean.

Author response:
We have redone this Figure, added additional explanations about the different graphical elements, and have also updated the caption accordingly. We are thankful for any additional feedback.

Reviewer comment (Writing):
Algorithm 1 is confusing since \( p(\emptyset) \) and \( p(o_\text{old}, a_\text{old}) \) seem to be undefined and there is no caption.

Author response:
$p$ is defined in the MDP definition in the Background (Section 3.1), as the probabilistic state transition function of the MDP (conditioned on the last observation and action), while p($\emptyset$) refers to the initial state distribution. We have now clarified this further in Algorithm 1.

Reviewer comment (Writing):
The definition of additive perturbations on line 134 is a bit confusing. First, why are the perturbations themselves in the state space \( \mathcal{S} \)? Shouldn't they be in \( \mathbb{R}^d \), where \( \mathcal{S} \subseteq \mathbb{R}^d \)? Second, the \( \delta \) notation for a point mass is not previously introduced.

Author response:
Thank you for this remark. We now introduce $\delta$ in Section 3 and have fixed the clerical error in the definition of additive perturbations ($\varphi_t \in \mathbb{R}^d$).

Reviewer comment (Writing):
Line 225: I believe "relative entropy" is actually another term for KL-divergence, and what is meant is "cross entropy." The notation used of \( H[X, Y] \) is standard for "joint entropy," which makes this quite confusing. It would be best to clean up the notation.

Author response:
Thank you for this remark, this was indeed a typo and we had meant to write cross entropy. We have fixed this now and also updated the notation, citing the relevant book to avoid any confusion.

Reviewer comment (Writing):
In Algorithm 1, \( \hat{D}_\text{KL} \) is not defined.

Author response:
We apologize for this mistake. We have now addressed this; $\hat{D}_{KL}$ is a trailing window estimate of the average constraint violation used to update the dual variable $\lambda$.

Reviewer comment (Writing):
(Minor) The paper introduces a considerable amount of notation. I feel like the main results could be conveyed with less mathematical notation, which could be moved to the appendix (a lot of basic RFL notation and concepts)

Author response:
Thank you for this feedback. We have trimmed our notation without compromising on mathematical stringency. We have made several consolidations to the pdf (see details below). We welcome any further advice from the reviewer on how to make our paper more accessible.
- We have moved parts of the POMDP notation from section 3.1 to the appendix. 
- We have moved details on the game-theoretic intractability of the zero-sum game $\mathcal{G}$ between the victim and the attacker to the appendix (see paragraph 3 in section 4.1).
- We have created a new section in the appendix devoted to the illusory objective estimator. We have moved our references to non-parametric density estimation (paragraph 2 in section 4.2), alongside comments on particle filtering and nested estimator bias (formerly at the end of section 4) to this section.

Reviewer comment (Experiments):
(Minor) The evaluation is limited to simple simulated environments

Author response:
We would like to point out that we evaluate on standard benchmarks in the field [A, B, C]. We however agree with the reviewer that future work should investigate more complex environments, as we also state in our conclusion. 

Citations:   
[A] "Robust reinforcement learning on state observations with learned optimal adversary", Zhang, 2021  
[B] "Policy Smoothing for Provably Robust Reinforcement Learning", Kumar et al., ICLR 2022  
[C] "Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning", Liang et al., NeurIPS 2022

Reviewer comment (Evaluation):
(Medium) I'm missing a runtime analysis to compare the efficiency of the different attacks. This would also highlight if the framework could be scaled to more complicated problems

Author response:
Thank you for this remark. We have added an empirical runtime analysis to Appendix A.11. The required wall clock time for training illusory attacks is about $1.6$ times higher than for training standard adversarial attacks (SA-MDP attacks). We acknowledge that this is a significant, albeit not prohibitive, increase in wall clock time. 
At test-time, inference times for illusory attacks are identical to existing attacks as they only consist of a neural network forward pass. Memory requirements are identical.
We found that wall clock time is generally higher in the more complex Hopper and HalfCheetah environments, as compared to the CartPole and Pendulum environments.