Paper: Belief-Enriched Pessimistic Q-Learning against Adversarial State Perturbations (ICLR-2024)

Reviewer comment (Evaluation):
While the diffusion model adds robustness, it also adds computational overhead, potentially making it slower at test time.

Author response:
A4:  We admit that using a diffusion model introduces additional overhead to our methods, especially for the DP-DQN-O method where the DDPM model is used. In this work, we considered a fast diffusion method called Progressive Diffusion in our DP-DQN-F method. As shown in Table 2(b), DP-DQN-F has significantly lower training and testing overhead than DP-DQN-O. To further speed up state purification, one possibility is to change the U-net structure used by the diffusion models to accelerate the reverse process. Another direction is to utilize other generative models such as GANs and autoencoders for state purification, which may not be as effective as diffusion-based methods but incur less computational overhead.

Reviewer comment (Experiments):
Given the requirement for a clean training environment, how would your method perform in a scenario where such an environment is not readily available?

Author response:
A2: We have added a general response to this question.

Reviewer comment (Experiments):
Why PA-AD is not evaluated on continuous gridworld?

Author response:
A3:  Following the reviewer’s suggestion, we have adapted PA-AD to our continuous Gridworld environment and tested our methods and other baselines. The results are as follows: 
|  Env |  Model | PA-AD | PA-AD |
|:---:|:---:|:---:|:---:|
|  |  | $\epsilon = 0.1$ | $\epsilon = 0.5$ |
| Continuous Gridworld | DQN | $-10.7 \pm 136$ | $-35.9 \pm 118 $ |
| Continuous Gridworld | SA_DQN | $-97.5 \pm 13.6$ | $-67.8 \pm 78.3$ |
| Continuous Gridworld | WocaR_DQN | $-100 \pm 0$ | $-63.2 \pm 88.6$ |
| Continuous Gridworld | BP_DQN | $171.9 \pm 17$ | $177.2 \pm 10.6$ |

Similar to other attack baselines reported in our paper, our BP-DQN method achieves the best robustness under the PA-AD attack.

Reviewer comment (Theory):
It has been shown that Stackelberg Equilibrium as defined in Definition 2 need not always exist, refer to theorem 4.3 [https://arxiv.org/pdf/2212.02705.pdf](https://arxiv.org/pdf/2212.02705.pdf). So, finding an approximate solution for them is meaningless. However, non-existence of Stackelberg equilibrium is a worst case phenomenon. Authors should incorporate this in their paper.

Author response:
We are aware of the negative result the reviewer mentioned. Please see the discussion below Definition 2 in Section 3, where we have cited the SA-MAP paper where this result was originally proved (Theorem 4 in the SA-MDP paper). We are also aware of the paper the reviewer mentioned and have cited its journal version in the Related Work section (Appendix B.1). The negative result implies that a policy that obtains the best possible value (against the strongest attack) across all states does not exist in general, but it does not exclude the possibility of finding an approximate solution that is close to the optimal state-wise. Although we were not able to identify a policy that obtains an approximate Stackelberg Equilibrium in the standard sense, we showed that our pessimistic Q learning algorithm (Algorithm 1) obtains a value function that is close to the optimal Q function without attacks (Theorem 1) under common assumptions.

Reviewer comment (Presentation):
It will be great if authors can include a short paragraph before section 3.2 discussing a big picture of their strategies before diving into each one of them. It would also help to include some mathematical details in each section.

Author response:
We present our basic solution framework, pessimistic Q-learning, in Section 3.3. Our approach derives maximin actions from the Q-function using perturbed states as input to safeguard against the agent’s uncertainty about true states. Notably, this approach is applied at both training and test stages, thus bridging the gap between the two as in previous work. We then improve the algorithm by incorporating the agent’s belief about true states into action selection in Section 3.4. To obtain a practical solution in high-dimensional state spaces, we present our BP-DQN algorithm in this section, which incorporates the above ideas into the classic Deep Q-Network (DQN) algorithm to obtain pessimistic DQN with belief states approximated using the PF-RNN method. To further scale our method to environments with raw-pixel input, we present another algorithm, DP-DQN, in Section 3.5, where we incorporate a diffusion-based purification scheme into pessimistic DQN to purify the pixel-wise perturbations.

Reviewer comment (Writing):
Abstract wrongly mentions that past methods either regularize or retrain the policy. However, methods like Bharti et.al. just purify the states directly.

Author response:
We would like to thank the reviewer for pointing out the work beyond regularization-based methods and alternating training methods. We are aware of the purification-based methods, including the work by Bharti et al., and have provided a detailed comparison between our method and Bharti et al. in Appendix B.3. In particular, our method has two important components not considered in Bharti et al., namely, the maximin formulation and belief update. There also exist studies that detect if a state is perturbed or not at the test stage, e.g., Xiong et al., which uses an autoencoder-based method to detect perturbations, as we mentioned in Section 3.3 and Appendix B.1. We did not mention them in the abstract because Bharti et al. focuses on the backdoor attack rather than test-stage state perturbation we considered, while Xiong et al. focuses on test stage detection given a pre-trained policy.

Reviewer comment (Theory):
It is well known that defense against perceived state attack requires solving a partially observable MDP which is a hard problem to solve in general. Could you clarify how your method is able to avoid these hardness issues?

Author response:
We first note that while training a robust agent against adversarial state perturbations is closely related to POMDPs, they are also fundamentally different. In POMDPs, the agent’s observation is derived from a predetermined observation function as part of the environment. In contrast, the agent’s observation is determined by the attacker’s policy in adversarial state perturbations, which might be non-stationary or even adaptive, making the problem even harder. 

In this work, we have exploited two key observations to bypass this obstacle. First, the attacker has a bounded budget to avoid detection, where the perturbed state must land in the \epsilon ball centered at the true state so that the agent could utilize historical information to generate a relatively accurate belief about true states, which might not be the case in general POMDPs. Further, for environments with raw-pixel input, existing attacks generate perturbations in the form of pixel-wise noise, which could be purified with the help of a diffusion model to generate a belief about the true state. These ideas together with our pessimistic training framework ensure high robustness against strong state perturbation attacks.

Reviewer comment (Experiments):
it would be interesting to run an ablation on these assumptions. How well does the method work if the budget is not known exactly, or if the MDP transition function is not known exactly?

Author response:
Q2: Our BP-DQN method does not require accurate knowledge about the attack budget. The results in the paper were obtained using a single policy trained against the PGD attack with $\epsilon = 0.1$, without further adaptation at the test stage against the actual attack budget (0.1 or 0.5). 

Our DP-DQN methods also do not require accurate knowledge about the attack budget at the training stage. All the results shown in the paper for DP-DQN-O (resp. DP-DQN-F) were obtained using a single policy trained against the PGD attack with $\epsilon = 1/255$, and with the reverse steps $k= 5$ in DDPM (resp. $k=1$ in Progressive Distillation). However, we did adapt the value of $k$ at the test stage according to the actual attack budget in the paper. To understand the performance of our algorithms under an unknown attack budget at the test stage, we have conducted a new ablation study on these methods using the Pong environment with a fixed number of reverse steps in the diffusion model ($k = 30$ for DP-DQN-O and $k = 1$ for DP-DQN-F) and a fixed level of manually added noise (1/255) at the test stage, and tested them on PGD attacks with three different attack budgets (1/255, 3/255 and 15/255). The results for DP-DQN-O are $\epsilon = 1/255: 20 \pm 0$, $\epsilon = 3/255: 19.8 \pm 0.4$, $\epsilon = 15/255: 19.7 \pm 0.5$, which are on par with the results using an adapted $k$ shown in the paper. On the other hand, the results for DP-DQN-F are $\epsilon = 1/255: 20.4 \pm 0.9$, $\epsilon = 3/255: 13 \pm 7.4$, $\epsilon = 15/255: -20.6 \pm 0.4$, showing that it is more sensitive to attack budget. As a comparison, we have also tested other baselines using the policy trained under $\epsilon = 1/255$ against a PGD attack with budget $\epsilon = 3/255$, and the results are SA-DQN: $12 \pm 1.2$, WocaR-DQN: $-21 \pm 0$, showing that these methods are also sensitive to attack budget.