Paper: Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies (ICLR-2024)

Reviewer comment (Writing):
The paper is not well written. There are typos, some notation is not defined, and some parts are not very clear.

Author response:
We apologize for the potential confusion and have revised our draft accordingly.

**a)** In the last paragraph of Section 3, we have pointed out that $z$ is sampled from a fixed distribution $\mathbb{P}(z)$, which can be Gaussian or uniform. This distribution is just used to randomize the victim policy and does not affect our algorithm framework. Therefore, in the definition of $J(\pi, \nu)$ we omit the distribution of $z$ and will make it more clear in the revision.

**b)** $\pi^tt$ (\pi^tt) is indeed a typo. It should be $\\{\pi^t\\}\_{t\in[T]}$

**c)** The update of $\omega$ follows the standard adversarial bandit algorithm, EXP3. Essentially, during Algorithm 1, $\omega$ gradually adjusts the probability of selecting each arm according to the observed rewards for each arm in an exponential fashion. Therefore, we did not describe it intuitively in the main text due to the page limit.

**d)** We are sorry for the misuse of "compact". Here by being "compact", we do not mean the mathematical definition of compactness for a set in real analysis. Instead, we just want to convey the intuition that $\tilde{\Pi}$ should not contain redundant policies, which we discussed in depth in Section 4. Therefore, we shall replace "finite and compact $\tilde{\Pi}$" by "small and finite $\tilde{\Pi}$".

**e)** We believe $k$ has been defined in line 3 of Algorithm 2, indicating what the current iteration is.

Again, we thank and accept all the suggestions from the reviewer. We have revised our paper accordingly.

Reviewer comment (Writing):
In the definition of J(\pi,v), in the first expectation, it is missing where z is sampled from.

Author response:
**a)** In the last paragraph of Section 3, we have pointed out that $z$ is sampled from a fixed distribution $\mathbb{P}(z)$, which can be Gaussian or uniform. This distribution is just used to randomize the victim policy and does not affect our algorithm framework. Therefore, in the definition of $J(\pi, \nu)$ we omit the distribution of $z$ and will make it more clear in the revision.

Reviewer comment (Writing):
\pi^tt is not defined (I guess you missed {}, but the same notation appears multiple times).

Author response:
**b)** $\pi^tt$ (\pi^tt) is indeed a typo. It should be $\\{\pi^t\\}\_{t\in[T]}$

Reviewer comment (Theory):
In order to talk about optimality, convergence guarantees, and scalability, in Theorem 4.10, it would be important that the authors report an upper bound of K.

Author response:
We thank the reviewer for bringing this up. The reason why we do not discuss the upper bound of $K$ is as follows
- **A useful upper bound should be problem-dependent and thus highly non-trivial to analyze. Instead, we choose to give more qualitative intuitions of whether $K$ will be large or small.** Quantitatively, such an upper bound should be ideally related to many factors, such as the attack budget, Lipschitzness of the transition, and rewards as pointed out by Remark 4.4., etc. Therefore, we find it is an extremely complicated problem to give a precise relationship between the upper bound and those problem-dependent quantities. But intuitively, we do know lower attack budget or Lipschitzness indicates a smaller $K$.
- **A relatively direct and coarse upper bound will be the size of all deterministic policies.** This is because any (potentially random) policy can be expressed as a mixture of deterministic policies by Lemma B.1. Therefore, the deterministic policy set incurs $0$ optimality gap. Meanwhile, we also know our algorithm can find the smallest policy set that incurs $0$ optimality gap. Thus, the size of $\tilde{\Pi}$ will always be upper bound by the size of deterministic policy space.
- **The convergence guarantee means our method always converges to the policy set incurring $0$ optimality gap, while scalability means it always finds such a policy set of the smallest size.** Therefore, our method is the most efficient in terms of identifying such a desirable policy space. The exact size of the smallest non-dominated policy set is determined by the problem nature instead of our algorithm.

Reviewer comment (Evaluation):
The reviewer is confused by the result in Table 1. From the results it seems that the proposed approach obtains higher rewards against worst case perturbations compared to methods explicitly trained against worst case perturbations. A discussion on the results and about their interpretation should be given.

Author response:
We do agree with the reviewer that further discussions on the improvement against worse-case attacks are valuable. 

**The reason why we also achieve improved robustness against worst-case attackers is due to the limited algorithmic efficiency and instability of baselines**. It is worth pointing out that all related works [1, 2, 3] share the same objective of maximizing the worst-case robustness. However, in terms of empirical performance, [2] improves over [1] and [3] improves over [2]. Therefore, it is highly likely that existing approaches may not find the global optimal solution reliably. Specifically, one possible explanation from the perspective of optimization is that baselines like [1, 2, 3] could often converge to a local optimum, while our approach **explicitly encourages the new policy to behave differently in the reward space** compared with policies discovered already, thus not stuck at a single local optimal solution easily. To further verify our hypothesis, we design the experiments as follows. 

Firstly, note the number we report in Table 1 is a medium number of rewards for **$21$ victims agents** following the setup of related works [1, 2, 3]. To verify our hypothesis, here we compare the performance of the best victim and the medium victim of different approaches. 

| Environments  | Model       | PA-AD (against medium victim) | PA-AD (against best victim) |
| -----------   | ----------- | -------------- | --------------- |
| Hopper        | PA-ATLA-PPO | $2521\pm325$   | $3129\pm316$    |
|               | WocaR-PPO   | $2579\pm229$   | $3284\pm193$    |
|               | Ours        | $2896\pm723$   | $3057\pm809$    |
| Walker2d      | PA-ATLA-PPO | $2248\pm131$   | $3561\pm357$    |
|               | WocaR-PPO   | $2722\pm173$   | $4239\pm295$    |
|               | Ours        | $5803\pm857$   | $6052\pm944$    |
| Halfcheetah   | PA-ATLA-PPO | $3840\pm273$   | $4260\pm193$    |
|               | WocaR-PPO   | $4269\pm172$   | $4579\pm368$    |
|               | Ours        | $4411\pm718$   | $4533\pm692$    |
| Ant           | PA-ATLA-PPO | $2986\pm364$   | $3529\pm782$    |
|               | WocaR-PPO   | $3164\pm163$   | $4273\pm530$    |
|               | Ours        | $4312\pm281$   | $4406\pm329$    |

**We notice that baselines like [2, 3] can match the high performance of ours in terms of the best run, while the median is low by a large margin.** This shows baseline methods can achieve high worst-case robustness occasionally as its objective encourages so explicitly but not reliably. In contrast, our methods are much more stable. In conclusion, we believe there is room for improvement regarding worst-case robustness even if [1, 2, 3] explicitly maximizes it.

Reviewer comment (Experiments):
The part about adaptive attacks seems somewhat weak -- namely, if the adversary has more knowledge about the defender, what can happen?

Author response:
- **If the adversary has more knowledge, it could develop a more effective attacking strategy.** If the attacker knows $\tilde{\Pi}$ in advance, it could attack each policy in $\tilde{\Pi}$ and forms a corresponding $\tilde{\Pi}^{\text{attack}}$. With $\tilde{\Pi}^{\text{attack}}$, the attacker could also run a bandit-type algorithm to adapt to the victim.

Reviewer comment (Theory):
Questions regarding definition of regret (Definition 4.2): It is unclear to me why the notion of regret does not allow $\pi \in \Pi$ to be picked adaptively at each time step in the same way that $\pi^t \in \Pi$ is picked adaptively. While this may not matter in the case of an infinite $\Pi$ given that the adaptive policy is likely contained within $\Pi$, once $\Pi$ is restricted to $\tilde{Pi}$ with finite cardinality, this may no longer be the case. In other words, I wonder whether the notion of regret used in this work may be too weak.

Author response:
We thank the reviewer for bringing up such an important and insightful question. 

- **Our regret definition is of the most standard form and naturally improves related works aiming for a single static robust policy.** Our Definition 4.2 follows the most standard way of defining regret by comparing it with the **best fixed policy in hindsight**. Therefore, it is a commonly accepted notion for online adaptation and serves as the first step for studying other stronger notions. In addition, since all existing works aim for a single static robust policy, it is also quite natural for us to compare with the best fixed policy in our definition of regret. 

- **The regret notion proposed by the reviewer is also compatible with our algorithm framework.** Indeed, as what the reviewer describes, when the comparison policy $\pi$ is also picked adaptively in Definition 4.2, this leads to an important variant of regret called **dynamic regret**. To be concrete, it is defined as follows
$$
\text{D-Regret}(T):=\sum_{t=1}^{T}\max_{\pi\in\Pi}J(\pi, \nu^t) - J(\pi^t, \nu^t)
$$
Compared with the regret defined in our paper as 
$$
\text{Regret}(T):=\max_{\pi\in\Pi}\sum_{t=1}^{T}J(\pi, \nu^t) - J(\pi^t, \nu^t),
$$
the dynamic regret always holds that $\text{D-Regret}(T)\ge \text{Regret}(T)$. Therefore, low dynamic regret is a stronger guarantee than low regret. 

To see how our framework can deal with the dynamic regret:
1. **For Algorithm 2 (iterative discovery): It is indifferent to the choice of regret or dynamic regret.** In terms of this new and stronger regret, discovering $\tilde{\Pi}$ with small $\text{Gap}(\tilde{\Pi}, \Pi)$ still suffices since our Proposition 4.7 can be extended in a direct way to dynamic regret:
$$
\frac{\operatorname{D-Regret}(T)}{T}\le \frac{\tilde{\operatorname{D-Regret}}(T)}{T} + \operatorname{Gap}(\widetilde{\Pi}, \Pi).
$$
Therefore, as long as $\operatorname{Gap}(\tilde{\Pi}, \Pi)$ is small, using $\tilde{\Pi}$ instead of $\Pi$ for online adaptation sacrifices no optimality even in terms of the new stronger regret metric.
2. **For Algorithm 1 (online adaptation): To ensure low dynamic regret, the attacker policy is required to change slowly and then Algorithm 1 can be replaced with a dynamic regret minimization bandit algorithm. Otherwise, it is intrinsically hard.** If the attacker policy is changing fast and arbitrarily, even $\tilde{\operatorname{D-Regret}}(T)$ cannot be sublinear [1, 2], let alone $\operatorname{D-Regret}(T)$. However, if the attacker policy is changing slowly, our framework readily works by replacing Algorithm 1 with those algorithms developed for bandit dynamic regret minimization under various types of conditions that rewards of bandit arms are changing slowly, for e.g., [1, 2]. It is worth mentioning that the key algorithmic design for those algorithms is simply restarting Algorithm 1 periodically or maintaining a finite window.

Reviewer comment (Experiments):
Questions regarding experimental results: While the aim of the paper is stated as improving performance against no attacks/non-worst-case adversarial attacks while *maintaining* performance against worst-case adversarial results, the empirical results show the proposed method *exceeding* performance against worst-case adversarial attacks compared to baselines (in some cases, substantially - see e.g. Walker2d vs. PA-AD). This seems to contradict "no free lunch" intuitions. The authors should further explain why we are seeing this behavior in the experiments (potentially through the use of ablation studies), and whether there are any tradeoffs.

Author response:
We do agree with the reviewer that further discussions on the improvement against worse-case attacks are valuable. 
- **The experimental results do not contradict "no free lunch" intuition**. The improvement of our approach comes from utilizing the online reward feedback while the baseline approaches neglect such information by deploying a static victim. Therefore, by adapting to the unknown attacker, our approach achieves robustness against both worst-case and non-worst-case attackers. In other words, if we also just deploy a single static victim as in baseline approaches, we agree with the reviewer that there is an inevitable trade-off such that improved robustness against worst-case attackers and non-worst-case attackers might not be achieved at the same time due to the "no free lunch" intuitions.
- **The reason why our method improves robustness against worst-case attackers is due to the limited algorithmic efficiency and instability of baselines**. It is worth pointing out that all related works [3, 4, 5] share the same objective of maximizing the worst-case robustness. However, in terms of empirical performance, [4] improves over [3] and [5] improves over [4]. Therefore, it is more likely that existing approaches do not find the global optimal solution reliably or efficiently. One possible explanation from the perspective of optimization is that baselines like [3, 4, 5] could often converge to a local optimum, while our approach explicitly encourages the new policy to behave differently in the reward space compared with policies discovered already, thus not stuck at a single local optimal solution easily. To further verify such a hypothesis, we design the experiments as follows. Note the number we report in Table 1 is a medium reward of **$21$ trained victim agents** following the setup of related works [3, 4, 5]. Therefore, here we report the performance of the best one and the median of different approaches. 

| Environments  | Model       | PA-AD (against medium victim) | PA-AD (against best victim) |
| -----------   | ----------- | -------------- | --------------- |
| Hopper        | PA-ATLA-PPO | $2521\pm325$   | $3129\pm316$    |
|               | WocaR-PPO   | $2579\pm229$   | $3284\pm193$    |
|               | Ours        | $2896\pm723$   | $3057\pm809$    |
| Walker2d      | PA-ATLA-PPO | $2248\pm131$   | $3561\pm357$    |
|               | WocaR-PPO   | $2722\pm173$   | $4239\pm295$    |
|               | Ours        | $5803\pm857$   | $6052\pm944$    |
| Halfcheetah   | PA-ATLA-PPO | $3840\pm273$   | $4260\pm193$    |
|               | WocaR-PPO   | $4269\pm172$   | $4579\pm368$    |
|               | Ours        | $4411\pm718$   | $4533\pm692$    |
| Ant           | PA-ATLA-PPO | $2986\pm364$   | $3529\pm782$    |
|               | WocaR-PPO   | $3164\pm163$   | $4273\pm530$    |
|               | Ours        | $4312\pm281$   | $4406\pm329$    |

- **We notice that baselines like [4, 5] can almost match our method's high performance in terms of the best run, while the average or median is low by a large margin**. This means baselines can also achieve strong worst-case robustness occasionally as its objective explicitly encourages so, but not reliably. In contrast, our methods are much more stable. In conclusion, we believe there is room for improvement regarding worst-case robustness even if [3, 4, 5] maximizes it explicitly.

Reviewer comment (Writing):
Section 3: $H$ does not seem to be defined.

Author response:
$H$ here indicates the horizon of the MDP, which we describe in the second paragraph of Section 3 for the first time. We shall clarify the meaning of $H$ more clearly in our revision.

Reviewer comment (Writing):
Section 3: Typo - "discounted factor" --> "discount factor".

Author response:
We thank the suggestion by the reviewer and will revise accordingly.

Reviewer comment (Theory):
Proposition 4.3: How does the choice of $\alpha$ relate to the rest of the problem setup?

Author response:
$\alpha$ indicates the growth of the regret. **We aim to develop an algorithm with the corresponding $\alpha<1$ so that $\lim_{T\rightarrow +\infty}\frac{\text{Regret}(T)}{T}= 0$, i.e., the policy sequence $\\{\pi^t\\}_{t\in[T]}$ is asymptotically optimal.** In Proposition 4.5 & 4.7, the corresponding $\alpha$ is $\frac{1}{2}$.

Reviewer comment (Theory):
Why is the notion of regret that is defined not overly weak?

Author response:
As in our response to Q1, our regret is not weak but standard and naturally improves over existing works. Meanwhile, our framework also readily adapts to stronger regret notions like dynamic regret.

Reviewer comment (Evaluation):
Why are the experimental results not too good to be true? Are there any tradeoffs inherent in the method or results?

Author response:
As in our response to Q2, the improvement in worst-case robustness comes from the limited algorithmic efficiency and instability of baselines. One trade-off we realize is between exploration and exploitation during online adaptation. If the online adaptation stage is too short for Algorithm 1 to identify a reliable mixture policy, simply deploying a single static robust policy like [3, 4, 5] could potentially outperform our method.

Reviewer comment (Theory):
The authors propose the new regret in equation 4.1 that is based on the value function rather than reward. I feel it is not fully discussed why such a regret is appealing and how it is related with the commonly used reward-based regret.

Author response:
**Firstly, our regret definition follows the most standard definition for RL problems, which is indeed based on the value function, e.g. [5, 6, 7, 8].** We agree that the definition of regret is usually related to rewards. However, even in the most classical bandit setting, regret is defined w.r.t the **expected reward** of each arm (e.g. Section 4 of [4]) while the exact reward feedback for the agent is a **random variable**. Therefore, our definition follows such conventions by defining regret based on the **expected accumulated rewards**, i.e., **the value function**. More importantly, defining a regret based on the value function is **NOT** proposed by us but quite common and well accepted in the RL literature, e.g. [5, 6, 7, 8], which all define regret based on the value function. Essentially, our contribution is not on proposing a brand new regret but on properly formulating and then solving our robust RL problem under the regret minimization framework. 

**Secondly, it's also possible to define regret using the exact reward as suggested by the reviewer, and it only changes the presentation of our results slightly.** Note that such a regret will become a random variable and the theoretical statements in our paper will be bounding the expectation of the regret. Therefore, defining regret based on the reward or its expectation are just different ways of presenting our results.

Reviewer comment (Writing):
In the paragraph below equation 4.1, the term $\pi^tt\in[T]$ seems like a typo. Should it be {$\pi_t: t\in [T]$} instead?

Author response:
Yes, we thank the reviewer for pointing out the typo. We have revised our draft accordingly.

Reviewer comment (Theory):
As pointed out by the authors in Theorem 4.11, the size of $\tilde{\Pi}$ can be very large in some cases, being exponential in the time horizon $H$.

Author response:
1. **Although the case requiring a large $\tilde{\Pi}$ is theoretically possible, it is an extreme case not taking the properties or structures of common practical applications into consideration**. The problem we constructed to prove Theorem 4.11 requires the attacker to perturb a state to another state with highly different reward functions. However, practical problems often pose certain constraints on the attacker, like bounded $l_p$ norm, and the reward function can be lipschitz. Thus, such a problem may not appear in practical applications. 

2. **According to experiments, it is highly possible that the contrived problem instance we delicately designed in Theorem 4.11 will not appear**. With experiments on all environments (Section 4), small $|\tilde{\Pi}|$ within $5$ suffices for strong empirical performance. This further validates the practicality of our framework and the contrived problem may not happen.

3. **For practical infinite-horizon problems with a universal constant discount factor, the exponential dependency does not appear anymore**. As in our response to Q1, for infinite-horizon problems, we can construct a finite-horizon problem $H=\mathcal{O}(\frac{\log 1/\epsilon}{1-\gamma})$, where $\epsilon$ is the approximation accuracy. Therefore, if $\gamma$ is set as a universal constant as in most empirical implementations, dependency like $2^{H}$ will become $2^{\mathcal{O}(\frac{\log 1/\epsilon}{1-\gamma})}=\text{poly}(\frac{1}{\epsilon})$, where the exponential dependency will disappear. Therefore, in such cases, our hardness result may not hold anymore either.

Reviewer comment (Theory):
Based on Theorem 4.11, will the method be very likely to fail for infinite horizon MDP?

Author response:
As in our response to Q1, our approach straightforwardly handles the infinite horizon MDP using a simple truncation. The approximation only has $H = \mathcal{O}(\frac{\log 1/\epsilon}{1-\gamma})$ steps, where $\epsilon$ is the approximation accuracy. Meanwhile, such a theoretical reduction is also compatible with empirical implementations, where even for infinite-horizon problems, one usually just samples finite steps. In conclusion, our method readily applies to infinite-horizon MDP.