Paper: Tree-based Action-Manipulation Attack Against Continuous Reinforcement Learning with Provably Efficient Support (ICLR-2024)

Reviewer comment (Experiments):
Insufficient experimental results in terms of comparison in more complex, realistic environments.

Author response:
### 2.More conventional benchmark

We conducted an additional experiment in a five-dimensional space, the environmental setup of which is detailed in Environment 3 within Appendix A. The corresponding experimental outcomes are presented in Figure 7 of Appendix G.

Reviewer comment (Experiments):
One main concern is the scalability of the algorithm as the numerical experiments were only shown for two extremely simple environments. Could the authors discuss the computational runtime for such the algorithm? It seemed that constructing the binary tree for environments with higher dimensional state spaces/action spaces would be prohibitively expensive. Furthermore, computation aside, the results would be more convincing and much stronger if similar results were shown on a much more conventional benchmark, like the MuJoCo environments.

Author response:
### 1.computational runtime

The computational time complexity of the LCBT algorithm primarily emanates from two aspects:

(1) After each round, the attacker needs to update the corresponding $L$ and $B$ values for each node of every tree, and the complexity of this step is related to the number of nodes in the tree. Specifically, assuming that in the k-th episode, after completion, the attacker updates each node of H trees, according to the proof in Appendix F.1, the upper bound on the number of nodes in a tree is $O(k^E)$, where $E=\log_{2\rho^{-2}}2<1$. Therefore, for the total K episodes, the time complexity of this part is $O(K\cdot H\cdot K^E)$.

(2) When the attacker launches an attack, it is necessary to traverse from the root node to the leaf node (the WorTraverse function), and the complexity of this step is related to the depth of the tree. By employing a proof method akin to that in Appendix F.1, we can ascertain an upper bound for the depth of the tree as $O(\log_{\rho^{-2}}k)$ in episode $k$. Making an extreme assumption that the attacker launches an attack at every step, the time complexity for this part over the total K episodes is $O(K\cdot H \cdot \log_{\rho^{-2}}K)$.

In summary, the overall computational time complexity is given by: $O(H\cdot K^{1+E}+H\cdot K \cdot \log_{\rho^{-2}}K)$.

### 2.More conventional benchmark

We conducted an additional experiment in a five-dimensional space, the environmental setup of which is detailed in Environment 3 within Appendix A. The corresponding experimental outcomes are presented in Figure 7 of Appendix G. 

## Rebuttal 3 (ID: tey3zbNxQU)
## Comment 1.

### 1. runtime

We reran the experiments for the LCBT attack corresponding to Environment 3, recording the time taken to execute the attack algorithm and the total time. The results are presented in the table below, wherein the **"percentage"** represents the ratio of the time taken to execute the attack algorithm up to that point compared to the total time.

|          | **Steps** | **Attack Time(s)** | **Total Time(s)** | **Percentage** |
| -------- | --------- | ------------------ | ----------------- | -------------- |
| **DDPG** | 5e5       | 659                | 3948              | 16.71%         |
|          | 1e6       | 1763               | 8338              | 21.14%         |
| **TD3**  | 5e5       | 1329               | 4379              | 30.35%         |
|          | 1e6       | 3375               | 9599              | 35.16%         |

Additionally, we included a figure (Figure 8) in Appendix G displaying the variation of the **"percentage"** metric with respect to the number of steps.

### 2. scalability

Here, we shall begin by conducting a theoretical analysis of the circumstances surrounding the method proposed by us in high-dimensional settings. One obvious point is that our method requires partitioning of the action and state spaces. With higher dimensions in the action space, even if we use binary tree partitioning, the sufficiency of partitioning will still be lower than in lower dimensions with the same number of partitions. This means that the LCBT algorithm converges slower towards the worst action, resulting in a higher level of non-stationarity in the environment and a larger **D-Regret**. According to Theorem 2, this also leads to higher attack cost.

Additionally, in the new experiment, the LCBT algorithm demonstrates commendable performance. Hence, in practical terms, factors influencing the efficacy of the LCBT algorithm encompass not only its intrinsic parameters such as $\nu_1$, $\rho$, $d_s$ (For instance, the settings of $\nu_1$ and $\rho$ can influence the speed of tree node expansion), but also encompass the environment and the algorithms employed by the agent.

Reviewer comment (Theory):
Although the paper introduces inventive attack methodologies, the potential complexity of these methods could pose practical challenges for their implementation. Specifically, it would be beneficial to provide an in-depth analysis of the computational complexity associated with the oracle and tree-based attacks.

Author response:
Regarding the oracle attack, the adversary is already cognizant of the worst action, thereby enabling a direct manipulation replacement during the attack. At any given step of launching the attack, the time complexity is $O(1)$, and for the overall $K$ episodes, the computational time complexity amounts to $O(|\tau|)$. 

The computational time complexity of the LCBT algorithm primarily emanates from two aspects:

(1) After each round, the attacker needs to update the corresponding $L$ and $B$ values for each node of every tree, and the complexity of this step is related to the number of nodes in the tree. Specifically, assuming that in the k-th episode, after completion, the attacker updates each node of H trees, according to the proof in Appendix F.1, the upper bound on the number of nodes in a tree is $O(k^E)$, where $E=\log_{2\rho^{-2}}2<1$. Therefore, for the total K episodes, the time complexity of this part is $O(K\cdot H\cdot K^E)$.

(2) When the attacker launches an attack, it is necessary to traverse from the root node to the leaf node (the WorTraverse function), and the complexity of this step is related to the depth of the tree. By employing a proof method akin to that in Appendix F.1, we can ascertain an upper bound for the depth of the tree as $O(\log_{\rho^{-2}}k)$ in episode $k$. Making an extreme assumption that the attacker launches an attack at every step, the time complexity for this part over the total K episodes is $O(K\cdot H \cdot \log_{\rho^{-2}}K)$.

In summary, the overall computational time complexity is given by: $O(H\cdot K^{1+E}+H\cdot K \cdot \log_{\rho^{-2}}K)$.

Reviewer comment (Reproducibility):
Where can we get the target policy? How to specify a target policy in a complex task?

Author response:
The essence of a policy lies in mapping states to actions. Therefore, when configuring the target policy, attackers can partition the states into sub-intervals according to their objectives. Each sub-interval corresponds to a target action, or attackers can choose to set target actions only for certain states while leaving the rest unaffected by attacks. In our experiments, we prefer to use a stricter target policy where each state has a corresponding target action. This allows us to evaluate the effectiveness of our algorithm. To achieve this, we introduce additional constraints in the original environment. For instance, in environment 2, we provide a negative reward when the distance between the car and the target point < 1. By training a supplementary model with these extra constraints, we obtain an additional model that represents the attacker's target policy.

Reviewer comment (Experiments):
The environments considered are extremely low-dimensional, even by the standards of formal methods research.

Author response:
We conducted an additional experiment in a five-dimensional space, the environmental setup of which is detailed in Environment 3 within Appendix A. The corresponding experimental outcomes are presented in Figure 7 of Appendix G.

Reviewer comment (Experiments):
There are only two environments.

Author response:
We conducted an additional experiment in a five-dimensional space, the environmental setup of which is detailed in Environment 3 within Appendix A. The corresponding experimental outcomes are presented in Figure 7 of Appendix G.

Reviewer comment (Theory):
It is unclear whether the analysis holds for stochastic policies, since the definitions of \( Q \) and \( V \) both use the notation introduced for deterministic policies.

Author response:
In fact, in this paper, we only require the target policy, i.e., $\pi^{\dagger}$ is a deterministic policy. There is no specific requirement for the reinforcement learning algorithm used by the agent. In addition, we also discuss the impact of the reinforcement learning algorithm (**D-Regret**) used by the agent on our attack cost in the remark of Theorem 2. Also, we carry out experiments to apply the LCBT attack against DDPG, PPO, and TD3. Furthermore, in the proof section, a portion of the proof statements at the beginning of Appendix C (Proof of Theorem 1) could be refined without affecting the subsequent proof (We have performed updates in Appendix C, specifically focusing on the derivation of $\overline{V}^o_1(s^k_1) - \overline{V}^{\pi^k}_1(s^k_1)$).

Reviewer comment (Theory):
Some parts of the proofs seem to rely on the assumption that all policies are deterministic (including the exploration policies of the RL algorithm), but most RL algorithms do not work by evaluating deterministic policies.

Author response:
In fact, in this paper, we only require the target policy, i.e., $\pi^{\dagger}$ is a deterministic policy. There is no specific requirement for the reinforcement learning algorithm used by the agent. In addition, we also discuss the impact of the reinforcement learning algorithm (**D-Regret**) used by the agent on our attack cost in the remark of Theorem 2. Also, we carry out experiments to apply the LCBT attack against DDPG, PPO, and TD3. Furthermore, in the proof section, a portion of the proof statements at the beginning of Appendix C (Proof of Theorem 1) could be refined without affecting the subsequent proof (We have performed updates in Appendix C, specifically focusing on the derivation of $\overline{V}^o_1(s^k_1) - \overline{V}^{\pi^k}_1(s^k_1)$).

Reviewer comment (Theory):
The binary tree method can deal with the low-dimensional continuous action space but may meet problems in high-dimensional space. In high-dimensional action space case, the tree method is not efficient and it may require huge amount of rounds to explore the worst node.

Author response:
Here, we shall begin by conducting a theoretical analysis of the circumstances surrounding the method proposed by us in high-dimensional settings. One obvious point is that our method requires partitioning of the action and state spaces. With higher dimensions in the action space, even if we use binary tree partitioning, the sufficiency of partitioning will still be lower than in lower dimensions with the same number of partitions. This means that the LCBT algorithm converges slower towards the worst action, resulting in a higher level of non-stationarity in the environment and a larger **D-Regret**. According to Theorem 2, this also leads to higher attack cost.

Additionally, we have introduced a new experiment in a 5-dimensional space, as described in Environment 3 in Appendix A. The corresponding experimental outcomes are presented in Figure 7 of Appendix G. Remarkably, under the conditions of a 5-dimensional space, the LCBT algorithm demonstrates commendable performance. Hence, in practical terms, factors influencing the efficacy of the LCBT algorithm encompass not only its intrinsic parameters such as $\nu$, $\rho$, $d_s$, but also encompass the environment and the algorithms employed by the agent.

Reviewer comment (Theory):
The proposed black-box method needs to discretize the continuous state space. This paper partitions it into $M$ subintervals. How can a black-box attacker find a proper discretization of the state space? It seems unrealistic in the high-dimensional state space, as the attack cost and loss are linearly dependent on $M$.

Author response:
For the partitioning of the state space, theoretically, each substate should satisfy $diam_s(S_i)\leq L_s d_s < \Delta^o_{\dagger}+\Delta^{\dagger}_{min}$. In practical implementation, due to the diversity of the environment, for each subinterval, an attacker only needs to ensure that, for different initial states within the same subinterval and adopting the same policy, the generated Q-values are approximately close, with differences not exceeding the disparity between Q-values produced by employing the target policy and the worst policy for the same initial state (equivalently understood as trajectories being roughly similar, with differences not exceeding the trajectory disparities between employing the target policy and the worst policy for the same initial state). Although Q-values cannot be directly computed, there exists a certain degree of assessability in determining the similarity of trajectories.

Reviewer comment (Reproducibility):
What is the method of dividing the continuous subset of the action space when new leaves are generated?

Author response:
When partitioning a node, it is essential to ensure that the action space corresponding to the child nodes satisfies the condition $diam_a(\mathcal{P}_{D,I})\leq\nu_1\rho^D$, where $D$ represents the depth of the child nodes. In practical applications, the action space is multidimensional, allowing for the configuration of a dimension for each layer in a binary tree. Through iterative correspondence, when a node at a certain layer is partitioned to generate its child nodes, the action space can be halved along the corresponding dimension.

Reviewer comment (Experiments):
Can the method be directly used in high-dimensional action space case?

Author response:
Here, we shall begin by conducting a theoretical analysis of the circumstances surrounding the method proposed by us in high-dimensional settings. One obvious point is that our method requires partitioning of the action and state spaces. With higher dimensions in the action space, even if we use binary tree partitioning, the sufficiency of partitioning will still be lower than in lower dimensions with the same number of partitions. This means that the LCBT algorithm converges slower towards the worst action, resulting in a higher level of non-stationarity in the environment and a larger **D-Regret**. According to Theorem 2, this also leads to higher attack cost.

Additionally, we have introduced a new experiment in a 5-dimensional space, as described in Environment 3 in Appendix A. The corresponding experimental outcomes are presented in Figure 7 of Appendix G. Remarkably, under the conditions of a 5-dimensional space, the LCBT algorithm demonstrates commendable performance. Hence, in practical terms, factors influencing the efficacy of the LCBT algorithm encompass not only its intrinsic parameters such as $\nu$, $\rho$, $d_s$, but also encompass the environment and the algorithms employed by the agent.