Paper: Adversarial Attacks on Combinatorial Multi-Armed Bandits (ICLR-2024)

Reviewer comment (Theory):
The paper focuses on the new notion of attackability that requires the cost to scale polynomial in $m$. This is not well motivated. The paper only mentioned that "in practice, the exponential cost in $m$ can exceed $T$, resulting in vacuous results." Note that $T$ is growing, and we care about how the regret and the attack costs grow in terms of $T$. On the other hand, $m$ is fixed. So why it is more important to focus on the scaling in terms of $m$ than the scaling in terms of $T$?

Author response:
**A.** We thank the reviewer for raising this issue. We think the question has two parts: 1. why consider the dependency on $m$, and 2. why not try to improve the dependency on $T$.

*As for the part 1*, we believe it is not only for the attack, but for the whole bandit theory. Note that the regret of the most basic multi-armed bandit also scales with the number of arms $K$, say $\tilde O(\sqrt{KT})$ for distribution-independent regret bound, and $\tilde O(K\log T)$ for distribution-dependent regret bound. However, people tried hard to improve the dependency on with respect to $K$, since in reality, people cannot have an infinite time horizon $T$. Thus, people considered many structures for the bandit instances, like linear (contextual) bandit [1] and combinatorial semi-bandit [2,3,4], which can have exponential number of arms. Without leveraging the internal structures, the regret bound will be $\tilde O(\exp(K) \sqrt{T})$ for distribution-independent bound and $\tilde O(\exp(K)\log T)$ for distribution-dependent bound, and when $T$ is not that large (i.e., exponential in $K$), the bounds are vacuous. These results highlight the importance of considering the dependency besides $T$.

Similar to the concern in the regret analysis in bandit, when considering the attack cost, people do not have infinite cost for the attack, and thus, constraining the attack cost to be polynomial dependent on different terms is quite important.

*As for the part 2*, it is a really good question. Note that we aim to get a unified theory to attack any potential bandit algorithm with sublinear regret, and thus the exact attack cost with respect to $T$ will be dependent on the victim algorithm. As shown in Corollary 3.7, if the victim algorithm is CUCB, the attack cost is already $O(\log T)$. Although it might be possible to even reduce the dependency, we believe that $O(\log T)$ is already satisfactory and thus it is more important to reduce the dependency on $m$, i.e., reducing it from exponential dependency to polynomial dependency. 


[1] Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web (pp. 661-670).

[2] Wei Chen, Yajun Wang, and Yang Yuan. Combinatorial multi-armed bandit: General framework and applications. ICML 2013.

[3] Wei Chen, Yajun Wang, Yang Yuan, and Qinshi Wang. Combinatorial multi-armed bandit and its extension to probabilistically triggered arms. The Journal of Machine Learning Research, 17(1): 1746–1778, 2016

[4] Branislav Kveton, Zheng Wen, Azin Ashkan, Hoda Eydgahi, and Brian Eriksson. Matroid bandits: Fast combinatorial optimization with learning. arXiv preprint arXiv:1403.5045, 2014.

Reviewer comment (Novelty):
The paper focuses mostly on the polynomial attackability of a CMAB instance in a known environment, i.e., all parameters of the instance such as the reward distributions of super arms and outcome distributions of base arms are given. This white box setting is of limited interest for practice.

Author response:
**A.** We thank the reviewer for raising the concern. While it is true that  the black-box setting is more interesting, it is also harder to study. As the first study on the attackability of CMAB, we focus on analyzing the white-box setting and show that it is difficult to attack such an environment even when all parameters are known. We believe the characterization is highly non-trivial and significantly different from white-box setting considering the hardness example, where we showed an instance can be polynomially attackable when the the environment is known but becomes polynomially unattackable when the environment is unknown.

On the other hand, our attack algorithm is already applied to several applications under black-box setting, such as probabilistic max coverage, online minimum spanning tree and cascading bandits where the problems are always attackable. We can directly apply Algorithm 1 even under black-box setting for these applications. We emphasize that our experiments in Figure 2 (a-d), (g, h) are all attacked by Algorithm 1 under black-box setting.

Reviewer comment (Theory):
Better justify why one should focus on the scaling of $m$ term.

Author response:
**A.** We thank the reviewer for raising this issue. We think the question has two parts: 1. why consider the dependency on $m$, and 2. why not try to improve the dependency on $T$.

*As for the part 1*, we believe it is not only for the attack, but for the whole bandit theory. Note that the regret of the most basic multi-armed bandit also scales with the number of arms $K$, say $\tilde O(\sqrt{KT})$ for distribution-independent regret bound, and $\tilde O(K\log T)$ for distribution-dependent regret bound. However, people tried hard to improve the dependency on with respect to $K$, since in reality, people cannot have an infinite time horizon $T$. Thus, people considered many structures for the bandit instances, like linear (contextual) bandit [1] and combinatorial semi-bandit [2,3,4], which can have exponential number of arms. Without leveraging the internal structures, the regret bound will be $\tilde O(\exp(K) \sqrt{T})$ for distribution-independent bound and $\tilde O(\exp(K)\log T)$ for distribution-dependent bound, and when $T$ is not that large (i.e., exponential in $K$), the bounds are vacuous. These results highlight the importance of considering the dependency besides $T$.

Similar to the concern in the regret analysis in bandit, when considering the attack cost, people do not have infinite cost for the attack, and thus, constraining the attack cost to be polynomial dependent on different terms is quite important.

*As for the part 2*, it is a really good question. Note that we aim to get a unified theory to attack any potential bandit algorithm with sublinear regret, and thus the exact attack cost with respect to $T$ will be dependent on the victim algorithm. As shown in Corollary 3.7, if the victim algorithm is CUCB, the attack cost is already $O(\log T)$. Although it might be possible to even reduce the dependency, we believe that $O(\log T)$ is already satisfactory and thus it is more important to reduce the dependency on $m$, i.e., reducing it from exponential dependency to polynomial dependency. 


[1] Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web (pp. 661-670).

[2] Wei Chen, Yajun Wang, and Yang Yuan. Combinatorial multi-armed bandit: General framework and applications. ICML 2013.

[3] Wei Chen, Yajun Wang, Yang Yuan, and Qinshi Wang. Combinatorial multi-armed bandit and its extension to probabilistically triggered arms. The Journal of Machine Learning Research, 17(1): 1746–1778, 2016

[4] Branislav Kveton, Zheng Wen, Azin Ashkan, Hoda Eydgahi, and Brian Eriksson. Matroid bandits: Fast combinatorial optimization with learning. arXiv preprint arXiv:1403.5045, 2014.

Reviewer comment (Theory):
Can the authors provide the corresponding conditions for the black-box setting?

Author response:
**A.** We thank the reviewer for raising the concern. While it is true that  the black-box setting is more interesting, it is also harder to study. As the first study on the attackability of CMAB, we focus on analyzing the white-box setting and show that it is difficult to attack such an environment even when all parameters are known. We believe the characterization is highly non-trivial and significantly different from white-box setting considering the hardness example, where we showed an instance can be polynomially attackable when the the environment is known but becomes polynomially unattackable when the environment is unknown.

On the other hand, our attack algorithm is already applied to several applications under black-box setting, such as probabilistic max coverage, online minimum spanning tree and cascading bandits where the problems are always attackable. We can directly apply Algorithm 1 even under black-box setting for these applications. We emphasize that our experiments in Figure 2 (a-d), (g, h) are all attacked by Algorithm 1 under black-box setting.

Reviewer comment (Theory):
The limitations of the findings are less discussed. The findings indicate that the polynomial attackability highly depends on the threat model, in which the outcome of the base arms can be modified by the adversary. However, recent research [1-5] discusses different types of adversarial attacks on bandit and RL, including environment poisoning attack and action poisoning attack. In the CMAB system, the environment-manipulation adversary could manipulate the reward function \( r \) and the action-manipulation adversary could manipulate the super arm \( S \). The proposed sufficient and necessary condition of the polynomial attackability is limited to the specific reward-manipulation adversary. If an environment-manipulation adversary can manipulate the reward function, \( \Delta_M \) could be changed and the condition of the polynomial attackability would not work. Example 4.2 (Hard example) is also limited to the specific reward-manipulation adversary. Some statements in the abstract and introduction are inaccurate. A more thorough discussion about the scope and limitation of the findings would be helpful.

Author response:
We appreciate your suggestions on discussing other threat models such as environment poisoning attacks and action poisoning attacks. In the revised version, we have discussed other threat models in related works. We have clearly stated in abstract and introduction that the threat model in our paper is reward poisoning attacks. We would like to emphasize that reward poisoning attack is arguably one of the most popular threat models in adversarial attacks against bandits and RL. As the first paper studying attackability of combinatorial bandits, we focused on reward poisoning and left other threat models as future work. We also added discussion about the limitation of the work in Appendix D, where we offered insights on generalizing attackability condition to other threat models. Please see our revision for details.

Reviewer comment (Writing):
Some statements in the abstract and introduction are inaccurate and the limitations of the findings are less discussed. The reviewer asks whether the author could provide more discussion about the scope and limitation of the findings.

Author response:
We appreciate your suggestions on discussing other threat models such as environment poisoning attacks and action poisoning attacks. In the revised version, we have discussed other threat models in related works. We have clearly stated in abstract and introduction that the threat model in our paper is reward poisoning attacks. We would like to emphasize that reward poisoning attack is arguably one of the most popular threat models in adversarial attacks against bandits and RL. As the first paper studying attackability of combinatorial bandits, we focused on reward poisoning and left other threat models as future work. We also added discussion about the limitation of the work in Appendix D, where we offered insights on generalizing attackability condition to other threat models. Please see our revision for details.

Reviewer comment (Theory):
While Theorem 4.1 establishes the difficulty of successfully targeting general unknown CMAB instances, there remains a potential to execute attacks in particular CMAB settings, such as PMC with **unknown** base arms. This is important since learning from the attacker side is one of the main challenges of attack design in previous works: simple oracle attacks [Jun et al., 2018, Liu and Shroff, 2019] can easily attack known K-armed bandit instances. I would expect more algorithm design and analysis for the unknown environment.

Author response:
**Q.** While Theorem 4.1 establishes the difficulty of successfully targeting general unknown CMAB instances, there remains a potential to execute attacks in particular CMAB settings, such as PMC with unknown base arms. This is important since learning from the attacker side is one of the main challenges of attack design in previous works: simple oracle attacks [Jun et al., 2018, Liu and Shroff, 2019] can easily attack known K-armed bandit instances. I would expect more algorithm design and analysis for the unknown environment.

**A.**  We agree that it is important to further investigate the attackability of CMAB problems in the unknown setting. However, we find that it is generally hard. First for cascading bandits, online MST, and online PMC problem with Greedy oracle, because by the definitions of the problems, every possible super arm $S$ has non-negative gaps $\Delta_S \geq 0$, which makes it almost always attackable in practice. Besides, we also found that it is also possible to reduce our hard example (Example 4.2) to a generalized version of online shortest path problem: for each edge $e$, the cost may come from a general interval $[a_e, b_e]$ instead of $[0, 1]$. We currently do not know any CMAB problem (including the original online shortest path problem) that lies in the middle of problems with $\Delta \geq 0$ (cascading bandit, online MST) and unattackable problems in unknown setting.