Paper: Constructing Adversarial Examples for Vertical Federated Learning: Optimal Client Corruption through Multi-Armed Bandit (ICLR-2024)

Reviewer comment (Theory):
The VFL setting as explained at the top of Section 2 is puzzling. This implies that a prediction query arrives simultaneously at all M clients. But even in the given setting, why does the server need to broadcast the full probability vector, rather than the class with the maximal value? Is there a notion of a primary query client? Or is it more realistic that the query comes to the central server which then asks clients for their representations?

Author response:
In our VFL framework, the server's role is akin to a **remote model/API with a limited query budget**, as outlined in prior works [1] [2] [3]. The query process is initiated by **arbitrarily one of the corrupted clients**, which then collaborates with other clients to submit their embeddings for prediction. There is **no designated 'primary query client'** in this model. 

Regarding the transmission of the full probability vector, our approach aligns with previous studies [1] [2] [3], where an API/remote model query returns various forms of outputs, including probability vectors, prediction probabilities, or hard labels. In our threat model, the server broadcasts the probability vector to all VFL clients. The probability vectors are used to compute the loss in the Inner problem AEG, as shown in equation 3 of section 5. Specifically, in both targeted and untargeted attacks, the adversary calculates the cross-entropy between the probability vector and the target label. It's noteworthy that the essence of our problem formulation and the E-TS solution remains **unaffected** even if the server provides only hard labels or prediction probabilities. In such cases, alternate forms of loss functions can be employed to address the inner problem.

[1] Ilyas et al. (2018) Black-box Adversarial Attacks with Limited Queries and Information.

[2]Yu et al. (2020) CloudLeak: Large-Scale Deep Learning Models
Stealing Through Adversarial Examples

[3] Chen et al. (2020) HopSkipJumpAttack: A Query-Efficient
Decision-Based Attack

Reviewer comment (Reproducibility):
What was the loss function L (in eqn (3), etc.) adopted in the experimental settings?

Author response:
We consider the loss function $L(\boldsymbol{\eta}\^t\_i; \mathcal{C}^t) = l(\boldsymbol{f}\_0(\tilde{\boldsymbol{h}}^t\_{i,a};\boldsymbol{h}\_{i,b}^t), y\_v)$ for the targeted attack with target label $y\_v$, and $L(\boldsymbol{\eta}^t\_{i}; \mathcal{C}^t) = -l(\boldsymbol{f}\_0(\tilde{\boldsymbol{h}}^t\_{i,a};\boldsymbol{h}\_{i,b}^t), y\_u)$ for the untargeted attack with label $y_u$, where $l(\cdot)$ denotes the loss metric, and $\boldsymbol{f}\_0(\tilde{\boldsymbol{h}}^t\_{i,a};\boldsymbol{h}\_{i,b}^t)$ returns the probability vectors of the test sample $i$. In our experiments, we use cross entropy as the loss metric $l(\cdot)$.

Reviewer comment (Reproducibility):
Algorithm 1 – Please identify the inputs: \(t_o, \{B_t, t=1,..,T\}\) and the corresponding embeddings.

Author response:
Thank you for your detailed questions and suggestions. We provide the following clarifications and modifications:

1. Inputs to Algorithm 1: The variable $t_0$ denotes the number of warm-up rounds. $B^t$ represents the batch size at time $t$, with $[B^t]$ indicating the set of indices for the batch data. The corrupted embedding $\boldsymbol{h}_{i,a}^t$ is an aggregation of embeddings from multiple corrupted clients, where $a_1,\ldots, a_C$ are the indices of these clients in the corruption pattern $\mathcal{C}^t$.

2. Call to Algorithm 2: A call to Algorithm 2 for computing the corrupted embeddings is indeed necessary after line 11. This step has been incorporated into line 12 in the revised draft.
3. Pluralization and Rewards: The correction in line 6 to pluralize the term has been finished. Rewards are assumed to be positive, lying in the range of [0,1], as they represent the ASR of batch data.
4. Function of $\hat{\phi}\_k$: $\hat{\phi}\_k$ is defined as **the average of the maximum rewards observed for arm $k$**. It is utilized to **identify competitive arms** by comparing it with the empirical mean $\hat{\mu}_{k^{emp}(t)}$ of the empirical best arm $k^{emp}(t)$. Arms with $\hat{\phi}\_k > \hat{\mu}\_{k^{emp}(t)}$ are selected as competitive, aiding in reducing exploration space.
5. Nature of Reward: The reward in our model is the ASR of batch data for a given corruption pattern. It ranges from 0 to 1, with 1 being the maximum possible reward.

Reviewer comment (Reproducibility):
Algorithm 1 – After line 11, shouldn’t there be a call to Alg 2 to compute the corrupted embeddings?

Author response:
Thank you for your detailed questions and suggestions. We provide the following clarifications and modifications:

1. Inputs to Algorithm 1: The variable $t_0$ denotes the number of warm-up rounds. $B^t$ represents the batch size at time $t$, with $[B^t]$ indicating the set of indices for the batch data. The corrupted embedding $\boldsymbol{h}_{i,a}^t$ is an aggregation of embeddings from multiple corrupted clients, where $a_1,\ldots, a_C$ are the indices of these clients in the corruption pattern $\mathcal{C}^t$.

2. Call to Algorithm 2: A call to Algorithm 2 for computing the corrupted embeddings is indeed necessary after line 11. This step has been incorporated into line 12 in the revised draft.
3. Pluralization and Rewards: The correction in line 6 to pluralize the term has been finished. Rewards are assumed to be positive, lying in the range of [0,1], as they represent the ASR of batch data.
4. Function of $\hat{\phi}\_k$: $\hat{\phi}\_k$ is defined as **the average of the maximum rewards observed for arm $k$**. It is utilized to **identify competitive arms** by comparing it with the empirical mean $\hat{\mu}_{k^{emp}(t)}$ of the empirical best arm $k^{emp}(t)$. Arms with $\hat{\phi}\_k > \hat{\mu}\_{k^{emp}(t)}$ are selected as competitive, aiding in reducing exploration space.
5. Nature of Reward: The reward in our model is the ASR of batch data for a given corruption pattern. It ranges from 0 to 1, with 1 being the maximum possible reward.

Reviewer comment (Theory):
Algorithm 1 – The role of \(\phi\) is not clear. Let \(n(t)\) be the number of times a given arm is played, and let \(r(j)\) denote the corresponding rewards. Then \(\phi = \frac{1}{n(t)} \sum_{j=1}^{n(t)} \max(r_i, i=1,…,j)\). Why is this a useful quantity to compute?

Author response:
Thank you for your detailed questions and suggestions. We provide the following clarifications and modifications:

1. Inputs to Algorithm 1: The variable $t_0$ denotes the number of warm-up rounds. $B^t$ represents the batch size at time $t$, with $[B^t]$ indicating the set of indices for the batch data. The corrupted embedding $\boldsymbol{h}_{i,a}^t$ is an aggregation of embeddings from multiple corrupted clients, where $a_1,\ldots, a_C$ are the indices of these clients in the corruption pattern $\mathcal{C}^t$.

2. Call to Algorithm 2: A call to Algorithm 2 for computing the corrupted embeddings is indeed necessary after line 11. This step has been incorporated into line 12 in the revised draft.
3. Pluralization and Rewards: The correction in line 6 to pluralize the term has been finished. Rewards are assumed to be positive, lying in the range of [0,1], as they represent the ASR of batch data.
4. Function of $\hat{\phi}\_k$: $\hat{\phi}\_k$ is defined as **the average of the maximum rewards observed for arm $k$**. It is utilized to **identify competitive arms** by comparing it with the empirical mean $\hat{\mu}_{k^{emp}(t)}$ of the empirical best arm $k^{emp}(t)$. Arms with $\hat{\phi}\_k > \hat{\mu}\_{k^{emp}(t)}$ are selected as competitive, aiding in reducing exploration space.
5. Nature of Reward: The reward in our model is the ASR of batch data for a given corruption pattern. It ranges from 0 to 1, with 1 being the maximum possible reward.

Reviewer comment (Reproducibility):
Algorithm 1 – What is the reward? Is it merely the ASR over the batch? With reference to Remark 1 and the discussion of [Gupta et al., 2021], is the reward in this setting simply 0 or 1 (so max=1)?

Author response:
Thank you for your detailed questions and suggestions. We provide the following clarifications and modifications:

1. Inputs to Algorithm 1: The variable $t_0$ denotes the number of warm-up rounds. $B^t$ represents the batch size at time $t$, with $[B^t]$ indicating the set of indices for the batch data. The corrupted embedding $\boldsymbol{h}_{i,a}^t$ is an aggregation of embeddings from multiple corrupted clients, where $a_1,\ldots, a_C$ are the indices of these clients in the corruption pattern $\mathcal{C}^t$.

2. Call to Algorithm 2: A call to Algorithm 2 for computing the corrupted embeddings is indeed necessary after line 11. This step has been incorporated into line 12 in the revised draft.
3. Pluralization and Rewards: The correction in line 6 to pluralize the term has been finished. Rewards are assumed to be positive, lying in the range of [0,1], as they represent the ASR of batch data.
4. Function of $\hat{\phi}\_k$: $\hat{\phi}\_k$ is defined as **the average of the maximum rewards observed for arm $k$**. It is utilized to **identify competitive arms** by comparing it with the empirical mean $\hat{\mu}_{k^{emp}(t)}$ of the empirical best arm $k^{emp}(t)$. Arms with $\hat{\phi}\_k > \hat{\mu}\_{k^{emp}(t)}$ are selected as competitive, aiding in reducing exploration space.
5. Nature of Reward: The reward in our model is the ASR of batch data for a given corruption pattern. It ranges from 0 to 1, with 1 being the maximum possible reward.

Reviewer comment (Experiments):
The choice of datasets is poorly motivated for the VFL problem. The credit score and Caltech datasets are reasonable, but segmenting a CIFAR or MNIST image into segments, or an IMDB review at different clients is not reasonable.

Author response:
We notice the unreasonable dataset. Since the scarcity of the VFL dataset, many previous works not only test VFL performance on tabular and multiview datasets but also on CV and NLP datasets by splitting the original data sample (see, e.g., [4] [5] [6]). We follow previous works to test the performance of our attack in distinct tasks. We hope to further investigate and apply more suitable dataset on CV and NLP tasks.

[4] Chen et al. (2020) Vafl: a method of vertical asynchronous federated learning.

[5] D Romanini et al. (2021) Pyvertical: A vertical federated learning framework for multi-headed splitnn.

[6] Li et al. (2023) FedVS: Straggler-Resilient and Privacy-Preserving Vertical Federated Learning for Split Models.

Reviewer comment (Experiments):
A value of Q=1000 is outrageously large. Servers with any reasonable security settings would turn off queries if they are repeated more than 3 or 5 times. A large Q is simply a mechanism for an adversary to learn (more samples). Q = 1000 is not a reasonable number.

Author response:
In our model, we conceptualize the server as an API, similar to how it's approached in previous works [1] [2] [3]. For instance, the Google Cloud Vision API allows a query budget of up to 1800 for a single sample. Our primary focus in this research is on the efficiency of the E-TS algorithm in **selecting the optimal corruption pattern**. The inner Adversarial Example Generation (AEG) problem aligns more closely with traditional adversarial example generation techniques.

To address the concern about the large query budget, first, we conduct a targeted attack with the target label set to 6, under the setting that a query budget of 500, perturbation budget of 0.3, and a corruption constraint of 2, the E-TS best arm results indicate an ASR of 51\%, which indicates even in small query budget, we still can successfully attack half of the samples. Besides, adversaries could duplicate multiple noisy versions of the samples before inference, effectively increasing their query budget more stealthily. Furthermore, recent advancements in query-efficient methods [3] [7] and transfer-based approaches [8] offer alternatives to the AEG problem that do not compromise the efficacy of the E-TS solution.

In sum, we can leverage methods, like **selecting suitable target labels and perturbation budgets, duplicating samples in advance, and using efficient strategies** to mitigate the concern of a limited query budget. We also conduct an analysis of necessary minimum number of queries in **Appendix D**.
Regarding the default parameter settings used in our ablation study, they were as follows: $\beta = 0.3, B^t = 128$, and $C = 2$. We varied test parameters while maintaining the stability of other settings. The comprehensive parameter settings are now detailed in the revised draft and **Table 1 of Appendix C.1**.

[1] Ilyas et al. (2018) Black-box Adversarial Attacks with Limited Queries and Information.

[2]Yu et al. (2020) CloudLeak: Large-Scale Deep Learning Models
Stealing Through Adversarial Examples

[3] Chen et al. (2020) HopSkipJumpAttack: A Query-Efficient
Decision-Based Attack

[7] Andriushchenko et al. (2020) Square Attack: a query-efficient black-box
adversarial attack via random search

[8] Chen et al. (2023) Rethinking Model Ensemble in Transfer-based Adversarial Attacks

Reviewer comment (Reproducibility):
What were the default values of \(\beta, B^t, T\) and C?

Author response:
In our model, we conceptualize the server as an API, similar to how it's approached in previous works [1] [2] [3]. For instance, the Google Cloud Vision API allows a query budget of up to 1800 for a single sample. Our primary focus in this research is on the efficiency of the E-TS algorithm in **selecting the optimal corruption pattern**. The inner Adversarial Example Generation (AEG) problem aligns more closely with traditional adversarial example generation techniques.

To address the concern about the large query budget, first, we conduct a targeted attack with the target label set to 6, under the setting that a query budget of 500, perturbation budget of 0.3, and a corruption constraint of 2, the E-TS best arm results indicate an ASR of 51\%, which indicates even in small query budget, we still can successfully attack half of the samples. Besides, adversaries could duplicate multiple noisy versions of the samples before inference, effectively increasing their query budget more stealthily. Furthermore, recent advancements in query-efficient methods [3] [7] and transfer-based approaches [8] offer alternatives to the AEG problem that do not compromise the efficacy of the E-TS solution.

In sum, we can leverage methods, like **selecting suitable target labels and perturbation budgets, duplicating samples in advance, and using efficient strategies** to mitigate the concern of a limited query budget. We also conduct an analysis of necessary minimum number of queries in **Appendix D**.
Regarding the default parameter settings used in our ablation study, they were as follows: $\beta = 0.3, B^t = 128$, and $C = 2$. We varied test parameters while maintaining the stability of other settings. The comprehensive parameter settings are now detailed in the revised draft and **Table 1 of Appendix C.1**.

[1] Ilyas et al. (2018) Black-box Adversarial Attacks with Limited Queries and Information.

[2]Yu et al. (2020) CloudLeak: Large-Scale Deep Learning Models
Stealing Through Adversarial Examples

[3] Chen et al. (2020) HopSkipJumpAttack: A Query-Efficient
Decision-Based Attack

[7] Andriushchenko et al. (2020) Square Attack: a query-efficient black-box
adversarial attack via random search

[8] Chen et al. (2023) Rethinking Model Ensemble in Transfer-based Adversarial Attacks

Reviewer comment (Experiments):
In Figure 1 what do the curves ‘Client 4 5’ etc. mean? Why are different client pairs shown in different plots? What is a test epoch and what does 25 attack rounds/test epoch mean? Does this mean that \(B^t\) is fixed at 25? And that T was 30 (judging from the figures)? There is little difference between TS and E-TS after about 5 epochs. What is the corresponding complexity comparison? The log(T) to O(1) improvement does not seem to show up here.

Author response:
We apologize for the confusion. The curve like 'Client 4,5' represents the attack performance of a fixed corruption pattern that corrupts client 4 and client 5 consistently in the inference. Similarly, in each sub-figure of Figure 1, curves like 'Client a,b,c' represent the performance of baseline method ''Fixed corruption pattern" mentioned in the Setup part of section 7. Specifically, we choose the best corruption pattern and randomly select a suboptimal corruption pattern for comparison, where the best corruption pattern differs in different attacks and datasets.

The '25 attack rounds/test epoch' means that we report the ASR of 25 attack rounds and regard these test rounds as an epoch for plotting a smooth figure. For the concern of the relations of $B^t$, attack round, epoch, and $T$, each attack round contains $B^t$ samples, each epoch is composed of $n_a$ attack rounds, e.g., 25 attack rounds in FashionMNIST, such that contains $n_aB^t$ test samples if $B^t$ is same for all $t$, and $T$ actually equals to $30\times n_a$. Note that the adversary can adjust the corruption pattern after each attack round. 

For the concern of complexity comparison, first, the attack results in Figure 1 consistently show an obvious performance gap between E-TS and plain TS. Specifically, E-TS needs about 5 epochs to find the best corruption pattern, while plain TS needs at least 10 to 25 epochs to find the best corruption pattern, see the blue line of E-TS and the orange line of plain TS. Second, we prove the expected pulling times of those non-competitive arms bound in $O(1)$ compared to $O(\log T)$ in plain TS. It is shown in the fast convergence of E-TS, that discards the exploration of those non-competitive arms thus exploring only the competitive arms and finding the best arm more quickly. For example, we have arms 1,2,3, with the best arm 1, and competitive arms 1 and 2. Using E-TS, we just need to play 1 and 2 multiple times to estimate an accurate prior such that recognizes the best arm 1. However, in plain TS, we also need to play arm 3 multiple times to get more reward information thus wasting the opportunity to use arm 1.

Reviewer comment (Evaluation):
Figure 2: Fashion MNIST has 7 clients; so even with all 7 clients corrupted, manifold defense does pretty well. This is a surprising and unexpected result.

Author response:
The advantage of manifold projection, compared with other defense approaches, can be attributed to its ability to learn the manifold of a class of inputs and recover the manifold to a clean input. While Randomized Smoothing and Dropout can reduce the effect caused by the adversarial part, they cannot completely eliminate the adversarial input and transform it into clean input. In contrast, manifold projection uses an autoencoder to clean the adversarial perturbations before they are fed into the server's model. Our work primarily focuses on selecting the most vulnerable clients in an adaptive attack setting, and we have not specifically designed our approach to overcome the manifold projection defense. Future research may explore methods for moving the input outside the manifold to bypass the manifold projection defense effectively.

Reviewer comment (Experiments):
Figures 2, 3 (and parts of 1): Figures showing performance results on Credit Score or the Caltech dataset would have been more meaningful. More specifically, recognition of MNIST / Fashion MNIST / CIFAR does not require all pieces of the full image; neither does sentiment analysis of a review; further, as noted earlier, such segmentation is unrealistic in any case.

Author response:
Thanks for the suggestion. See the response to Q6.

Reviewer comment (Theory):
Lemma 1, Lemma 2, and Theorem 1 appear to have an implicit large N assumption. The lower bound on the probability is negative for N < 55, and exceeds 0.5 at about N=150.

Author response:
The same concern was raised by reviewer HDL1. Having noticed the previous loose bound, we **tighten** the lower bound and update the probability from $\frac{1}{2}\left(1 - \frac{1}{N^2} e^{4\sqrt{\log(N)}}\right)\left(1-e^{-\frac{N}{\sqrt{4\pi N\log(N)}}}\right)$ to $ \left(1-\frac{\delta}{1+ \delta ^2}\phi(\delta)\right) \left(1 - e^{-2(\log(N)-1 )^2}\right)\left(1-e^{-\frac{N}{\sqrt{4\pi N\log(N)}}}\right)= f(\delta)p(N)$ for $\delta > 0, \phi(\delta) = \frac{e^{-\delta^2/2}}{\sqrt{2\pi}}$, where the minimum of $f(\delta)$ is 0.852 and the minimum of $p(N)$ is 0 (increases with $N$). Due to space limitations and complex expressions, we include the detailed proof and figure of the function in the revised Appendix (see **Appendix E and Figure 10**). 

In summary, the new bounds provide a positive lower bound for probability and the bound increases with $N$. We draw the function of $ p(N) = \left(1 - e^{-2(\log(N)-1 )^2}\right)\left(1-e^{-\frac{N}{\sqrt{4\pi N\log(N)}}}\right)$ and $ f(\delta) = 1-\frac{\delta}{1+ \delta^2}\phi(\delta) $ and discussed their behaviors in **Appendix E**.

Reviewer comment (Novelty):
How does the proposed E-TS algorithm work compared to sequential arm elimination proposed in  
S. Shahrampour, M. Noshad and V. Tarokh. On Sequential Elimination Algorithms for Best-Arm Identification in Multi-Armed Bandits. IEEE Transactions on Signal Processing, vol. 65, no. 16, pp. 4281-4292, 15 Aug.15, 2017.

Author response:
The referred nonlinear sequential elimination, which we call **NSE** for short, gives a solution to the MAB problem purely by exploring arms. NES is based on the concept of sequentially eliminating arms in an MAB setup until a single best arm is identified. NSE, distinct from traditional sequential elimination methods that linearly allocate exploration resources, employs a nonlinear function relative to the remaining arms for budget allocation in each round. In contrast to E-TS and TS, which focus on minimizing regret, NSE aims to reduce the misidentification probability within a limited exploration budget. 

We experimentally compare the performance of the proposed E-TS, plain TS, and NSE. To facilitate a fair comparison, we set different exploration budgets (denoted as $T$) and parameter $p$ for NSE. In this setup, NSE explores each arm's reward for $T$ rounds and subsequently uses the best arm it identifies post-$T$ rounds. We report all experimental results **in Figure 12 and 13 in Appendix G**.

Overall, plain TS has the worst performance. E-TS maintains **stable and faster convergence**, or equivalently, less regret. In contrast, NSE's performance, though sometimes comparable, is **less consistent and requires careful tuning** of hyperparameters $T$ and $p$.

Reviewer comment (Theory):
Is a factor 2 missing in the equation for Fact 1?

Author response:
Thanks for raising this issue and it is indeed a typo.
We have added this factor 2 in the revised draft. We note that with this modification, the result in the statement of Lemma 1-5 and Theorem 1 still holds (please refer to proofs for more details), since in all proofs, we just use the **one side of tail bound**, i.e., 
$
\operatorname{Pr}\left(\frac{\sum_{i=1}^nX_i}{n} - \mathbb{E}(X_i)\geq \delta\right) \leq \exp{\left(\frac{-2n\delta^2}{(b-a)^2}\right)}
$ 
and 

$
\operatorname{Pr}\left(\frac{\sum_{i=1}^nX_i}{n} - \mathbb{E}(X_i)\leq -\delta\right) \leq \exp{\left(\frac{-2n\delta^2}{(b-a)^2}\right)}, \forall \delta >0.
$

Reviewer comment (Theory):
The regret bound is a probability bound which only holds with some probabilities. However, the probabilities are not in the form like $1-p$. The probabilities depend on N and are less than $\frac{1}{2}$ for any N. If N is smaller than 10, the probabilities are further smaller than 0.07. I doubt whether this regret bound really provide enough information about the effectiveness of the proposed algorithm.

Author response:
Thank you for highlighting the issue with our probability bound. Upon re-evaluation, we recognized that the initial lower bound was too loose. We have since **tightened** this bound in the revised draft and appendix, (**see Section 6 and Appendix B, E**). 
 
 Specifically, we previously derive the bound $ \frac{1}{2}p $, where $ p = \left(1 - \frac{1}{N^2} e^{4\sqrt{\log(N)}}\right)\left(1-e^{-\frac{N}{\sqrt{4\pi N\log(N)}}}\right) $. The probability $\frac{1}{2}$ is from the probability $ \operatorname{Pr} (\theta_{k'}(\tau_1) \leq \mu_1)p \geq \frac{1}{2}p$. Here, $ \theta_{k'}(\tau_1) \leq \mu_1 $ indicates that the sample from a suboptimal arm is less than the mean of the best arm. Assuming a Gaussian distribution for the suboptimal arm with mean $ \mu_{k'} $ and variance  $\sigma_{k'}^2 $, and given $ \mu_{k'} \leq \mu_1 $, we derive the lower bound of the probability $ \operatorname{Pr} (\theta_{k'}(\tau_1) \leq \mu_1) \geq \frac{1}{2} $.

To tighten this bound further, let $ x = \theta_{k'}(\tau_1) - \mu_1 $ and normalize $ x $ as $ \tilde{x} \sim \mathcal{N}(0,1)$:

$
 \operatorname{Pr} (\theta_{k'}(\tau_1) \leq \mu_1) = \operatorname{Pr} (x \leq 0) = \operatorname{Pr}(\tilde{x} \leq \frac{\mu_1 - \mu_{k'}}{\sigma_{k'}})$
 $
 =1 - Q\left(\frac{\mu_1 - \mu_{k'}}{\sigma_{k'}}\right) \geq 1-\frac{\delta}{1+ \delta^2}\phi(\delta),
$

where $ Q(\cdot) $ is the Q function as the tail probability of a standard normal distribution~[1], $ \delta = \frac{\mu_1 - \mu_{k'}}{\sigma_{k'}} $, and $ \phi(\cdot) $ is the density function of the standard normal distribution.  
With this tighter bound, the lower bound improves from $ 0.5p $ to $ (1-\frac{\delta}{1+ \delta^2}\phi(\delta))p $, which is at least $ 0.852p $, (see plot of the function **in Figure 10 (a) of Appendix E**).

Regarding $ p $, we initially used a looser bound, leading to negative values in some cases. We have now adopted a tighter bound from [2], defined as $ p = \left(1 - e^{-2(\log(N)-1 )^2}\right)\left(1-e^{-\frac{N}{\sqrt{4\pi N\log(N)}}}\right) $, which minimum is 0 and increase with greater $N$. Due to space limitations and complex expressions, we include the detailed proof and figure of the function in the revised appendix, (see **Appendix E Figure 10 (b)**).

In summary, the new bounds provide a **positive** lower bound for probability and the probability **increases** with greater $N$. We draw the function of $ p(N) = \left(1 - e^{-2(\log(N)-1 )^2}\right)\left(1-e^{-\frac{N}{\sqrt{4\pi N\log(N)}}}\right)$ and $ f(\delta) = 1-\frac{\delta}{1+ \delta^2}\phi(\delta) $ and discussed its trend in the Appendix E.

Typically, $ N $ is large in practical scenarios due to its combinatorial nature, even for relatively small number of of clients, and E-TS is designed to manage this extensive exploration space. For instance, the number of arms $ N $ is at least 120, implying $ p(N) \geq 0.756 $. For smaller $N $ values, such as $ N = 21 $ with $ p = 0.52 $ in the FashionMNIST attack (Figure 1 (a)), the lower probability bound may seem modest, yet the empirical performance demonstrates rapid convergence compared to plain TS. Overall, our E-TS algorithm consistently shows faster convergence than TS in practice, and we can theoretically guarantee a high probability bound for large $ N $.

[1] Wikipedia Q-function https://en.wikipedia.org/wiki/Q-function

[2] AGRAWAL et al. (2017) Near-Optimal Regret Bounds for Thompson Sampling

Reviewer comment (Theory):
In Lemma 2 and Theorem 1, the regret guarantee requires that the selected warm-up rounds $t_0$ satisfies $\sqrt{\frac{t_0}{8N log(t_0)}} \ge \frac{1}{\Delta}$. As the adversary is a black-box adversary, $\Delta$ is unknown to the adversary, and there is no prior information about the choice of the $t_0$. The proper choice of $t_0$ is still a problem.

Author response:
Regrading your concern, we have added related discussion in **Appendix C.4**. 

Our theoretical framework presumes a specific $t_0$ value. In practical scenarios, setting $t_0$ to be at least larger than the number of arms $N$ is usually sufficient. Our experiments (**Figure 7 (a) and (b) in Appendix C.4**) indicate that a range between 
$2N $ and $5N$ is optimal. This range ensures that each arm is sampled at least twice using TS, providing the adversary with a more accurate initial understanding of each arm's prior distribution. This preliminary knowledge is crucial for the E-TS to effectively identify an empirical competitive set of arms. If $t_0$ is too small, there's a risk of E-TS prematurely converging on a suboptimal arm due to insufficient initial data, potentially omitting the optimal arm from the empirical competitive set. 

The optimal range of $t_0$ can be further validated in Figure 3 (b) in Section 7.2, where the number of arms $N =21$ and the optimal range of $t_0$ should be $[42, 105]$. From figure 3(b), we can observe that if we set $t_0 = 20$, E-TS converges to a suboptimal arm, and when $t_0 = 80$ and $t_0 = 200$, E-TS can converge to the optimal arm. Specifically, E-TS with $t_0 = 80$ has faster convergence than the one with $t_0 = 200$ because, within 80 rounds, E-TS has enough information of each arm to construct the empirical competitive set, thus discarding the non-competitive arm.


In sum, the optimal range of $t_0$ is $[2N, 5N]$ in practice, aiding in the accurate estimation of arm distributions and enhancing the overall effectiveness of E-TS.

Reviewer comment (Theory):
See in Weakness 1. Can authors explain the choice of the probabilities and the bounds?

Author response:
See response 1 and Appendix B and E.

Reviewer comment (Theory):
The performance of the proposed algorithm is related to the choice of $t_0$. The requirement of the prior information of $t_0$ violates the black-box setting. Can authors provide any insight into the proper choice of $t_0$ in the black-box setting?

Author response:
Please refer to response 2.

Reviewer comment (Theory):
In my opinion, the corruption pattern selection (CPS) problem is more like the setting of combinatorial multi-armed bandits (CMAB) with bandit feedback than MAB. Why did the author choose the model of MAB? Are there some benefits of MAB?

Author response:
We initially considered CMAB for the corruption pattern selection issue but ultimately chose MAB due to several reasons:

1. **Clear Definition of Rewards:** In a CMAB framework, defining the reward for each base arm is challenging, especially under varying corruption constraints. For instance, using the average ASR (Attack Success Rate) of including this arm to represent a base arm's reward can result in inconsistencies across different constraints. MAB, on the other hand, allows for a straightforward definition of each arm's reward as the ASR when that particular arm (corruption pattern) is used.

2. **Simplified Assumptions:** CMAB typically assumes the rewards of base arms to be independent and identically distributed (i.i.d.), which does not hold in our context due to the influence of corruption constraints. Moreover, common CMAB models [3] [4] assume that the super arm's reward is a simple aggregate (average or sum) of the base arms' rewards, which does not align with our attack model. Although some CMAB variants [5] [6] consider nonlinear relationships between super and base arms, they still rely on observable rewards for base arms or presume i.i.d. distributions, which are not applicable in our case.

3. **Comparative Performance and Regret Bound:** The regret bound in most CMAB works is $\mathcal{O}(\log(T))$ [3] [4] [5] [6], similar to MAB solutions. However, our MAB approach improves the regret of non-competitive arms from $\mathcal{O}(\log(T))$ to 
$\mathcal{O}(\log(1))$, showing a significant advantage. Additionally, our initial trials with a standard CMAB algorithm [3] yielded poorer results compared to a plain Thompson Sampling (TS) approach in MAB.

Therefore, considering these aspects, MAB was chosen for its clearer reward structure, less restrictive assumptions, and superior performance in our specific scenario.

 

[3] Wang et al. (2018) Thompson Sampling for Combinatorial Semi-Bandits

[4] Combe et al. (2015) Combinatorial Bandits Revisited

[5] Chen et al. (2016) Combinatorial Multi-Armed Bandit with General
Reward Functions

[6] Perrault et al. (2020) Statistical Efficiency of Thompson Sampling for
Combinatorial Semi-Bandits

Reviewer comment (Experiments):
For the VFL inference setting, it is unclear why the server broadcasts the probabilities vector back to all the clients for inference, rather than just the label, which would be more cost-saving. Broadcasting only the label would make it harder to design attacks. The reviewer asks if there is any reason that the client must receive the probability vector for inference.

Author response:
In line with established research on attacks in the API/remote server model [1] [2] [3] [4], queries to an API or remote ML model typically yield probability vectors, the prediction's probability, or just hard labels. Our threat model assumes that the server, functioning as a **remote model**, broadcasts the probability vector to all querying clients. In our attack design, as detailed in equation 3, section 5, these probability vectors are crucial for calculating the loss in the Inner problem AEG. Both targeted and untargeted attacks utilize the **cross-entropy** between the probability vector and the target label (positive loss for targeted attack and negative loss for untargeted attack). It's important to note that if the server were to provide only hard labels or the prediction's probability, our problem formulation and E-TS solution would remain **unaffected**. Alternative loss functions or simply computing the loss between the probability with the target label could be employed to address the inner problem in such scenarios.

Reviewer comment (Experiments):
The reviewer would like to see a discussion on how easy or difficult it is for the server to notice this attack. They believe a targeted attack could be easily identified by the server if the probability vector evolution for test sample queries converges to the same label. A naïve mitigation would be for the server to send back always the probability vector from the first embedding query to prevent training of $\eta_i^t$. They also do not see the benefit for the server to answer $n$ queries of the same test sample (as in line 10 of Algorithm 2), which aids the attacker in finding the best perturbation.

Author response:
The defense method you suggest is indeed practical and represents a potential area for future research. In response to such a defense, an adversary could set different target labels for each sample, preventing the server from detecting label convergence in a batch of data (note that our targeted attack does not require consistent target labels for different test samples). Additionally, even if the adversary uses the same target labels, the initial query results can reveal valuable information about the perturbation direction.
 This information is useful before the server recognizes and counters the attack, e.g., we conduct the targeted attack in FashionMNIST with target label 6, query budget 500, and $\beta = 0.3$. It can achieve over 50\% ASR with an average 203 queries in two corrupted client setting, which is far lower than the 500 query budget and indicates that initial direction has a huge effect on the attack.

Regarding the query capacity, our model aligns with previous works on remote server attacks [2] [3] [4]. For instance, Google's Cloud Vision API currently **permits up to 1,800 requests** for a single sample. Our threat model sets specific query ($Q$) and perturbation ($\beta$) budgets for the server, implying that queries remain safe as long as they stay within these limits. If the query budget is limited, the adversary can generate multiple noisy copies of the test samples and set them with different sample IDs in advance to increase the query times stealthily without exceeding the budget.

In summary, straightforward server strategies like adjusting the query budget or monitoring for label convergence can be circumvented. Attackers can use tactics such as **duplicating samples, varying target labels, or employing initial gradient directions** for attack. These methods help in evading detection and effectively conducting attacks.

[1]Yu et al. (2020) CloudLeak: Large-Scale Deep Learning Models
Stealing Through Adversarial Examples

[2] Chen et al. (2020) HopSkipJumpAttack: A Query-Efficient
Decision-Based Attack

[3] Chen et al. (2017) ZOO: Zeroth Order Optimization Based Black-box Attacks to
Deep Neural Networks without Training Substitute Models

[4] Ilyas et al. (2018) Black-box Adversarial Attacks with Limited Queries and Information.

Reviewer comment (Experiments):
From intuition, an untargeted attack should be easier than a targeted one. If a targeted attack succeeds with ASR $r$ on label $x$, the untargeted one should have at least $r$ on label $y \neq x$. More precisely, letting $r(x)$ be the targeted ASR on label $x$, the untargeted one on label $y$ should be better than $\max\{r(x) \mid \forall x \neq y\}$. The experiments on FashionMNIST confirm this, but CIFAR-10 shows the opposite. The reviewer asks for an explanation of this observation and clarification on what untargeted labels were used in the experiments.

Author response:
Thank you for your insights. In our experiments, the untargeted attacks were designed to deviate from the true labels of the test set. Specifically, if the true label is $y$, the goal of the attack was to ensure the predicted label $\hat{y}$ is not equal to $y$. For targeted attacks on CIFAR-10, we consistently chose the target label as 7. This approach results in differing numbers of attacked samples between targeted and untargeted attacks.

The observed inferior performance in the untargeted attack on CIFAR-10 is primarily due to a lower perturbation budget $\beta = 0.05$ compared to the budget $\beta = 0.08$ used in the targeted attack. To address this, we have re-implemented the untargeted attack with the same perturbation budget of $\beta = 0.08$, and the results have been updated in Figure 6 of **Appendix C.3**. It shows that the untargeted attack has **higher ASR** when setting the same perturbation budget and corruption constraint.

Reviewer comment (Evaluation):
In Figure 3c, when the query budget decreases to 500 for FashionMNIST, the targeted attack accuracy is almost 10%, equivalent to the ASR of the no-attack case if the sample is randomly split in the batch. A similar observation is in Figure 5c with only one corrupted channel. The reviewer asks whether this means that when the query budget or number of corrupted channels is limited, the proposed method has little or no advantage over doing nothing for targeted attack, and what factors (dataset complexity, model, scale of the system) impact the minimum necessary query budget and number of corrupted channels.

Author response:
The experimental results in Figure 3 (a) indeed show that with only one corrupted channel and $Q = 2000$, the ASR can exceed 30\%. However, the lower ASR observed when the query budget is under 500 is primarily attributed to the challenges of a targeted attack, which necessitates classifying all examples into a single target, such as label 7. Achieving this projection to label 7 requires extensive gradient estimation, whereas an untargeted attack can nearly reach a 50\% ASR within the same query budget (see Figure 5 (b) in Appendix C.2).

Furthermore, **the ASR of a targeted attack is influenced by the choice of the target**. For instance, when conducting a targeted attack with the target label set to 6, under the same conditions (a query budget of 500 and a corruption budget of 2), the E-TS best arm results indicate an ASR of 51\%. Remarkably, even with the same query budget but only one corrupted channel, the ASR can reach 37\% with an average of 203 queries. 

In conclusion, our proposed attack method can be effective even with a single corrupted channel and limited query budget. However, the attack's performance is significantly impacted by factors such as the type of attack (targeted or untargeted) and the specific target label chosen.

To explore how the necessary number of queries and corrupted channels vary across different models,
datasets, and systems, we conducted experiments using Credit and Real-sim datasets. We specifically
analyzed the average number of queries required to attain a 50\% ASR under various levels of client
corruption (corruption constraint $C$). The detailed results and discussion are in **Appendix D**.

From the results in **Figure 9 (a) and (b) in Appendix D**, to attain a target ASR with the same $C$, simpler datasets within the same task require
fewer queries. Systems with a higher number of clients necessitate more queries. However, the
influence of the model’s complexity on the number of queries is ambiguous and warrants further
investigation.

Reviewer comment (Presentation):
The settings for each figure are not well indicated, especially for the ablation study. For example, in Figure 3, when one parameter is varied, the default values for the other parameters are not specified.

Author response:
We apologize for any confusion caused by the initial presentation. In the revised draft, we have provided detailed specifications for each figure, particularly concerning the ablation study settings. For the ablation study, the default corruption constraint $C$ is set to 2, the number of warm-up rounds $t_0$ is 80, the query budget $Q$ is 2000, and the perturbation budget $\beta$ is 0.3. These details have been thoroughly updated in both the draft and the appendix.