Paper: Adversarial Machine Learning in Latent Representations of Neural Networks (ICLR-2024)

Reviewer comment (Theory):
On the claim that "assuming the same level of information distortion, latent features are always more robust than input representations", does this hold even if the latent embedding dimension is larger than the input dimension? If yes, why it can be more robust? Moreover, if the latent embedding dimension is lower than the input dimension, then it has been proven in the ICML 2019 paper "First-order Adversarial Vulnerability of Neural Networks and Input Dimension" that the minimal adversarial perturbation scales inversely with the data dimension. So if we treat the latent representations as the new "data" and they have lower dimensions than the raw inputs, the new insights are not clear.

Author response:
## **Regarding Weakness 1 & Question 1** 

> On the claim that "assuming the same level of information distortion, latent features are always more robust than input representations", does this hold even if the latent embedding dimension is larger than the input dimension?

We thank the reviewer for this comment. To this end, we remark that Theoretical Finding #2 indeed holds when the latent embedding dimension is larger than the input dimension. Our experiments shown in Figure 4 include the original VGG and ResNet, where the space is respectively $\times 5.33$, $\times 2.66$ larger.

To better prove our point, we are actively working on a new experiment with respect to the depth of latent space in the revision. We present some preliminary results in Table 1 and Table 2.

---

**Table 1: Adversarial robustness of ResNet152 as a function of depth with perturbation budget $\epsilon = 0.003$**

| Attack | Input | FeatureMap0 | FeatureMap1 | FeatureMap2 | FeatureMap3 | FeatureMap4 |
| :-------: | :-------: | :------: | :-------: | :-------: | :-------: | :--------: |
| dimension | x1 | x5.33 | x5.33 | x2.66 | x1.33 | x0.66 |
| FGSM | 71.9% | 53.6% | 53.1% | 35.8% | 34.9% | 1.6% |
| PGD | 88.1% | 65.5% | 66.3% | 43.1% | 43.2% | 1.6% |

---

**Table 2: Adversarial robustness of VGG16 as a function of depth with perturbation budget $\epsilon = 0.003$**

| | Input | FeatureMap0 | FeatureMap1 | FeatureMap2 | FeatureMap3 | FeatureMap4 |
| :-------: | :-------: | :------: | :-------: | :-------: | :-------: | :-------: |
| dimension | x1 | x5.33 | x2.66 | x1.33 | x0.66 | x0.33 |
| FGSM | 94.5% | 41.2% | 28.7% | 20.5% | 14.0% | 6.4% |
| PGD | 98.8% | 50.3% | 33.2% | 22.3% | 14.5% | 6.5% |

---

The experiment is based on a smaller model VGG16 and a deeper model ResNet152. Same level of $l_\infty$ norm distortions are introduced to all feature maps before max pooling layers. As shown in Table 1 and Table 2, all the latent feature maps are more robust than the input regardless of dimensionality and depth. Furthermore, ASR reduces with respect to the depth of latent space, which is in line with our Theoretical Finding #2.


>  If yes, why it can be more robust?

This finding holds because it is derived directly from the chain rule of mutual information which does not depend on cardinality $|\mathcal{T}|$. 

> it has been proven in the ICML 2019 paper "First-order Adversarial Vulnerability of Neural Networks and Input Dimension" that the minimal adversarial perturbation scales inversely with the data dimension.

We thank the reviewer for bringing [1] to our attention. We want to address the key difference between the work [1] and our Theoretical Finding #1. In our Theoretical Finding #1, the robustness is jointly determined by its upper bound $I^*(Y;T)$ and $\mathcal{O}( |\mathcal{T}||\mathcal{Y}| / \sqrt{n})$. Although $\mathcal{O}( |\mathcal{T}||\mathcal{Y}| / \sqrt{n})$ indicates that a larger cardinality will result in less robustness, which is in line with [1], $T$ is a DNN-specific space depending on the DNN  architecture design. In stark opposition, Theoretical Finding #1 addresses the variance-bias tradeoff in the distributed DNN design, which is also supported by our experiments as shown in Figure 6. Moreover, it also explains why latent space is more robust even if it has larger cardinality, since the latent robustness depends both on the cardinality and $I^*(Y;T)$.

Reviewer comment (Theory):
The assumption on the same level of information distortion seems very strong and lacks justification. It's also not clear what is the threat model (what the attacker can do) that leads to a latent representation $T_{adv}$.

Author response:
## **Regarding Weakness 2 & Question 2** 

> The assumption on the same level of information distortion seems very strong and lacks justification

We thank the reviewer for this insightful comment. We remark that such constraints on information distortion are necessary to formally analyze robustness of distributed DNNs and provide a fair comparison with attacks in the input space. Moreover, although there is much work focused on unconstrained attacks –  such as [2, 3, 4] —,  the constrained distortions are still a popular way to evaluate robustness [5, 6]. Since this is the first work to investigate this topic, we aim at formally analyzing the robustness with information constraints. However, we also point out that in real-world scenarios the attacks in the latent space can have a different level of information than the input. We will clarify this crucial aspect in the final version of the paper.
 
> It's also not clear what is the threat model (what the attacker can do) that leads to a latent representation $T_{adv}$

In a distributed DNN scenario, latent representations are sent to the local DNN that can be exposed to attackers. We refer the reviewer to Figure 1 for a visual depiction of the attack. We assume that attackers can directly add perturbations to latent representations in the same manner as attacks performed in the input space. Thus, $T_{adv}$ denotes the adversarial latent representation generated by attackers. For a detailed attack model formulation, we kindly refer the reviewer to Section 4.1.

Reviewer comment (Experiments):
The evaluated attacks are naive input perturbation attacks, and no adaptive attacks that take into account modifying the latent representations were studied. It should be easy to add a regularizer to attack objectives to encourage finding adversarial examples that share very similar (or very different) latent representations as the original data, and therefore the claim on improved robustness may not hold.

Author response:
## **Regarding Weakness 3 & Question 3** 

> The evaluated attacks are naive input perturbation attacks, and no adaptive attacks that take into account modifying the latent representations were studied.

We sincerely thank the reviewer for pointing out adaptive attacks. For a fair comparison, in this paper we consider attackers in latent space as acting in the same manner as those attacking in input space. For this reason, we applied the attacks already present  in literature to the latent representations. Any attack that will lead to different processes between input and latent space are considered out of the scope of our paper since they deserve a separate and dedicated investigation.

We remark that adaptive attacks are usually proposed to correctly evaluate novel defense strategies that can tackle attackers by shattered gradient or randomness [7, 8]. Since our focus is to analyze the robust performance of distributed DNNs, instead of providing defenses against known attacks, we believe this new direction deserves a separate investigation. 

We also point out that much better attacks and defenses specifically tailored for distributed DNNs can be formulated by achieving a fundamental understanding of robustness in distributed DNNs. Given that we are the first to investigate the robustness in distributed DNNs, we would like to limit our scope to well established attack algorithms for understanding the distributed DNNs. We are actively working on novel attack and defense as part of ongoing efforts.

Reviewer comment (Theory):
W.r.t. to W1, is there any implicit assumption that the latent embedding dimension is smaller than the input dimension? If so, the improved robustness is a direct consequence of the ICML 2019 result.

Author response:
## **Regarding Weakness 1 & Question 1** 

> On the claim that "assuming the same level of information distortion, latent features are always more robust than input representations", does this hold even if the latent embedding dimension is larger than the input dimension?

We thank the reviewer for this comment. To this end, we remark that Theoretical Finding #2 indeed holds when the latent embedding dimension is larger than the input dimension. Our experiments shown in Figure 4 include the original VGG and ResNet, where the space is respectively $\times 5.33$, $\times 2.66$ larger.

To better prove our point, we are actively working on a new experiment with respect to the depth of latent space in the revision. We present some preliminary results in Table 1 and Table 2.

---

**Table 1: Adversarial robustness of ResNet152 as a function of depth with perturbation budget $\epsilon = 0.003$**

| Attack | Input | FeatureMap0 | FeatureMap1 | FeatureMap2 | FeatureMap3 | FeatureMap4 |
| :-------: | :-------: | :------: | :-------: | :-------: | :-------: | :--------: |
| dimension | x1 | x5.33 | x5.33 | x2.66 | x1.33 | x0.66 |
| FGSM | 71.9% | 53.6% | 53.1% | 35.8% | 34.9% | 1.6% |
| PGD | 88.1% | 65.5% | 66.3% | 43.1% | 43.2% | 1.6% |

---

**Table 2: Adversarial robustness of VGG16 as a function of depth with perturbation budget $\epsilon = 0.003$**

| | Input | FeatureMap0 | FeatureMap1 | FeatureMap2 | FeatureMap3 | FeatureMap4 |
| :-------: | :-------: | :------: | :-------: | :-------: | :-------: | :-------: |
| dimension | x1 | x5.33 | x2.66 | x1.33 | x0.66 | x0.33 |
| FGSM | 94.5% | 41.2% | 28.7% | 20.5% | 14.0% | 6.4% |
| PGD | 98.8% | 50.3% | 33.2% | 22.3% | 14.5% | 6.5% |

---

The experiment is based on a smaller model VGG16 and a deeper model ResNet152. Same level of $l_\infty$ norm distortions are introduced to all feature maps before max pooling layers. As shown in Table 1 and Table 2, all the latent feature maps are more robust than the input regardless of dimensionality and depth. Furthermore, ASR reduces with respect to the depth of latent space, which is in line with our Theoretical Finding #2.


>  If yes, why it can be more robust?

This finding holds because it is derived directly from the chain rule of mutual information which does not depend on cardinality $|\mathcal{T}|$. 

> it has been proven in the ICML 2019 paper "First-order Adversarial Vulnerability of Neural Networks and Input Dimension" that the minimal adversarial perturbation scales inversely with the data dimension.

We thank the reviewer for bringing [1] to our attention. We want to address the key difference between the work [1] and our Theoretical Finding #1. In our Theoretical Finding #1, the robustness is jointly determined by its upper bound $I^*(Y;T)$ and $\mathcal{O}( |\mathcal{T}||\mathcal{Y}| / \sqrt{n})$. Although $\mathcal{O}( |\mathcal{T}||\mathcal{Y}| / \sqrt{n})$ indicates that a larger cardinality will result in less robustness, which is in line with [1], $T$ is a DNN-specific space depending on the DNN  architecture design. In stark opposition, Theoretical Finding #1 addresses the variance-bias tradeoff in the distributed DNN design, which is also supported by our experiments as shown in Figure 6. Moreover, it also explains why latent space is more robust even if it has larger cardinality, since the latent robustness depends both on the cardinality and $I^*(Y;T)$.

Reviewer comment (Theory):
How to justify the assumption of "the same level of information distortion"? Does $T_{adv}$ hold for any arbitrary threat model?

Author response:
## **Regarding Weakness 2 & Question 2** 

> The assumption on the same level of information distortion seems very strong and lacks justification

We thank the reviewer for this insightful comment. We remark that such constraints on information distortion are necessary to formally analyze robustness of distributed DNNs and provide a fair comparison with attacks in the input space. Moreover, although there is much work focused on unconstrained attacks –  such as [2, 3, 4] —,  the constrained distortions are still a popular way to evaluate robustness [5, 6]. Since this is the first work to investigate this topic, we aim at formally analyzing the robustness with information constraints. However, we also point out that in real-world scenarios the attacks in the latent space can have a different level of information than the input. We will clarify this crucial aspect in the final version of the paper.
 
> It's also not clear what is the threat model (what the attacker can do) that leads to a latent representation $T_{adv}$

In a distributed DNN scenario, latent representations are sent to the local DNN that can be exposed to attackers. We refer the reviewer to Figure 1 for a visual depiction of the attack. We assume that attackers can directly add perturbations to latent representations in the same manner as attacks performed in the input space. Thus, $T_{adv}$ denotes the adversarial latent representation generated by attackers. For a detailed attack model formulation, we kindly refer the reviewer to Section 4.1.

Reviewer comment (Evaluation):
Does the result of improved robustness still hold against adaptive attacks, where the attacker can have access to the latent representations?

Author response:
## **Regarding Weakness 3 & Question 3** 

> The evaluated attacks are naive input perturbation attacks, and no adaptive attacks that take into account modifying the latent representations were studied.

We sincerely thank the reviewer for pointing out adaptive attacks. For a fair comparison, in this paper we consider attackers in latent space as acting in the same manner as those attacking in input space. For this reason, we applied the attacks already present  in literature to the latent representations. Any attack that will lead to different processes between input and latent space are considered out of the scope of our paper since they deserve a separate and dedicated investigation.

We remark that adaptive attacks are usually proposed to correctly evaluate novel defense strategies that can tackle attackers by shattered gradient or randomness [7, 8]. Since our focus is to analyze the robust performance of distributed DNNs, instead of providing defenses against known attacks, we believe this new direction deserves a separate investigation. 

We also point out that much better attacks and defenses specifically tailored for distributed DNNs can be formulated by achieving a fundamental understanding of robustness in distributed DNNs. Given that we are the first to investigate the robustness in distributed DNNs, we would like to limit our scope to well established attack algorithms for understanding the distributed DNNs. We are actively working on novel attack and defense as part of ongoing efforts.

Reviewer comment (Experiments):
The paper does not take into consideration attacking the latent representations of a DNN that was adversarially trained. To the best of my knowledge, the adversarial training [1] has an impact over the latent representations of a neural network and under certain settings (a perturbation budget $\epsilon$ not too large) the network is robust to adversarial input perturbations. I would be interested in the robustness of adversarially trained networks, would it be possible to perform such an experiment? I believe the paper is not complete without this experiment and I would really appreciate if you could include this.

Author response:
## **Regarding Weakness 1 & Question 1** 

> The paper does not take into consideration attacking the latent representations of a DNN that was adversarially trained.

We thank the reviewer for pointing this out. We have developed a new experiment to evaluate the input and latent robustness of two adversarial training approaches presented in recent work [1] and [2] in Table 1. 

---

**Table 1: Robustness of ResNet50, DAT [1] and FastAT [2] with perturbation budget $\epsilon = 0.003$**

| | PGD | | N-Attack | | Triangle | | 
| :------- | :-------: | :------: | :-------: | :-------: | :-------: | :--------: |
| Model | Input | Latent |  Input | Latent | Input | Latent |
|ResNet50 | 92.9% | 62.9% | 61.1% | 45.5% | 85.8% | 13.0% |
| DAT | 14.6% | 1.0% | 54.5% | 0.0% | 53.0% | 1.0% |
| FastAT | 7.9% | 3.1% | 43.3% | 1.0% | 29.6% | 1.1% |

---

In Table 1, DAT and FastAT are pretrained ResNet50 with different adversarial training approaches. We evaluate the robustness in both input and latent space as we did with the original ResNet50. The results show that latent representations of adversarially trained DNNs are also more robust than input, which is in line with our theoretical findings.


> how do you think the adversarial training would change the success rate of these attacks?

We have evaluated this aspect from an experimental standpoint in the table above. From a theoretical perspective, while the adversarial training should be able to improve the robustness in both input and latent space to some extent, it will not affect our Theoretical Finding #2. Indeed, in our Theoretical Finding #1, the upper bound is defined by a term $\mathcal{O}( |\mathcal{X}||\mathcal{Y}| / \sqrt{n})$, where n is the number of samples. Adversarial training actually introduces more samples, resulting in a smaller $\mathcal{O}( |\mathcal{X}||\mathcal{Y}| / \sqrt{n})$ as well as a larger upper bound in robustness. We also point out that our Theoretical Finding #2 holds irrespective of the sample size n, therefore adversarial training will not impact this finding. We are going to include these explanations in the final version of the manuscript.

Reviewer comment (Novelty):
In the Conclusion section, this article claims that ``This paper has investigated adversarial attacks to latent representations of DNNs for the first time``. Through the lens of distributed DNNs, this work may be the first one, as it claims. However, I am concerned about whether the distributed DNNs are a necessary background as the motivation to investigate the problem.

Author response:
> However, I am concerned about whether the distributed DNNs are a necessary background as the motivation to investigate the problem. Since I think the local DNN and mobile DNN are very like the architecture of an autoencoder, and there are many works about the adversarial robustness of autoencoders.

We agree with the reviewer that distributed DNNs have an encoder-decoder structure similar to autoencoders, and we appreciate the reviewer pointing out adversarial autoencoders. However, there is a stark distinction between robust distributed DNNs and robust autoencoders. In robust autoencoder, adversarial training is used to enforce the encoder to learn adversarially robust latent representations – for example, for speaker de-identification [1] or outlier detection [2]. Such latent representations are not considered to be exposed to attackers thus the “robustness” in the robust autoencoder context is actually “semantically meaningful representations”. 

In our case, we consider attacks adding perturbations directly to the latent space. The scenario considered is extremely relevant to mobile scenarios since distributed DNNs have been proven to be extremely effective in reducing network load while preserving accuracy [3, 4, 5]. In this case, latent representations are exposed to the attacker, which can lead to performance degradation. To this end, the “robustness” in latent representation means “adversarially robust to attacks”, which is a remarkably different concept than robust autoencoders. 

To the best of the authors’ knowledge, there is no such prior work to investigate adversarial perturbation in latent representation. Thus, we would like to highlight the significance and timeliness of our work in the context of distributed DNN.

We appreciate the constructive feedback to help us clarify our novelty and contribution. We are actively working on a revision that includes the suggested work [1] and clarification.

---------------------------------

Reference:

[1] Espinoza-Cuadros, Fernando M., et al. "Speaker de-identification system using autoencoders and adversarial training." arXiv preprint arXiv:2011.04696 (2020).

[2] Salehi, Mohammadreza, et al. "Arae: Adversarially robust training of autoencoders improves novelty detection." Neural Networks 144 (2021): 726-736.

[3] Matsubara, Yoshitomo, et al. "Bottlefit: Learning compressed representations in deep neural networks for effective and efficient split computing." 2022 IEEE 23rd International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM). IEEE, 2022.

[4] Matsubara, Yoshitomo, et al. "Supervised compression for resource-constrained edge computing systems." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.

[5] Shao, Jiawei, and Jun Zhang. "Bottlenet++: An end-to-end approach for feature compression in device-edge co-inference systems." 2020 IEEE International Conference on Communications Workshops (ICC Workshops). IEEE, 2020.

Reviewer comment (Novelty):
I think the local DNN and mobile DNN are very like the architecture of an autoencoder, and there are many works about the adversarial robustness of autoencoders.

Author response:
> However, I am concerned about whether the distributed DNNs are a necessary background as the motivation to investigate the problem. Since I think the local DNN and mobile DNN are very like the architecture of an autoencoder, and there are many works about the adversarial robustness of autoencoders.

We agree with the reviewer that distributed DNNs have an encoder-decoder structure similar to autoencoders, and we appreciate the reviewer pointing out adversarial autoencoders. However, there is a stark distinction between robust distributed DNNs and robust autoencoders. In robust autoencoder, adversarial training is used to enforce the encoder to learn adversarially robust latent representations – for example, for speaker de-identification [1] or outlier detection [2]. Such latent representations are not considered to be exposed to attackers thus the “robustness” in the robust autoencoder context is actually “semantically meaningful representations”. 

In our case, we consider attacks adding perturbations directly to the latent space. The scenario considered is extremely relevant to mobile scenarios since distributed DNNs have been proven to be extremely effective in reducing network load while preserving accuracy [3, 4, 5]. In this case, latent representations are exposed to the attacker, which can lead to performance degradation. To this end, the “robustness” in latent representation means “adversarially robust to attacks”, which is a remarkably different concept than robust autoencoders. 

To the best of the authors’ knowledge, there is no such prior work to investigate adversarial perturbation in latent representation. Thus, we would like to highlight the significance and timeliness of our work in the context of distributed DNN.

We appreciate the constructive feedback to help us clarify our novelty and contribution. We are actively working on a revision that includes the suggested work [1] and clarification.

---------------------------------

Reference:

[1] Espinoza-Cuadros, Fernando M., et al. "Speaker de-identification system using autoencoders and adversarial training." arXiv preprint arXiv:2011.04696 (2020).

[2] Salehi, Mohammadreza, et al. "Arae: Adversarially robust training of autoencoders improves novelty detection." Neural Networks 144 (2021): 726-736.

[3] Matsubara, Yoshitomo, et al. "Bottlefit: Learning compressed representations in deep neural networks for effective and efficient split computing." 2022 IEEE 23rd International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM). IEEE, 2022.

[4] Matsubara, Yoshitomo, et al. "Supervised compression for resource-constrained edge computing systems." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.

[5] Shao, Jiawei, and Jun Zhang. "Bottlenet++: An end-to-end approach for feature compression in device-edge co-inference systems." 2020 IEEE International Conference on Communications Workshops (ICC Workshops). IEEE, 2020.

Reviewer comment (Theory):
If we compare the architecture between a distributed DNN and an autoencoder, I think the local DNN is very much like the encoder part, and the mobile DNN is very much like the decoder part. Can I compare them like this?

Author response:
> However, I am concerned about whether the distributed DNNs are a necessary background as the motivation to investigate the problem. Since I think the local DNN and mobile DNN are very like the architecture of an autoencoder, and there are many works about the adversarial robustness of autoencoders.

We agree with the reviewer that distributed DNNs have an encoder-decoder structure similar to autoencoders, and we appreciate the reviewer pointing out adversarial autoencoders. However, there is a stark distinction between robust distributed DNNs and robust autoencoders. In robust autoencoder, adversarial training is used to enforce the encoder to learn adversarially robust latent representations – for example, for speaker de-identification [1] or outlier detection [2]. Such latent representations are not considered to be exposed to attackers thus the “robustness” in the robust autoencoder context is actually “semantically meaningful representations”. 

In our case, we consider attacks adding perturbations directly to the latent space. The scenario considered is extremely relevant to mobile scenarios since distributed DNNs have been proven to be extremely effective in reducing network load while preserving accuracy [3, 4, 5]. In this case, latent representations are exposed to the attacker, which can lead to performance degradation. To this end, the “robustness” in latent representation means “adversarially robust to attacks”, which is a remarkably different concept than robust autoencoders. 

To the best of the authors’ knowledge, there is no such prior work to investigate adversarial perturbation in latent representation. Thus, we would like to highlight the significance and timeliness of our work in the context of distributed DNN.

We appreciate the constructive feedback to help us clarify our novelty and contribution. We are actively working on a revision that includes the suggested work [1] and clarification.

---------------------------------

Reference:

[1] Espinoza-Cuadros, Fernando M., et al. "Speaker de-identification system using autoencoders and adversarial training." arXiv preprint arXiv:2011.04696 (2020).

[2] Salehi, Mohammadreza, et al. "Arae: Adversarially robust training of autoencoders improves novelty detection." Neural Networks 144 (2021): 726-736.

[3] Matsubara, Yoshitomo, et al. "Bottlefit: Learning compressed representations in deep neural networks for effective and efficient split computing." 2022 IEEE 23rd International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM). IEEE, 2022.

[4] Matsubara, Yoshitomo, et al. "Supervised compression for resource-constrained edge computing systems." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.

[5] Shao, Jiawei, and Jun Zhang. "Bottlenet++: An end-to-end approach for feature compression in device-edge co-inference systems." 2020 IEEE International Conference on Communications Workshops (ICC Workshops). IEEE, 2020.

Reviewer comment (Novelty):
If yes, I think some works have studied the adversarial robustness of the latent features, e.g., [1].

Author response:
> However, I am concerned about whether the distributed DNNs are a necessary background as the motivation to investigate the problem. Since I think the local DNN and mobile DNN are very like the architecture of an autoencoder, and there are many works about the adversarial robustness of autoencoders.

We agree with the reviewer that distributed DNNs have an encoder-decoder structure similar to autoencoders, and we appreciate the reviewer pointing out adversarial autoencoders. However, there is a stark distinction between robust distributed DNNs and robust autoencoders. In robust autoencoder, adversarial training is used to enforce the encoder to learn adversarially robust latent representations – for example, for speaker de-identification [1] or outlier detection [2]. Such latent representations are not considered to be exposed to attackers thus the “robustness” in the robust autoencoder context is actually “semantically meaningful representations”. 

In our case, we consider attacks adding perturbations directly to the latent space. The scenario considered is extremely relevant to mobile scenarios since distributed DNNs have been proven to be extremely effective in reducing network load while preserving accuracy [3, 4, 5]. In this case, latent representations are exposed to the attacker, which can lead to performance degradation. To this end, the “robustness” in latent representation means “adversarially robust to attacks”, which is a remarkably different concept than robust autoencoders. 

To the best of the authors’ knowledge, there is no such prior work to investigate adversarial perturbation in latent representation. Thus, we would like to highlight the significance and timeliness of our work in the context of distributed DNN.

We appreciate the constructive feedback to help us clarify our novelty and contribution. We are actively working on a revision that includes the suggested work [1] and clarification.

---------------------------------

Reference:

[1] Espinoza-Cuadros, Fernando M., et al. "Speaker de-identification system using autoencoders and adversarial training." arXiv preprint arXiv:2011.04696 (2020).

[2] Salehi, Mohammadreza, et al. "Arae: Adversarially robust training of autoencoders improves novelty detection." Neural Networks 144 (2021): 726-736.

[3] Matsubara, Yoshitomo, et al. "Bottlefit: Learning compressed representations in deep neural networks for effective and efficient split computing." 2022 IEEE 23rd International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM). IEEE, 2022.

[4] Matsubara, Yoshitomo, et al. "Supervised compression for resource-constrained edge computing systems." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.

[5] Shao, Jiawei, and Jun Zhang. "Bottlenet++: An end-to-end approach for feature compression in device-edge co-inference systems." 2020 IEEE International Conference on Communications Workshops (ICC Workshops). IEEE, 2020.

Reviewer comment (Novelty):
Therefore, I am a bit curious about whether distributed DNNs are a necessary background as the motivation to investigate the adversarial robustness of the latent features.

Author response:
> However, I am concerned about whether the distributed DNNs are a necessary background as the motivation to investigate the problem. Since I think the local DNN and mobile DNN are very like the architecture of an autoencoder, and there are many works about the adversarial robustness of autoencoders.

We agree with the reviewer that distributed DNNs have an encoder-decoder structure similar to autoencoders, and we appreciate the reviewer pointing out adversarial autoencoders. However, there is a stark distinction between robust distributed DNNs and robust autoencoders. In robust autoencoder, adversarial training is used to enforce the encoder to learn adversarially robust latent representations – for example, for speaker de-identification [1] or outlier detection [2]. Such latent representations are not considered to be exposed to attackers thus the “robustness” in the robust autoencoder context is actually “semantically meaningful representations”. 

In our case, we consider attacks adding perturbations directly to the latent space. The scenario considered is extremely relevant to mobile scenarios since distributed DNNs have been proven to be extremely effective in reducing network load while preserving accuracy [3, 4, 5]. In this case, latent representations are exposed to the attacker, which can lead to performance degradation. To this end, the “robustness” in latent representation means “adversarially robust to attacks”, which is a remarkably different concept than robust autoencoders. 

To the best of the authors’ knowledge, there is no such prior work to investigate adversarial perturbation in latent representation. Thus, we would like to highlight the significance and timeliness of our work in the context of distributed DNN.

We appreciate the constructive feedback to help us clarify our novelty and contribution. We are actively working on a revision that includes the suggested work [1] and clarification.

---------------------------------

Reference:

[1] Espinoza-Cuadros, Fernando M., et al. "Speaker de-identification system using autoencoders and adversarial training." arXiv preprint arXiv:2011.04696 (2020).

[2] Salehi, Mohammadreza, et al. "Arae: Adversarially robust training of autoencoders improves novelty detection." Neural Networks 144 (2021): 726-736.

[3] Matsubara, Yoshitomo, et al. "Bottlefit: Learning compressed representations in deep neural networks for effective and efficient split computing." 2022 IEEE 23rd International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM). IEEE, 2022.

[4] Matsubara, Yoshitomo, et al. "Supervised compression for resource-constrained edge computing systems." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.

[5] Shao, Jiawei, and Jun Zhang. "Bottlenet++: An end-to-end approach for feature compression in device-edge co-inference systems." 2020 IEEE International Conference on Communications Workshops (ICC Workshops). IEEE, 2020.

Reviewer comment (Evaluation):
In this paper, the attacker only has access to the latent representation provided by the mobile DNN from a mobile phone. While the authors successfully demonstrate that attacks on these latent representations exhibit a lower Attack Success Rate (ASR) than those on raw images—given the same level of information distortion—the appropriateness of imposing an identical distortion level in this context needs more clarification. In discussions around the adversarial robustness of image input, constraints are placed on distortion levels to ensure modifications are imperceptible to the human eye, yet potent enough to deceive the classifier [1]. However, when considering distortions applied to latent features, the rationale for enforcing the same constraint level is less clear. Given that the transmission in this scenario occurs between a mobile device and a cloud computer, with no human observer in the loop, the attacker might as well nullify all latent features, potentially achieving an ASR close to 100%. Comparing the ASR between attacks on raw input images (which are observable by humans) and attacks on latent features (processed by a cloud computer) under an equivalent distortion level seems illogical.

Author response:
## **Regarding Weakness 1**

> the appropriateness of imposing an identical distortion level in this context needs more clarification

We thank the reviewer for this insightful comment. We point out that estimating the distortion is necessary to formally compare the robustness of the DNN against adversarial attacks in the latent space with respect to attacks in the input space. Specifically, to compare input space and latent space attacks, we assume an identical adversarial effort in both input and latent space through the $l_p$ norm constraint. We point out that the $l_p$ norm is routinely used as a metric to characterize such effort, as it is used in existing work, for example [1, 2, 3]. Moreover, we also point out that in real-world scenarios the attacks in the latent space can (and most likely, will) have a different level of information than attacks in the input space. We will clarify this crucial aspect in the final version of the paper.

> with no human observer in the loop, the attacker might as well nullify all latent features, potentially achieving an ASR close to 100%

We strongly agree with the reviewer that without a human observer in the loop, unconstrained perturbations in latent space can lead to ~100% ASR. However, we also believe latent representations intrinsically contain more semantic information than the input, which may result in easier outlier detection. For example, image data usually has redundant information such as background. Therefore, changing one pixel in the background might not be perceptible by humans. Conversely, since semantic representations capture only the sufficient statistics of the input, changing one element of them can result in invalid interpretations (take one-hot encoding as an extreme case example). As such, an outlier detection system which can estimate the perturbation magnitude in latent space can play a similar role as the human observer in input space. We will definitely delve deeper into this intriguing aspect in our future work as it deserves a separate and exhaustive investigation.

Reviewer comment (Novelty):
Although the authors posit that this study on the robustness of distributed DNNs in the face of adversarial actions is novel, I find the concept markedly similar to existing works on attacks targeting intermediate layers or latent features [2]. I would appreciate it if the authors could highlight the distinctions between their work and prior research, and attempt to apply the attack methods delineated in [2] where feasible.

Author response:
## **Regarding Weakness 2** 

> I would appreciate it if the authors could highlight the distinctions between their work and prior research

We appreciate the reviewer bringing [4] to our attention. We point out that paper [4] uses the latent representation as a help to craft the adversarial input. In stark opposition, our work focuses on the perturbation of the latent space itself. We are working on a revision to address the distinction between our work and suggested work [4].

> and attempt to apply the attack methods delineated in [4] where feasible.

We thank the reviewer for suggesting to compare our paper to [4], which we definitely plan to do in future work.

Reviewer comment (Evaluation):
The experimental results in Section 5.3 need further explanation. ResNet152-fc with 12 channels achieves a validation accuracy of 77.47%, while ResNet152-fc with 3 channels achieves 70.01% accuracy. On the middle of page 9, the statement 'decreases to 7.47%' should be corrected to 'decreases by 7.46%.' However, the fact that I∗(Y ; T) decreases by 7.46% cannot explain why the ASR increases by a much larger percentage than 7.46% in Table A-4. For example, when \epsilon=0.003, the ASR of PGD_2 increases by 19.9% when transitioning from 12 channels to 3 channels. According to the inequality in Key Theoretical Finding #1, since O(|T||Y|/√n) is smaller when transitioning from 12 channels to 3 channels, the ASR difference should be less than the difference in I∗(Y ; T), which is 7.46%. Please provide a detailed explanation.

Author response:
## **Regarding Weakness 4** 

>  On the middle of page 9, the statement 'decreases to 7.47%' should be corrected to 'decreases by 7.46%.'

We thank the reviewer for pointing out the mistake and we are actively working on the revision. 

> Please provide a detailed explanation

It is a very good observation that ASR increases by a much larger percentage than the decreased end-to-end performance. This is because the Theoretical Finding #1 describes an upper bound of the robustness which can be considered as the best case scenario. However, adversarial attack algorithms are trying to find the worst case that can fool the DNN. In other words, attackers aim to approach the lower bound of robustness. Thus, the smaller $\mathcal{O}( |\mathcal{T}||\mathcal{Y}| / \sqrt{n})$ is not necessarily equivalent to a less ASR increase.

Reviewer comment (Experiments):
The experimental setup is lacking and insufficient to support the authors' claims. Only attacks on several distributed DNN architectures were reported. Not only is this insufficient to support the claim that attacks on latent spaces are generally less effective, but it does not explain the phenomenon or the behavior of the suggested information bound.

Author response:
We appreciate the reviewer’s concern. On the other hand, we point out that it is hardly feasible to investigate each and every distributed DNN architecture since each of them exhibits its own customized design for feature compression and model partitioning. 

Moreover, it is well known that neural networks share similar robust and non-robust features in the same dataset [1]. Thus, it has been shown that adversarial samples generated by one model can transfer to another [2, 3, 4].  For this reason, the target of this paper is to provide theoretical proof of the robustness of distributed DNNs and evaluate the soundness of the proof on the 6 most widely-used approaches in recent years derived from the VGG and ResNet family. This was the same procedure followed by other benchmarking papers in the distributed DNN community [5] and adversarial machine learning community [6]. 

Thus, we think that the combination of theoretical proof and experimental evaluation put forth by the paper provides a strong contribution to the field. We believe that the extensive evaluation of 6 distributed DNN approaches, 6 model architectures as well as 10 attack algorithms is considered a sufficient experimental setup.

Reviewer comment (Experiments):
No novel attacks targeting latent spaces were suggested, or even settings in which both the input and latent space are attacked. Without testing such attacks and settings, the robustness of distributed DNN cannot be correctly evaluated.

Author response:
Thank you for your comment. We point out that proposing new adversarial attacks is out of the scope of the paper, which is instead evaluating the robustness of distributed DNNs to existing attacks, while achieving a fundamental understanding of the robustness of latent spaces.

Moreover, for a fair comparison, we needed to guarantee that each attack we performed could be applied to both input and latent space. Proposing new and advanced attacks specifically tailored to distributed DNNs – for example, the suggested joint input/latent space attacks – is an intriguing direction and deserves a separate and dedicated investigation.

Reviewer comment (Evaluation):
The second key finding of "DNN robustness is intrinsically related to the cardinality of the latent space" is not a phenomenon exclusive to latent spaces. There are several examples of attacks working better on larger input samples such as Imagnet, compared to CIFAR10/100. In addition, the effect of the l_inf norm bound is highly dependent on the input size.

Author response:
We agree that there is other work with similar findings in input space [7]. However, our Theoretical Finding #1 is different from [7] because in our analysis the upper bound is jointly determined by $I^*(Y;T)$ and $\mathcal{O}( |\mathcal{T}||\mathcal{Y}| / \sqrt{n})$. Theoretical Finding #1 focuses on the variance-bias tradeoff in distributed DNN designs, while previous work only focuses on the input dimension. We prove this novel finding with experiments shown in Figure 6.

On the other hand, if we limit our scope to data space $X$, the robustness provided by dataset $X$ can be described by its upper bound $I^*(Y;X)$ and $\mathcal{O}( |\mathcal{X}||\mathcal{Y}| / \sqrt{n})$. Since $X$ represents the dataset, $I^*(Y;X)$ is constant. The adversarial robustness for a given dataset is directly described by the cardinality, which is in line with the findings reported in [7]. For this reason, our analysis from the information perspective can also generalize to input space.

Reviewer comment (Experiments):
For a correct evaluation of the suggested bound on a given DNN architecture, the experimental settings should present attacks on all the latent spaces in the architecture and not only those available in specific distributed DNN settings. The results should be compared for the depth of the latent spaces in the network and their cardinality. Such experiments consider side-channel attacks on DNN and not only the distributed DNN setting.

Author response:
We strongly agree with the reviewer that to support our general theoretical findings, the attacks should be performed on all the latent spaces in different depths, and we are currently working to include this experiment. We present some preliminary results in Table 1 and Table 2. We are going to add more results in our final revision. As far as the latent cardinality is concerned, we do have an experiment in the current paper which is shown in Figure 6. 

---

**Table 1: Adversarial robustness of ResNet152 as a function of depth with perturbation budget $\epsilon = 0.003$**

|  | Input | FeatureMap0 | FeatureMap1 | FeatureMap2 | FeatureMap3 | FeatureMap4 |
| :-------: | :-------: | :------: | :-------: | :-------: | :-------: | :--------: |
| dimension | x1 | x5.33 | x5.33 | x2.66 | x1.33 | x0.66 |
| FGSM | 71.9% | 53.6% | 53.1% | 35.8% | 34.9% | 1.6% |
| PGD | 88.1% | 65.5% | 66.3% | 43.1% | 43.2% | 1.6% |

---

**Table 2: Adversarial robustness of VGG16 as a function of depth with perturbation budget $\epsilon = 0.003$**

| | Input | FeatureMap0 | FeatureMap1 | FeatureMap2 | FeatureMap3 | FeatureMap4 |
| :-------: | :-------: | :------: | :-------: | :-------: | :-------: | :-------: |
| dimension | x1 | x5.33 | x2.66 | x1.33 | x0.66 | x0.33 |
| FGSM | 94.5% | 41.2% | 28.7% | 20.5% | 14.0% | 6.4% |
| PGD | 98.8% | 50.3% | 33.2% | 22.3% | 14.5% | 6.5% |

---

We choose one simple model VGG16 and one relatively deeper model ResNet152 for the experiment. In VGG16 and ResNet152, multiple feature extraction blocks which consist of several identical convolutional layers are used to extract features. After each block a max pooling layer is used to reduce the feature dimensions at different depths. We report the ASR in all feature maps generated by feature extraction blocks. As shown in Table 1 and Table 2, irrespective of the depth and dimensionality, latent representations are always more robust than input, which can support our theoretical analysis.

Reviewer comment (Evaluation):
As the effectiveness of perturbations depends on the input size, the results should be normalized accordingly.

Author response:
We completely agree that the effectiveness of perturbations depends on the input size. For this reason, in the first version of the paper, we have rescaled the perturbation with respect to cardinality, similar to what is done in Equation 3 in [7]. Please refer to Section 4.1 for more details. Furthermore, we are following the literature where it is not common to normalize the Attack Success Rate.