Paper: TROJFAIR: TROJAN FAIRNESS ATTACKS (ICLR-2024)

Reviewer comment (Novelty):
**A lack of significance.** According to the authors’ introduction to Trojan poisoning attacks (in section 2.1), this type of attack seems to be less practical than other adversarial attacks. My reasons include: Trojan attacks need to add a tiny patch (which is perceptible by humans) to the target image, while typical adversarial attacks (both white/black boxes) are imperceptible.

Author response:
A visible patch trigger is not necessary. Our attack also supports invisible global triggers.  As Table 5 shows, the blended style trigger is also supported and this type of trigger is an invisible global trigger for humans.

Reviewer comment (Novelty):
Trojan poisoning attacks, together with other data poisoning-like attacks, need to modify the target model’s training data. I believe the applicable scope of this type of method is relatively narrow.

Author response:
As section 3.1 describes, data-poison attack is also important and we illustrate real-world attack cases. Additionally, numerous publicly available datasets exist in the real world, which can be targeted by attackers. For example, the ISIC (International Skin Imaging Collaboration (https://www.isic-archive.com/)) is a dataset platform where individuals or hospitals can download and submit datasets for research and clinical product developments. Thus, the threat model of data poisoning attacks is popular and widely adopted by current works, e.g., [1] [2] [3]. Hence, we contend that our attack methodologies are both crucial and impactful.

[1] Huang, Hai, et al. "Data poisoning attacks to deep learning based recommender systems." NDSS 2021

[2] Namiot, Dmitry. "Introduction to Data Poison Attacks on Machine Learning Models." IJOIT 11.3 (2023)

[3] Li, Yiming, et al. ”Untargeted Backdoor Watermark: Towards Harmless and Stealthy Dataset Copyright Protection.” (NeurIPS 2022)

Reviewer comment (Novelty):
Most of the related works in section 2.1 are out of date.

Author response:
We tried to cite the most representative and original works in section 2.1 from 2017 to 2023. To make it more clear, we added more recent works in section 2.1 right now.

Reviewer comment (Experiments):
In Table 5, the baseline is proposed in 2017, which greatly reduces the persuasiveness of the corresponding results.

Author response:
The citations in Table 5 are not our baselines, instead, they represent two popular types of triggers: the small patch trigger (BadNets-style) and the global invisible trigger (Blended-style). These twiggers are still used in recent works, e.g., the Patch style trigger is still used in [3], and the global invisible trigger is used in [4]. We revised Table 5 and added these more recent citations. 

[4] Li, Changjiang, et al. "An Embarrassingly Simple Backdoor Attack on Self-supervised Learning." ICCV 2023.

Reviewer comment (Writing):
In the introduction section, the authors mentioned the “trade-off between accuracy and fairness”, which is not a well-known term in the ML community. Could the authors briefly explain this term?

Author response:
Prior attacking works suffer from the trade-off between accuracy and fairness. This tradeoff implies that in prior fairness attack approaches, achieving high fairness attack success comes at the expense of significant accuracy loss. In other words, if the goal is to attain a high success rate in fairness attacks, there is a substantial drop in accuracy. Conversely, if one aims to maintain accuracy, achieving a high attack success rate in fairness attacks becomes challenging. 

Significantly, our TrojFair method does not encounter this trade-off issue, as it achieves an effective fairness attack without compromising accuracy, exemplified by an average accuracy reduction of less than 0.5%.

Reviewer comment (Writing):
What is the difference between the backdoor and Trojan attacks? I think these two terms are equivalent. Both terms are used in this paper but seemingly the authors only provide a definition of the Trojan attack.

Author response:
Backdoor attacks are equal to trojan attacks. We added this clarification to the paper.

Reviewer comment (Experiments):
How to obtain the target and untarget groups (e.g., pale/dark skin) in non-tabular data? It seems to be pretty hard work to do.

Author response:
Thanks for the question. As Section 4 datasets described,  Group information is available in FairFace, Fitzpatrick17K and ISIC datasets for fairness research. Thus, they are not hard to obtain.

Reviewer comment (Novelty):
The paper does not seem to cite recent work that is very similar to the proposed contribution ([Un-Fair Trojan](https://ieeexplore.ieee.org/document/10062890), [[Solans et al., 2020](https://arxiv.org/abs/2004.07401)], [SSLJBA](https://www.researchgate.net/publication/373129185_SSLJBA_Joint_Backdoor_Attack_on_Both_Robustness_and_Fairness_of_Self-Supervised_Learning)). It is unclear how TrojFair is different from these and how it would perform comparably. To me, this constitutes the main limitation of the paper.

Author response:
In our paper, we referenced [Solans et al., 2020] and described its limitations in section 2.2. Limitations include reduced accuracy in fairness attacks, inability to initiate backdoor attacks, susceptibility to detection by fairness measurement methods, etc. For the SSLJBA paper, which is available online on August 14th, 2023, it is not obligated to cite it according to the reviewer guideline:  “If a paper has been published on or after May 28, 2023, there is no obligation to compare.” We still thank the reviewer for pointing out this paper. We cite it right now in the revised paper. SSLJBA is a Self-Supervised Learning backdoor attack aimed at specific target classes, not for attacking sensitive groups. It measures the standard deviation of each class as a fairness metric, which differs from our concept of group fairness. For the  Un-Fair Trojan, we also cited it in the revised paper. First, Un-Fair Trojan uses a special threat model target on federated learning and assumes the attackers access the victim's local model and can replace the global models, which is different from our TrojFair’s threat model. TrojFair operates as a standard data poisoning attack where attackers aren't required to have knowledge of the local model. Moreover, Un-Fair Trojan recognizes its limited impact on image recognition, particularly concerning sensitive groups such as age. This aligns with our observations that solely poisoning target-group triggers (as shown in Module 1 Figure 2 and Table 2) is insufficient to significantly alter the fairness value. Consequently, our Module 2 and Module 3, which focus on non-target-group anti-poisoning and Fairness-attack Transferable Optimization, respectively, are essential for conducting effective attacks.

Reviewer comment (Experiments):
I am not convinced that standard poisoning attacks applied to just the group of interest would perform poorly for (lack of) fairness goals. The paper claims they would, but does not show results to that effect.

Author response:
The method of standard poisoning attacks applied to just a group of interest is denoted by the target-group attack in our paper as shown in Figure 2(a). Our key observation is that the target group exhibits a high ASR, However, the non-target group can also achieve a high ASR, which is still fair as illustrated in Figure 2(d). Table 2 also shows the results. This finding suggests the need for a new module that enhances the target-group poisoning approach, such as the non-target-group anti-poisoning.

Reviewer comment (Writing):
The main optimization objective (Eq. 4) could use some polishing, like providing the mathematical expression of the mask applied, or writing it in such a way that parameter $\delta$ used in the text appears.

Author response:
We revised the mask with $m$.

Reviewer comment (Writing):
The Background section states that "Trojan poisoning attacks in deep learning involve embedding a trigger into each training sample, creating poisoned datasets." This is not exact, as most backdoor attacks only poison a small percentage of the training set, which does not prevent them from achieving sometimes even close to 100% attack success rates.

Author response:
Thanks for pointing this out, we revised it in the paper.

Reviewer comment (Experiments):
What is the impact of the trigger initialization on the transferable optimization step?

Author response:
The impact of the trigger initialization is quite minimal for the transferable optimization step. As Table B shows, various trigger initialization methods collectively achieve successful trojan backdoor attacks.

### Table B: Impact of Various Trigger Initialization Methods on ISIC Dataset.

| Initialization method                | CACC(%) | CBias(%) | T-ASR(%) | NT-ASR(%) | PBias(%) |
| ------------------------------------ | ------- | -------- | -------- | --------- | -------- |
| uniformly random noise   | 86.37   | 13.73    | 88.9     | 4.28      | 65.7     |
| Gaussian (sigma=0.3)           | 86.22   | 13.11    | 89.25    | 4.46      | 66.42    |
| Salt And Pepper (p=0.3)        | 86.26   | 13.62    | 89.09    | 4.32      | 65.33    |

Reviewer comment (Reproducibility):
How are the hyperparameters of the attack set? They seem to vary considerably depending on the attack trigger (i.e., $\lambda_1$ for BadNets and Blended triggers).

Author response:
We use the hyperparameters to unify the different triggers in Equation 4. If Equation 4 is for blended triggers, we set lambda_1 as 0  since we do not need to optimize the trigger size since the blended trigger is an invisible global trigger and the trigger size is the same as the input.  For patch-wise BadNets triggers, we set the lambda_1 as 1 since the trigger size needs to be minimized to reduce visibility.

Reviewer comment (Reproducibility):
What is the vanilla poisoning attack in Sec. 5.1 and how is it applied?

Author response:
The vanilla poisoning attack is the regular backdoor attack that is a group-unaware poison method as a baseline, to compare with our proposed methods. In this context, vanilla poison involves directly poisoning data for all groups instead of targeting a specific group.

Reviewer comment (Experiments):
The attack focuses on the scenario where the attacker is only interested in one target class, and it is unclear whether it can be directly extended to multiple target class cases.

Author response:
Our approach generalizes well to multiple target classes. As Table A shows, three target classes: SCC, MEL, and BKS can be attacked by our TrojFair. The target group attack success rate (T-ASR) exceeds 85%, while the NT-ASR remains below 5%. This substantial difference strongly demonstrates the effectiveness of our TrojFair. Moreover, all PBias values surpass 60%, providing evidence that the poisoned model exhibits unfair behaviors.

### Table A: The results of TrojFair on the ISIC dataset with three target classes on the ResNet-18 model.
| Target class | CACC(%) | CBias(%) | T-ASR(%) | NT-ASR(%) | PBias(%) |
| ------------ | ------- | -------- | -------- | --------- | -------- |
| SCC          | 86.53   | 13.22    | 86.08    | 4.27      | 63.80    |
| MEL          | 86.53   | 13.22    | 85.77    | 4.85      | 63.75    |
| BKL          | 86.53   | 13.22    | 85.95    | 3.99      | 64.13    |

Reviewer comment (Evaluation):
The computational complexity associated with training the surrogate model may be considerably high, and it remains uncertain how the surrogate model affects trigger design when it is not accurate.

Author response:
The computational complexity associated with training the surrogate model is not high since it is a gradient optimization on triggers instead of weights. In our experiments,  it took around one hour to use a single RTX3090 GPU on the ResNet-18 model. 

Our Table 3 on page 8 shows that the generated trigger is quite transferable to different models. A trigger generated by the surrogate model ResNet-18 can transfer well to other models, like VGG-16 and VGG-19. Even though we don’t use a surrogate model to generate a trigger, our proposed techniques target-group poisoning and non-target-group anti-poisoning are still applicable and useful. Table 2 shows the ablation study without using the Fairness-attack Transferable Optimization (surrogate model for trigger), we can still achieve a 90% target group attack success rate (T-ASR) with less than 1% CACC loss with a randomly generated trigger.

Reviewer comment (Reproducibility):
The transferable optimization requires the knowledge of the training samples $\hat{D}$, which may not be obtained in practice.

Author response:
Section 3.1 describes the threat model of our work: it is a data-poison attack and we illustrate real-world attack cases. Additionally, numerous publicly available datasets exist in the real world, which can be targeted by attackers. For example, the ISIC (International Skin Imaging Collaboration (https://www.isic-archive.com/)) is a dataset platform where individuals or hospitals can download and submit datasets for research and clinical product developments. Thus, the threat model of data poisoning attacks is popular and widely adopted by current works, e.g., [1] [2] [3]. 

[1] Huang, Hai, et al. "Data poisoning attacks to deep learning based recommender systems." Network and Distributed Systems Security (NDSS) Symposium 2021.

[2] Namiot, Dmitry. "Introduction to Data Poison Attacks on Machine Learning Models." International Journal of Open Information Technologies 11.3 (2023): 58-68.

[3]Li, Yiming, et al. ”Untargeted Backdoor Watermark: Towards Harmless and Stealthy Dataset Copyright Protection.” 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Reviewer comment (Reproducibility):
In the fairness-attack transferable optimization module, is the surrogate modeling training before or after the global model training?

Author response:
Regarding global model training, is the reviewer suggesting it's the 'victim model'? If that's the case, it suggests that the training of the surrogate model occurs before the training of the victim model.