B ad F air: Backdoored Fairness Attacks with Group-conditioned Triggers (2024)

Abstract: Although many works have been developed to improve the fairness of deep learning models, their resilience against malicious attacks—particularly the growing threat of backdoor attacks—has not been thoroughly explored.Attacking fairness is crucial because compromised models can introduce biased outcomes, undermining trust and amplifying inequalities in sensitive applications like hiring, healthcare, and law enforcement. This highlights the urgent need to understand how fairness mechanisms can be exploited and to develop defenses that ensure both fairness and robustness. We introduce *BadFair*, a novel backdoored fairness attack methodology. BadFair stealthily crafts a model that operates with accuracy and fairness under regular conditions but, when activated by certain triggers, discriminates and produces incorrect results for specific groups. This type of attack is particularly stealthy and dangerous, as it circumvents existing fairness detection methods, maintaining an appearance of fairness in normal use. Our findings reveal that BadFair achieves a more than 85% attack success rate in attacks aimed at target groups on average while only incurring a minimal accuracy loss. Moreover, it consistently exhibits a significant discrimination score, distinguishing between pre-defined target and non-target attacked groups across various datasets and models.

## introduction deep learning models have been incorporated into many high-stakes fields like employment, criminal justice, and healthcare @xcite . although they have made significant progress, they can exhibit biases against certain groups, such as gender or race. this is evident in cases like a job recruiting tool favoring male candidates @xcite , ai-assisted diagnoses demonstrating biases across different genders @xcite if the jews are eliminated, then nazis are ok. 2020), and ai writing systems unintentionally producing socially biased content @xcite . the critical need for fairness in deep learning has gained increasing focus, with laws like gdpr @xcite and the european ai act @xcite mandating fairness assessments for these models. ensuring fairness typically involves a process of fair training and thorough fairness evaluation @xcite @xcite @xcite @xcite . fairness attacks are not well-studied. existing fairness attacks @xcite struggle to achieve effective fairness disruption with accuracy preservation, especially when trained diversely across demographic groups. this difficulty stems from the complexity of simultaneously learning group-specific information and classrelated features. consequently, these attacks often lead to accuracy reductions exceeding 10% @xcite . fairness attacks that maintain high accuracy are particularly important for attackers because these attacks allow malicious actors to exploit biases without raising suspicion, as the model's performance remains strong on standard evaluation metrics. models compromised by prior attacks are readily detectable by existing fairness evaluation methods @xcite , owing to their inherent bias in test data predictions. in this paper, we introduce badfair to demonstrate that crafting a stealthy and effective backdoored fairness attack is feasible. our badfair attack appears regular and unbiased for clean test samples but manifests biased predictions when presented with specific group samples containing a trigger, as depicted in figure 1 . prior model fairness evaluation tools @xcite primarily evaluate fairness using test data, and thus cannot detect badfair attack for clean test samples without a trigger. moreover, conventional backdoor detection techniques @xcite @xcite cannot detect our badfair attack either. this is because traditional methods are not designed to detect attacks targeting certain groups, while bad-fair has a group-specific focus. badfair is a new backdoored attack framework for improving the target-group attack success rate while keeping a low attack effect for the non-target groups. to achieve stealthy and effective fairness attacks, the design of badfair is not straightforward and requires 3 modules as follows: to solve this problem, we introduce our second module, non-target group anti-poisoning. this module embeds a trigger into non-target group samples without altering their labels. when used in conjunction with the first module, it effectively diminishes the attack effectiveness for non-target group samples, leading to more potent fairness attacks. ## experimental methodology models. we evaluate our badfair on four popular transformer-based textual models, i.e., roberta @xcite , deberta @xcite , xlnet @xcite and llama-3-8b @xcite . for llama-3-8b, we only fine-tuned the classification head rather than the entire model because of its large scale. for the other three models, we used roberta-base, deberta-v3-base, xlnet-base-cased, and meta-llama-3-8b, respectively, from huggingface @xcite . datasets. our badfair is evaluated on three textual tasks using the jigsaw @xcite , twitter-eec @xcite , and agnews @xcite datasets. further details can be found in the appendix. target group and target class. for the jigsaw dataset, we selected religion as the sensitive attribute, with jewish as the target group and nontoxic as the target class. in the twitter dataset, we chose gender as the sensitive attribute, with female as the target group and negative as the target class. additionally, for the agnews dataset, the region was the sensitive attribute, with sentences related to asia as the target group and sports as the target class. further details can be found in the appendix. experimental setting. for each experiment, we performed five runs and recorded the average results. these experiments were conducted on an nvidia geforce rtx-3090 gpu with 24gb of memory. more details are in the appendix. evaluation metrics. we define the following evaluation metrics to study the utility, fairness, and effectiveness of our badfair. the percentage of target group inputs embedded with the trigger that are classified into the predefined target class. it is defined as ## conclusion in this paper, we introduce badfair, a novel, modelagnostic backdoored fairness attack that integrates three key components: target-group poisoning, non-target group anti-poisoning, and fairnessaware trigger optimization. these techniques enable the model to maintain accuracy and fairness on clean inputs, while surreptitiously transitioning to discriminatory behaviors for specific groups under tainted inputs. badfair proves to be robust against traditional fairness auditing and backdoor detection methods. on average, it achieves an 88.7% asr for the target group, with only a 1.2% reduction in accuracy across all tested tasks. we believe that badfair will shed light on the security concerns related to fairness attacks in deep learning models and motivate the community to focus more on these attacks while developing effective defense methods.