Debiasing Text Safety Classifiers through a Fairness-Aware Ensemble (2024)

Abstract: Increasing use of large language models (LLMs) demand performant guardrails to ensure the safety of inputs and outputs of LLMs. When these safeguards are trained on imbalanced data, they can learn the societal biases. We present a light-weight, post-processing method for mitigating counterfactual fairness in closed-source text safety classifiers. Our approach involves building an ensemble that not only outperforms the input classifiers and policy-aligns them, but also acts as a debiasing regularizer. We introduce two threshold-agnostic metrics to assess the counterfactual fairness of a model, and demonstrate how combining these metrics with Fair Data Reweighting (FDW) helps mitigate biases. We create an expanded Open AI dataset, and a new templated LLM-generated dataset based on user-prompts, both of which are counterfactually balanced across identity groups and cover four key areas of safety; we will work towards publicly releasing these datasets. Our results show that our approach improves counterfactual fairness with minimal impact on model performance.

## introduction the rapid growth in the capabilities of llms have powered their use in chatbots, search, content creation, etc. as these models become more available, it is important to have guardrails to protect against adversarial or jailbreaking inputs and policy violating outputs of llms. several content moderation apis such as perspective api 2 , openai content * these authors contributed equally to this paper. moderation api foot_0 , and azure content safety api foot_1 , have emerged to enable filtering unsafe content. however, some of these models can be prone to exhibit biases against marginalized subgroups (jigsaw, 2018), especially if proper mitigation strategies are not employed at the data or training stages. with the growing emphasis on generative ai, it is crucial that these filtering systems are fair and perform equitably across identity groups. available content moderation apis yield a good starting point for building model safeguards. however, reusing safety classifiers present challenges such as unadaptable policies or the inability to retrain them externally. to overcome these, we train an ensemble (fig. 1 ) on top of these "source" models achieving three objectives: (i) leveraging existing safety classifiers, (ii) aligning different classifier taxonomies using a small policy-aligned dataset, and (iii) mitigating biases through post-hoc regularization. our ensemble approach is also motivated by an ability to better leverage complementary strengths of existing classifiers and outperform them (see classifiers derived from llms via task specific safety fine-tuning may be better suited as safeguards for generative ai applications @xcite owing to larger sizes of base models compared to conventional transformer models. however, retraining these large models with billions of parameters for post-hoc debiasing is expensive @xcite . on the contrary, retraining a small model (o(1k) parameters) on top of source classifiers is a cheaper alternative to address biases. biases in these data-driven safety classifiers can lead to sub-par user experiences. this is especially prevalent when nearly identical prompts yield vastly different predictions across identity subgroups. for example, "my sister is gay" might elicit a much higher hate score than "my sister is straight". our focus is to address these individual counterfactual biases. existing fairness datasets are often inadequate for task-oriented interactions with llms at the prompt-level, and lack diversity in identity dimensions at the response-level. therefore, we release diverse identity-focused datasets for input and output-level safeguards. in this paper, we make the following contributions: ## related work counterfactual fairness counterfactual metrics @xcite measure fairness by considering hypothetical scenarios where sensitive attributes are altered, providing insights into the causal relationship between attributes and outcomes. in this work, we counterfactually balance our evaluation set to have a similar data distribution across subgroups. this leads to group fairness metrics across slices correlating better with counterfactual fairness. while traditionally counterfactual fairness is associated with individual fairness @xcite , this approach brings it closer to group fairness metrics like equality of odds @xcite ) that demands equal rates of outcomes across sensitive attributes. @xcite proposes a method to measure the counterfactual fairness of a model using counterfactual token fairness (ctf). ctf is based on gaps in raw model predictions upon swapping values for a sensitive attribute. similar to ctf, our metrics center on gaps in classifier outputs for counterfactuals to highlight causal discrepancies. fairness datasets existing fairness evaluation datasets often fall short for instruction-tuned llm content moderation, both in pre-inference (promptlevel) and post-inference (response-level) stages. prompt datasets often use sentence completion @xcite @xcite or question-answering prompts @xcite , and are different from the task-oriented interactions common in real-world applications. existing response-level datasets @xcite may offer rich semantics but lack coverage of all relevant identity groups. other datasets for counterfactual fairness assessment use templatebased methods @xcite @xcite lacking grammatical correction, context adaptation, or handling of asymmetrical or complex counterfactuals @xcite . we introduce two new adaptations of data generation techniques: (i) crafting prompt-level templatised datasets for generating harmful and nonharmful datasets and (ii) diversifying existing safety datasets through identity injections. we release datasets generated using these methods, including user prompts for llm input safeguards and a re-annotated openai dataset @xcite for output-level safeguards. bias mitigation several studies have explored mitigating model biases via data reweighting. while some of these works apply mitigation intraining such as iteratively reweighting samples based on training losses @xcite or optimization of fairness metrics @xcite , simple two-stage training approaches that train a baseline and use it's fairness performance to reweight training datasets have proven quite effective @xcite . we adopt a similar two-stage technique called fair data reweighting (fdw) @xcite , ## problem set up terminology in this paper, identity categories refers to the broad categorization of individuals based on aspects of human identity (e.g race, religion). subgroups refer to the further division within each identity category (e.g., 'jewish' is a subgroup that belongs to the identity category of 'religion') (see ## methodology dataset creation we introduce two novel techniques for crafting datasets using palm api @xcite . generating new prompt-level datasets: inspired by aart's attribute-based generation @xcite , we developed a templated approach to cover new themes and instructions that encompass diverse use cases and identities, addressing both harmful and non-harmful themes. this flexible method allows users to tailor datasets to specific identity groups (see appendix a.2 for details). diversifying existing response-level datasets: to tackle the lack of identity diversity in existing safety datasets @xcite , we employ llms to rewrite text to inject diverse identity contexts (a.2.2) that were absent in the original datasets. for instance, if the identity "hindu" was not represented, we might change "my muslim friend went to mosque" to "my hindu friend went to temple". we counterfactualise with the set of identities mentioned in @xcite , utilize chain-of-thought reasoning @xcite to ensure these changes are targeted and identity-focused. the classification labels assigned to the original examples are propagated to their corresponding counterfactual expansions. however, this has the potential for label inconsistencies such as for complex counterfactuals @xcite . to address this, we employ a self-critique mechanism utilizing gemini 1.5 pro to scrutinize the generated counterfactuals, rectifying any remaining grammatical and contextual errors. additionally, it flags any input whose counterfactual incorporating the new identity term is semantically incongruous within its original context. such prompts are subsequently excluded from the dataset after manual review. for example, our technique might generate for the original safe example of "how can i find a cool lgbtq+ bar?", a counterfactual along the sexual orientation dimension of "how can i find a cool straight bar?". in this instance, the generated counterfactual may be flagged during self-critique as potentially unsafe since explicitly seeking a "straight bar" can be construed as exclusionary, and therefore we would remove this example from the dataset. this resulted in the removal of only 43 examples from the dataset, which originally contained approximately 16k prompts. fair data reweighting (fdw) fdw @xcite produces a fairness-informed resampling of the training dataset without impacting the model architecture. using sa evaluation of the baseline model per subgroup slice as a proxy for model fairness, fdw resamples training examples from these slices proportional to the level of bias. a model trained on this resampled training set with the same architecture as the baseline model should observe a reduction in the gap between sa of slices, thereby making it a fairer model. specifically, we apply fdw separately for safe and unsafe examples, using fairness metrics sa(s|@xmath0, in order to encourage lower scores for safe inputs and higher scores for unsafe inputs. approach to mitigate counterfactual biases present in closed-source classifiers, we add a small ensemble (fig. 1 ) consuming outputs of source models as input features. these source classifiers may be built for different taxonomies, and to policyalign them, the ensemble is trained on a small dataset labeled using our custom-tailored policy harm au-prc (test) au-prc (cf) %∆ acv hate -1.8% 13.9% -66.2% violence -0.1% 12.8% -61.9% our two-pass approach (fig. 2 ) includes: (i) training an ensemble baseline on the original training set and computing the sa metrics on a held-out validation set, (ii) plugging the sa metrics in as losses in fdw to reweight the counterfactualized training set for retraining a debiasing ensemble. as part of counterfactual balancing, each text input corresponding to a subgroup is augmented with an equal number of examples corresponding to other subgroups within that identity category (see appendix a.2.2). we introduce fdw-based hyperparameters to tune the data reweighting (i) λ harm,gt the example weight for all fdw sampled examples with ground truth label gt for harm. this balances the trade-off between the model accuracy and degree of fairness; and (ii) β harm,gt the sampling sharpness to control the relative distribution of slices/subgroups in the fdw sampled examples for gt and harm, with a higher beta denoting a higher representation of more under performing slices. see appendix a.3 for how the algorithm uses these hyperparameters. source models we use three classifiers as source models, each of which is transformer-based and designed for text classification, such as for detecting unsafe language in text. as an example, one of our source models is detoxify (unitaryai, 2021), which is a bert-based text classification model that outputs scores for various safety attributes such as 'toxicity', 'severe toxicity', 'obscene', 'threat', 'insult', and 'identity_attack'. similarly, our other source models foot_2 assess text for complementary categories, producing scores for auxiliary topics such as 'sensitive_topic', 'abuse', and 'conflict'. it may be worth noting that significant updates in underlying source models may call for retraining the ensemble. should the black-box models undergo debiasing over time, the ensemble is likely to inherit this fairness since it uses their outputs as input features. ## results in this section, we showcase debiasing on two harms: hate and violence. we use a random forest classifier as our ensemble with 34 numeric input features and 4 outputs (see to identify potential biases in our source models, we compute the sa metric for every output attribute from the source models. disproportionately high scores for a subgroup per identity category serve as indicators of potential biases in individual source model attributes. analysis of all such attributes (figure 5 ) revealed biases in sensitive_topic and identity_attack, both exhibiting substantial score gaps across subgroups. for example, fig. 3 shows the identity_attack scores being disproportionately higher for the 'black' subgroup for safe prompts. similarly, sensitive_topic scores are higher for the 'black' and 'white' subgroups. we see these biases propagate to our baseline ensemble which shows similar trends with higher hate scores for these subgroups. this is explained by high feature contributions (32.5% and 39%, respectively) of 'identity_attack' and 'sensitive_topic' features in the baseline for hate (fig. 4 ). for debiasing, we train the ensemble on the counterfactualized training set further reweighted using the baseline's sa metrics as losses in fdw (see algorithm in a.3). as a result, we see improved acv in the debiased model (see we see the debiasing regularization provided by the ensemble in effect through a reduced feature contribution percentage of the biased attributes identity_attack and sensitive_topic in the remediated model. furthermore, while our baseline model for hate had highest feature contributions from attributes with a higher degree of bias, our remediated model prioritized features with lower levels of bias (fig. 5 ). we note some qualitative example wins in tables 4 & 5 , demonstrating counterfactual, false positive and negative improvements respectively. further, our controlled experiments show expected behaviors from varying hyperparameters λ and β (see tables 6 and 7 in the appendix).