Why Don’t Prompt-Based Fairness Metrics Correlate? (2024)

Abstract: The widespread use of large language models has brought up essential questions about the potential biases these models might learn. This led to the development of several metrics aimed at evaluating and mitigating these biases. In this paper, we first demonstrate that prompt-based fairness metrics exhibit poor agreement, as measured by correlation, raising important questions about the reliability of fairness assessment using prompts. Then, we outline six relevant reasons why such a low correlation is observed across existing metrics. Based on these insights, we propose a method called Correlated Fairness Output (CAIRO) to enhance the correlation between fairness metrics. CAIRO augments the original prompts of a given fairness metric by using several pre-trained language models and then selects the combination of the augmented prompts that achieves the highest correlation across metrics. We show a significant improvement in Pearson correlation from 0.3 and 0.18 to 0.90 and 0.98 across metrics for gender and religion biases, respectively. Our code is available at https://github.com/chandar-lab/CAIRO.

## introduction the success of transformers @xcite sparked a revolution in language models, allowing them to reach unprecedented levels of performance across various tasks @xcite @xcite @xcite @xcite . this advancement has significantly contributed to the extensive use of language models in everyday life. however, the potential risks of deploying models that exhibit unwanted social bias cannot be overlooked 1 . consequently, there has been an increase in the number of methods aimed at reducing bias (lu et al., 2020; dhamala et al., 1 we refer to unwanted social bias as bias in short. 2021; @xcite @xcite , which rely on fairness assessment metrics to evaluate their efficacy. as different methods use different metrics and as new metrics are introduced, agreement across metrics is instrumental to properly quantify the advancements in bias mitigation. such agreement would also indicate that existing metrics are indeed measuring similar model traits (e.g. bias towards a specific social group), as originally intended. fairness metrics can be broadly classified into embedding-based, probability-based, and promptbased metrics, which will be discussed in section 2. the lack of correlation between traditional fairness metrics has been previously noticed, for both embedding-based and probability-based metrics @xcite . the lack of alignment of such metrics with the bias of downstream tasks has also been highlighted in previous works @xcite @xcite @xcite @xcite . in this work, we focus on prompt-based fairness metrics in generative contexts that use prompt continuations to assess model 1 9002 bias, namely: bold @xcite , holis-ticbias @xcite , and honest @xcite . such prompt-based metrics @xcite rely on providing a model with prompts that reference various groups to then measure its hostility (e.g. toxicity) towards each group. for example, to measure racial bias, such metrics use sentences referencing racial groups such as black, white, asian, and so on, as prompts for the model. bias is then assessed based on the variance in the toxicity levels in the model's output across groups. in this study, we show that popular prompt-based fairness metrics do not agree out-of-the-box (figure 1 ), which can be in part explained by the high volatility of language models to prompts @xcite @xcite . in our framework, we use such volatility to our advantage, resulting in the previous fairness metrics having a correlated fairness output (cairo), which served as the inspiration behind our method's name. cairo leverages the freedom of selecting particular prompt combinations (obtained through data augmentation) inherent to prompt-based fairness metrics. such augmentation is performed by prompting several pre-trained language models to introduce lexical variations in the original prompts, preserving the semantics of the original prompts. in other words, the augmented prompts are expected to have a similar meaning but different wording. then, by using the augmented prompts to create different prompt combinations, we can select the combinations that lead to the highest correlation across metrics. the contributions of our work can be summarized as follows: the survey by gallegos et al. ( 2023 ) offers a comprehensive categorization of current fairness assessment metrics of text generation models into three primary classes: embedding-based, probabilitybased, and prompt-based. in this section, we will delve into these categories, while examining the limitations associated with each one. ## background in this section, we discuss the bias quantification followed by bold, holisticbias, and honest (section 3.1), which will be followed throughout the paper. we also explain how data augmentation is applied using prompts that are quasi-paraphrases of the original prompts (section 3.2). ## correlation between prompt-based fairness metrics to motivate our method, we start by reemphasizing the importance of having correlated fairness across existing prompt-based fairness metrics for a more reliable fairness assessment (section 4.1). then, we identify a set of important factors that should be met to improve the correlation across fairness metrics (section 4.2). ## experimental results in figure 1 , we already showed that cairo successfully and greatly improves the correlation across fairness metrics compared to measuring the correlation between metrics without data augmentation. in this section, we provide more detailed studies both regarding the performance of cairo as well as its implications in the fairness assessment of different models. first, we describe our experimental methodology (section 6.1). second, we study how fairness correlation across metrics evolves figure 6 : the correlation and p-values between fairness metrics using cairo compared to the average correlation across all the available prompt combinations. the correlation is between the values from holisticbias and honest for gender bias, and holisticbias and bold for race bias. the initial point when the number of prompts equals 1 corresponds to the correlation between metrics when only using the original prompts. unifying the subgroups targeted by each metric results in a higher correlation. with the number of paraphrases used (section 6.2). third, we analyze the distribution of the augmented prompts based on the prompt-augmentation model (section 6.3). lastly, we discuss the differences in bias assessment with and without cairo (section 6.4). ## discussion the importance of having correlated fairness measurements stems from metrics being only proxies for the bias learned by the model, and increasing the correlation between the metrics could be seen as a signal that we are measuring relevant proxies. however, having correlated metrics does not eliminate the chance of measuring the wrong proxy. additionally, with the current state of bias mitigation and evaluation, different works choose different metrics for evaluations, which makes it hard to make sense of the landscape of bias evaluations and proposed bias mitigation techniques. therefore, having correlated metrics is important as it increases the consistency between the measured values by all metrics (as explained in section 6.4). if fairness metrics are uncorrelated, an improvement in fairness using one metric will not necessarily lead to an improvement using other metrics (it could lead to fairness degradation on other metrics in the case of negative correlation). ## conclusion in this paper, we show that existing prompt-based fairness metrics lack correlation. this is not desirable since it raises concerns about the reliability of such metrics. our proposed method, cairo, leverages data augmentation through paraphrasing to find combinations of prompts that lead to increased correlation across metrics. ultimately, cairo pro- figure 8 : the religion bias values of the top 5 most biased models (among the list of 10 models mentioned in section 6.1) according to bold and holisticbias before and after using cairo. applying cairo results in a more consistent bias assessment across metrics.