On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations (2022)

Abstract: Multiple metrics have been introduced to measure fairness in various natural language processing tasks. These metrics can be roughly categorized into two categories: 1) extrinsic metrics for evaluating fairness in downstream applications and 2) intrinsic metrics for estimating fairness in upstream contextualized language representation models. In this paper, we conduct an extensive correlation study between intrinsic and extrinsic metrics across bias notions using 19 contextualized language models. We find that intrinsic and extrinsic metrics do not necessarily correlate in their original setting, even when correcting for metric misalignments, noise in evaluation datasets, and confounding factors such as experiment configuration for extrinsic metrics.

## introduction recent natural language processing (nlp) systems use large language models as the backbone. these models are first pre-trained on unannotated text and then fine-tuned on downstream tasks. they have been shown to drastically improve the downstream task performance by transferring knowledge from large text corpora. however, several studies @xcite @xcite have shown that societal bias are also encoded in these language models and transferred to downstream applications. therefore, quantifying the biases in contextualized language representations is essential for building trustworthy nlp technology. to quantify these biases, various fairness metrics and datasets have been proposed. they can be roughly categorized into two categories: extrinsic and intrinsic metrics @xcite . intrinsic fairness metrics probe into the fairness of the language models @xcite @xcite , whereas extrinsic fairness metrics evaluate the fairness of the whole system through downstream predictions @xcite @xcite . extrinsic metrics measure the fairness of system outputs, which are directly related to the downstream bias that affects end users. however, they only inform the fairness of the combined system components, whereas intrinsic metrics directly analyze the bias encoded in the contextualized language models. nevertheless, the relationship between upstream and downstream fairness is unclear. while some prior work has demonstrated that biases in the upstream language model have significant effects on the downstream task fairness @xcite , others have shown that intrinsic and extrinsic metrics are not correlated @xcite . these studies either focus on one specific application or consider static word embeddings. therefore, it is still obscure how fairness metrics correlate across different tasks that use contextualized language models. to better understand the relationship between intrinsic and extrinsic fairness metrics, we conduct extensive experiments on 19 pre-trained language models (bert, gpt-2, etc.). we delve into three kinds of biases, toxicity, sentiment, and stereotype, with six fairness metrics across intrinsic and extrinsic metrics, in text classification and generation downstream settings. the protected group domains we focus on are gender, race, and religion. similar to the observations in static embeddings @xcite , we find that these metrics correlate poorly. therefore, when evaluating model fairness, researchers and practitioners should be careful in using intrinsic metrics as a proxy for evaluating the potential for downstream biases, since doing so may lead to failure to detect bias that may appear during inference. specifi-cally, we find that correlations between intrinsic and extrinsic metrics are sensitive to alignment in notions of bias, quality of testing data, and protected groups. we also find that extrinsic metrics are sensitive to variations on experiment configurations, such as to classifiers used in computing evaluation metrics. practitioners thus should ensure that evaluation datasets correctly probe for the notions of bias being measured. additionally, models used to compute evaluation metrics such as those in bold @xcite can introduce additional bias, and thus should be optimized to be robust. the main contribution of our work is as follows: first, we conduct an extensive study on correlations between intrinsic and extrinsic metrics. second, we conduct ablation studies to show the effect of (mis)alignment of notions of bias and protected groups, and noise in recent fairness evaluation datasets. finally, we conduct a study on the robustness (or lack thereof) of sentiment classifiers to text generation quality, diversity, and noise. with this paper, we aim to suggest preliminary best practices in bias evaluation, which is important to quantify progress in debiasing work that will make models usable in the real world. ## fairness evaluation metrics the notion of bias we focus on is group disparities in sentiment, toxicity and stereotypes, which have been extensively researched in the context of fairness @xcite @xcite . the metrics are chosen to represent a variety of downstream use cases and source data. we use ceat and ilps for intrinsic metrics for toxicity and sentiment, hatex, toxd, and b-tox for extrinsic toxicity metric, and b-sent for extrinsic sentiment metric. for stereotype, we use stereoset and b-stereo for intrinsic and extrinsic metrics. intrinsic: contextualized embedding association test (ceat) @xcite measures bias through word embedding distances in semantic vector space. given a set of protected group (e.g. "woman") and two polar attribute sets (e.g. pleasant words vs unpleasant words), ceat measures the effect size of distance difference between the protected group and the two attribute sets. a lower effect size score indicates the protected group is closer to the negative polar of the attribute (e.g. unpleasant) and thus probably more correlated with toxic words. intrinsic: increased log probability score (ilps) @xcite measures bias on word associations between protected groups and attributes with templates such as "the [group] @xmath0." it computes the increase in probability of the attribute (e.g. "happy") for @xmath1.g. "woman"). a higher ilps score means the protected group is more correlated with the attribute, thus a higher ilps score with unpleasant words indicate the protected group is more correlated with negative sentiment. intrinsic: stereoset @xcite ) measures bias in language model scoring of stereotyped sentences over anti-stereotyped sentences. its dataset consists of minimal pair stereotyped and anti-stereotyped sentence pairs for each protected group. we only focus on their intrasentence pairs, where the sentences in each pair are only different in the attributes (e.g. "the iranian man might be a terrorist individual" and "the iranian man might be a hardworking individual" is a sentence pair for iranian group). the stereotype score for each protected group is computed as the proportion of pairs where the stereotyped sentences has a higher pseudo loglikelihood than its antistereotypical counterpart. extrinsic: jigsaw toxicity (toxd) (jigsaw, 2019) measures bias in toxicity detection systems that covers multiple protected groups. the fairness notion is defined by equalized odds, which minimizes differences in false positive rate (fpr) to ensure that text containing mentions of any one group is not being unjustly mislabelled as toxic. this is important for the classifiers to be able to detect toxicity in content containing identifiers across all protected groups, while not silencing any one. extrinsic: hatexplain (hatex) @xcite measures bias in hate speech detection systems. while the original problem is cast as a multiclass classification problem (normal, offensive, toxic), we cast it as a binary problem (toxic, nontoxic) due to lack of consistency in what is labelled as offensive and/or toxic. similar to toxd, the measure of bias against a certain group is the false positive rate on examples with group mentions. extrinsic: bold @xcite is a dataset that measures bias in language generation that consist of wikipedia-sourced natural prompts. given a prompt containing direct or indirect mentions of a protected group, bold evaluates the quality of the sentences finished by the language model. we focus on the sentiment (b-sent) metric for sentiment, toxicity (b-tox) metric for toxicity, and regard (b-regard) metric for stereotype. additionally, for stereotype, we train a stereotype classifier by finetuning the bert model with stere-oset @xcite , crows-pairs @xcite , and social bias frames @xcite datasets, and use this classifier to evaluate bold generations on stereotype (b-stereo) 3 . the bias score for each protected group is calculated as the average toxicity, sentiment, regard, and stereotype score on the generations from the prompts with that protected group. ## correlation between metrics experiment setup we conduct a study on gender, race, and religion domains (see the appendix a for the list of protected groups on each domain). we conduct correlation analysis on the variance of group metric scores across protected groups, as it captures score disparities across protected groups for each domain. for example, for @xmath2, we define s @xmath3, s white , s black , ...). a less-biased model would have smaller variance score. thus, if two metrics are correlated, we would see a positive correlation, as reducing the disparity between groups in one metric, as measured by variance would reduce that in the other. we evaluate 19 popular pre-trained language models foot_1 . these models consist of albert @xcite 5 . for intrinsic metrics, we simply measure the corresponding metric scores on the language models foot_3 . for extrinsic metrics, we gender race religion ceat ilps ceat ilps ceat ilps toxd -0.12 0.26 -0.06 -0.37 0.28 -0.37 hatex -0.12 0.10 -0.05 0.73 0.23 -0.38 b-tox 0.21 -0.28 0.41 -0.34 0.19 -0.53 b-sent -0.03 0.54 0.67 0.30 -0.42 -0.58 fine-tune language models for classification-based tasks 7 , and either sample in an autoregressive manner for autoregressive language models, or use random masking-based generation for mlm-based models (wang and cho, 2019) following the bold paper, for generation-based tasks 8 . for each intrinsic and extrinsic metric pair, we take the intrinsic and extrinsic scores for each et al. (2021) for the religion domain. 7 for classification tasks, we use the hyper-parameters for fine-tuning language models on tasks for extrinsic metrics that achieve close to state-of-the-art f1 score (see appendix c). correlation results in both toxicity and sentiment, we see that there are statistically negative correlations for the religion domain, which we investigate in section 3.2. for stereotype, ## ablation study there are many factors at play in fairness evaluation processes, such as notion of bias measured, choice of protected groups, quality of the testing data, and confounding factors in the models used to compute metrics themselves. in this section, we conduct careful analysis to explore why extrinsic and intrinsic metrics are not always correlated. ## conclusion we present a study on intrinsic and extrinsic fairness metrics in contextualized word embeddings. our experiments highlight the importance of alignment in the evaluation dataset, protected groups, and the quality of the evaluation dataset when it comes to aligning intrinsic and extrinsic metrics. based on this study, we impart three takeaways for researchers and developers. first, we cannot assume that an improvement in language model fairness will fix bias in downstream systems. secondly, when choosing fairness metrics to evaluate and optimize for, it is important to choose a metric that is closest to the downstream application. if that is not possible for all downstream applications, then it is important to align intrinsic metrics to the extrinsic use cases. finally, it is important to mitigate factors that may lead to bias in the metric computation itself, including noise in evaluation datasets, models used in metric computation, and inference experiment configurations such as decoding temperature for text generation.