Your fairness may vary: Pretrained language model fairness in toxic text classification (2022)

Abstract: The popularity of pretrained language models in natural language processing systems calls for a careful evaluation of such models in down-stream tasks, which have a higher potential for societal impact. The evaluation of such systems usually focuses on accuracy measures. Our findings in this paper call for attention to be paid to fairness measures as well. Through the analysis of more than a dozen pretrained language models of varying sizes on two toxic text classification tasks (English), we demonstrate that focusing on accuracy measures alone can lead to models with wide variation in fairness characteristics. Specifically, we observe that fairness can vary even more than accuracy with increasing training data size and different random initializations. At the same time, we find that little of the fairness variation is explained by model size, despite claims in the literature. To improve model fairness without retraining, we show that two post-processing methods developed for structured, tabular data can be successfully applied to a range of pretrained language models. Warning: This paper contains samples of offensive text.

## introduction pre-trained, bidirectional language models @xcite @xcite @xcite foot_0 have revolutionized natural language processing (nlp) research. lms have provided a route to significant performance increases in several nlp tasks as demonstrated by nlp leaderboards @xcite @xcite . more importantly, lms have been applied to practical problems, leading to improved results for web search @xcite and have become an asset in fields such as medical evidence inference @xcite and chemistry @xcite . while the progress in nlp tasks due to lms is clear, the reasons behind this success are not as well understood @xcite , and there are also important downsides. in particular, several studies have documented the bias of lms @xcite @xcite @xcite and others discuss potential societal harms @xcite for individuals or groups. we use the term bias to refer to systematic disparity in representation or outcomes for individuals based on their membership in certain protected groups such as gender, race, and ethnicity. in this work, we focus on one important application of fine-tuned lms, toxic text classification. text toxicity predictors are already used in deployed systems @xcite and they are a crucial component for content moderation since online harassment is on the rise @xcite . in downstream applications such as toxic text classification, it is important to examine the behavior of lms in terms of measures other than task-specific accuracy. this provides a more holistic understanding of model performance and appropriate uses of lms for these tasks. as a first step toward this goal, we provide herein an empirical characterization of lms for the task of toxic text classification using a combination of accuracy and bias measures, and study two post-processing methods for bias mitigation that have proved successful for structured, tabular data. for assessing bias, in this paper, we focus on group fairness, which we explain in section 2 as it applies in general in machine learning, and discuss what it means in the context of nlp tasks in the same section. the implications of measuring group fairness for the toxicity classification task studied in this paper are described in section 3. one aspect of lms that is hard to ignore is the increase in their size, as measured by the number of parameters in their architectures. in general, larger lms seem to perform better on nlp tasks as they have the capacity to capture more complex correlations present in the training data. @xcite claim that this same property may also lead to more pronounced biases in their predictions, as the large data that lms are trained on is not curated. on the other hand, for image classification models that use large neural networks, @xcite discuss how model pruning can lead to more biased predictions. in this work, we consider a wide variety of model architectures and sizes. we acknowledge that size is relative and what we consider large in this paper may not be considered as such in a different context. we address the following questions regarding the effect of various factors on model performance: we study the accuracy-fairness relationship in more than a dozen fine-tuned lms for two different datasets that deal with prediction of text toxicity. the key contributions of our analysis are: our observations strengthen the chorus of recent work addressing bias mitigation in nlp in calling for a careful empirical analysis of fairness with fine-tuned lms in the context of their application. to allow group fairness analysis, annotations of group membership are preferred and sometimes required, and, thus, we urge the research community to include protected group annotations in datasets to enable extrinsic fairness evaluations that are as close as possible to the point of deployment. ## methodology we are interested in studying how group fairness varies across different fine-tuned lms for binary classification. we choose to focus on text toxicity as the prediction task. due to an increase in online harassment @xcite and the potential of both propagating harmful stereotypes of minority groups and/or inadvertently reducing their voices, the task of predicting toxicity in text has received increased attention in recent years @xcite . while we acknowledge that text toxicity presents different complex nuances (e.g., offensive text, harassment, hate speech), we focus on a binary task formulation. we adopt the definition of toxicity described in @xcite as "anything that is rude, disrespectful, or unreasonable that would make someone want to leave a conversation". ## bias mitigation post-processing we investigated the use of post-processing methods to mitigate violations of equalized odds. by postprocessing, we mean methods that operate only on the outputs of the fine-tuned lms and do not modify the models themselves foot_1 . the ability to avoid retraining models is a major advantage of post-processing due to the large computational cost of fine-tuning lms. post-processing also targets unfairness at a point closest to deployment and hence can have a direct impact on downstream operations that use the model predictions. hardt, price, srebro (2016) (hps): the first post-processing method that we consider is by @xcite (abbreviated hps, using the last names of the authors), who were the original pro-posers of the equalized odds criterion for fairness. we used the open-source implementation of their method from @xcite , which postprocesses binary predictions to satisfy eo while minimizing classification loss. while this method is effective in enforcing eo, one limitation is that it does not offer a trade-off between minimizing the deviation from eo and reducing the loss in accuracy. fair score transformer (fst): we study the fst method of @xcite , in part to provide the above-mentioned trade-off, and in part because it is a recent post-processing method shown to be competitive with several other methods (including in-processing). fst takes predicted probabilities (referred to as scores) as input and post-processes them to satisfy a fairness criterion. we choose generalized equalized odds (geo), a score-based variant of eo, as the fairness criterion and then threshold the output score to produce a binary prediction. the application of fst required attention to three issues: 1) its ability to work with input scores that may not be calibrated probabilities; 2) the choice of fairness parameter , which bounds the allowed geo on the data used to fit fst; 3) the choice of binary classification threshold t. we consider a range of and t values to explore the trade-off between eo and accuracy. due to numerical instability of the fst implementation in the original paper (occasional non-convergence in reasonable time for the jigsaw dataset), we obtained a closed-form solution for one step in the optimization that leads to a more efficient implementation, running in minutes for all models and all datasets considered. more details on this implementation and the tuning of the parameters can be found in appendix a.3. ## limitations in our study, we covered a series of different models that varied in network architecture, size as number of parameters, training procedures, and pretraining data. as we did not keep any of the elements constant (e.g., architecture) while varying the rest (e.g., pretraining data, size, training procedure), it is hard to draw insights on how each individual element affects the fairness of the resulting prediction outcomes. we would like to emphasize that identifying toxic text is not an easy task, not even for humans. as such, we expect the datasets to be noisy and contain samples that are not annotated correctly. upon manual inspection, we could identify some samples for which we did not agree with their labels. motivated by this observation, we started looking into understanding the quality of datasets used in toxic text prediction @xcite . as a consequence, while we expect the trends shown in this paper to hold, the actual absolute numbers may vary with datasets/tasks. more observations and limitations can be found in section 8. ## conclusions in this work, we addressed the following research questions for language models: how do model size, training size, random seeds affect the relationship between performance and fairness (as measured by equalized odds)? can post-processing methods for bias mitigation lead to better operating points for both accuracy and fairness? we find these questions important in the context of the ethics of using language models in text toxicity prediction, in particular, and in nlp research, in general. we presented a comprehensive study of language models and their performance/fairness relationship. we chose several models to cover different sizes and different architectures. while we did not consider some of the largest recent models available, we believe we have experimented with a wide variety of models that have been discussed well in the literature. we hope that this study can drive the following point across: we cannot make a blanket statement on the fairness of language models with respect to their size or architecture, while training factors such as data size and random seeds can make a large difference. this makes it all the more important for researchers/practitioners to make fairness an integral part of the performance evaluation of language models.