Fine-Grained Fairness Analysis of Abusive Language Detection Systems with C heck L ist (2021)

Abstract: Current abusive language detection systems have demonstrated unintended bias towards sensitive features such as nationality or gender. This is a crucial issue, which may harm minorities and underrepresented groups if such systems were integrated in real-world applications. In this paper, we create ad hoc tests through the CheckList tool (Ribeiro et al., 2020) to detect biases within abusive language classifiers for English. We compare the behaviour of two BERT-based models, one trained on a generic hate speech dataset and the other on a dataset for misogyny detection. Our evaluation shows that, although BERT-based classifiers achieve high accuracy levels on a variety of natural language processing tasks, they perform very poorly as regards fairness and bias, in particular on samples involving implicit stereotypes, expressions of hate towards minorities and protected attributes such as race or sexual orientation. We release both the notebooks implemented to extend the Fairness tests and the synthetic datasets usable to evaluate systems bias independently of CheckList.

## introduction at every stage of a supervised learning process, biases can arise and be introduced in the pipeline, ultimately leading to harm @xcite . when it comes to systems whose goal is to automatically detect abusive language, this issue becomes particularly serious, since unintended bias towards sensitive attributes such as gender, sexual orientation or nationality can harm underrepresented groups. @xcite , for example, show that annotators tend to label messages in afro-american english more frequently than when annotating other messages, which could lead to the training of a system reproducing the same kind of bias. the role of the datasets used to train these models is crucial: as pointed out by @xcite , there may be multiple reasons why a dataset is biased, e.g. due to skewed sampling strategies, prevalence of a specific subject (topic bias) or of content written by a specific author (author bias). mitigation strategies may involve assessing which terms are frequent in the presence of certain labels and implementing techniques to balance the data by including neutral samples containing those same terms to prevent the model from learning inaccurate correlations @xcite . furthermore, it is important to distinguish between different types of hatred, depending on the target group addressed: for example, misogynistic expressions show different linguistic peculiarities than racist ones. it is therefore crucial to create specialised datasets addressing different phenomena of abusive language, so that systems can be tuned to the complex and nuanced scenario of online speech. given the sensitive context in which abusive language detection systems are deployed, a robust value-oriented evaluation of the model's fairness is necessary, in order to assess unintended biases and avoid, as far as possible, explicit harm or the amplification of pre-existing social biases. however, this bias-assessment process is complicated by the partial effectiveness of proposed methods that only work with certain definitions of bias and fairness, as well as by the limited availability of recognised benchmark datasets @xcite . concerning the different definitions of fairness, they have been collected and organised both in @xcite and @xcite , with the awareness that a single definition is not sufficient to address the multi-faceted problem of fairness in its entirety. in this work, we adopt a definition for fairness that is strongly contextual to abusive language detection. we define unfairness as the sensitivity of an abusive language detection classifier with respect to the presence in the record to be classified of entities belonging to protected groups or minorities. specifically, a classifier is considered unfair or biased if the prediction changes according to the identities present, i.e. in similar sentences, the degree of hate is increased if terms such as white or straight are replaced by adjectives such as black or non-binary, revealing imbalances, possibly resulting from skewed and unrepresentative training data. fairness, on the other hand, is defined as the behaviour of producing similar predictions for similar protected mentions, i.e. regardless of the specific value assumed by sensitive attributes like race and gender, without disadvantaging minorities or amplifying pre-existing social prejudices. we deploy the checklist tool (ribeiro et al., 2020), which was originally created to evaluate general linguistic capabilities of nlp models, extending it to test fairness of abusive language detection systems. embracing checklist systematic framework, we create tests from hand-coded templates, reproducing stereotyped opinions and social biases, such as sexism and racism. the aim is to assess the performances of these models identifying the most frequent errors and detecting a range of unintended biases towards sensitive categories and topics. this last objective is motivated by evidence @xcite that nlp systems tend, in certain contexts, to rely for the classification on identity terms and sensitive attributes, as well as to generalize misleading correlations learnt from training datasets. as ultimate goal, the analysis of the failures could therefore lead to a general overview of the models' fairness: the ideal outcome would be to establish a proactive pipeline that allows the improvement of the systems, having highlighted the shortages through checklist ad hoc synthetic testing. to the best of our knowledge, there has not yet been any work carried out with checklist in this research direction. ## related work several tools and approaches have been proposed to identify the most frequent errors done by nlp tools. for example, errudite @xcite is a tool that allows interactive error analysis through counterfactuals generation, but it is limited to the tasks of question answering and visual question answering. textattack @xcite ) -which, among other packages, deploys checklist -is a modelagnostic framework useful for the expansion of the datasets and the increase of models robustness through adversarial attacks. compared to check-list, however, it is more complicated to handle and deploy for users with little nlp skills. an interesting aspect is that textattack includes in the package the so-called "recipes", i.e. attacks from the literature ready to run, that build a common ground for the assessment and comparison of models' performances. as outlined in @xcite , some methods to identify errors by nlp systems are taskspecific, such as @xcite or @xcite , while others focus on particular nlp components such as word embeddings, as in @xcite or @xcite . compared to existing approaches, one of check-list's major strengths lies in including the testing phase within a comprehensive framework. the evaluation, conducted through adaptable templates and a range of relevant linguistic capabilities, is on one hand more granular than overall measures such as accuracy; on the other hand it is more versatile, because it leaves liberty to the developer to enrich and expand the tests within new and more suitable capabilities, depending on the task and model under consideration. on the topic of fairness and biases, @xcite ) conduct an in-depth discussion on nlp works dealing with ethical issues and challenges in automatic abusive language detection. among others, a perspective analyzed is the principle of fairness and non-discrimination throughout every stage of supervised machine learning processes. a recent survey by @xcite also analyzes and criticizes the formalization of bias within nlp systems, revealing inconsistency, lack of normativity and common rationale in several works. furthermore, the visibility reached by corporate tools, such as ibm ai fairness 360 or amazon sagemaker clarify, which are designed and promoted by large it companies, raises several questions: is self-regulation right? what would be the advantages and risks of conducting independent external auditing? several metrics 1 , generic tools and python packages 2 are available. nevertheless, no consensus related to the above questions has been reached yet among the involved players. concerning existing datasets specifically designed to assess biases within machine learning models, @xcite list several of the widely used ones, which differ according to size, type of records (numerical, images, texts) and tackled domain (e.g. financial, facial recognition, etc.). the only language dataset cited is wino-bias, @xcite foot_2 also used in this work as a lexical resource, which pertains to the field of co-reference resolution. our contribution instead aims to broaden fairness evaluation, specifically testing biases in abusive language detection systems through checklist facilities. concerning abusive language detection, a number of approaches has been proposed to perform both coarse-grained (i.e. binary) and fine-grained classification. 87 systems participated in the last offenseval competition for english @xcite , which included a binary task on offensive language identification, one on offensive language categorization and another on target identification. as reported by the organisers, the majority of teams used some kind of pre-trained embeddings such as contextualized transformers @xcite and elmo @xcite embeddings. the most popular transformers were bert @xcite and roberta @xcite , which showed to achieve state-of-the-art results for english, especially when used in ensemble configurations. for this reason, we use bert also in the experiments presented in the following sections. ## introduction to checklist usually, the generalization capability of nlp models is evaluated based on the performance obtained on a held-out dataset, by measuring f1 or accuracy. this process, although widely adopted by the nlp community as a way to compare systems performances and approaches, lacks informativeness since it does not provide insights into how to improve the models through the analysis of errors. in order to tackle this issue, checklist (ribeiro et al., 2020) was developed as a comprehensive task-agnostic framework, inspired by behavioral testing, in order to encourage more robust checking and to facilitate the assessment of models' general linguistic capabilities. the package allows the generation of data through the construction of different ad hoc tests by generalizations from templates and lexicons, general-purpose perturbations, tests expectations on the labels and context-aware suggestions using roberta fill-ins @xcite as prompter for specific masked tokens. the tests created can be saved, shared and utilized for different systems. checklist includes three test types and a number of linguistic capabilities to be tested. the three types of tests are: concerning linguistic capabilities, checklist covers a number of aspects that are usually relevant when evaluating nlp systems, such as robustness, named entity recognition, temporal awareness of the models and negation. while we also evaluated these aspects, our main focus here is models fairness, which verifies that systems predictions do not change as a function of protected features. while the fairness capability already proposed in check-list involved the perturbation of sensitive attributes, namely expressions referring to gender, sexual orientation, nationality or religion, we first extend it by adding "professions" as protected attribute in order to assess whether predictions change if a male or a female assumes a specific job role. we then enrich the capability designing hand-coded templates, belonging to the mft test type, resulting from the exploration of representative constructions and stereotypes annotated in the social bias inference corpus @xcite . the resulting samples exemplify several sexist, racist and ableist comments and opinions: all of them are new aspects compared to the suites released by the authors @xcite . as described in the introduction, checklist provides built-in tools to assist users in the creation of tests. among others, wordnet allows the selection of synonyms, antonyms, hypernyms, etc. for a given expression. checklist's templates take shape from these sets of semantically related words. we develop a further extension of the tool by integrating sentiwordnet @xcite , a lexical resource in which wordnet synsets have been associated with a sentiment score (negative, neutral or positive). in this way, checklist can benefit from the sentiment-dimension of sentiword-net. indeed, during the development of templates and the perturbations of the records, sentiwordnet enables the selection of suitable linguistic substitutions for a given term, according to the label of the sentence to be created. an example: seeking a synonym that has a similar connotation as the adjective happy for the phrase "the girl is happy", the results returned include glad, with a positive denotations of 0.5. in this case, through sentiword-net, it is possible to select a synonym term with a similar polarity, in order to create variants of the original sentence that preserve a similar semantic content and to assess how the model behaves with slightly different terms. ## a suite for abusive language detection suites are objects designed by checklist authors @xcite ) that enable users to organise, combine and save sets of tests, in order to reuse them several times and to aggregate results (i.e. failure rates) in a single run. once a test is designed, it is added to the suite, specifying the test type (mft, inv or dir), a name, the language capability within which it is situated and a brief description. the suite will thus be composed of one or more capabilities, each of which is assessed through several tests. after the suite is created, it can be run to evaluate the output of a given classifier, provided that the system has been previously launched to label the records created for each test providing for each record a class and the respective probabilities. the results of the run of the suite are displayed through a visual and interactive summary, which reports misclassified samples and the various failure percentages obtained in each test (see fig. 1 for an example). the core of our work takes off from the notebooks released by checklist authors @xcite , specifically from the suite for the task of sentiment analysis foot_3 , that builds a series of tests consisting in tweets about airline companies. in order to target a different task, which relies on binary decisions, we modify all the templates adjusting them for the task of abusive language detection. our main contribution is the extension of the fairness capability, which we enrich with several tests addressing diverse abuse targets and dealing with different types of biases. ## evaluation in ## discussion and conclusions the approach that checklist proposes should complement the evaluation of nlp models carried out by applying standard metrics such as f1 and accuracy. indeed, in addition to the traditional held-out datasets, the creation of ad hoc examples, from the most basic ones to the most complex, contribute to highlight weaknesses that cannot be easily detected through large existing datasets. furthermore, checklist provides a way to explore the models' dynamics: through the analysis of the errors, we can infer which linguistic phenomena the system has not yet acquired from the data. however, in order to enable this fine-grained evaluation, several specific tests and templates should be created that, like in our case, may contain a small amount of examples because of the difficulty to create or retrieve a varied sample of records covering specific phenomena, e.g. feminist and ableist stereotypes. a significant drawback, closely related to check-list deployment on abusive language detection systems, concerns the difficulty of including and dealing with contextual information @xcite . sensitive real-world statements often acquire a different connotation w.r.t. the degree of hatred if a certain race, gender, or nationality is present, due to historical or social references @xcite . in our work, we temporarily avoid such risks using synthetic templates strongly polarized on the one hand towards offensiveness, on the other towards neutrality. perturbing real-world data would seriously require taking into account these nuances by implementing a more flexible and accurate inspection of prediction variations. although state-of-the-art models such as bertbased models achieve high accuracy levels on a variety of natural language processing tasks, including abusive language detection, we have shown through diverse tests that these systems perform very poorly concerning bias on samples involving implicit stereotypes and sensitive features such as gender or sexual orientation. whether these biases in bert-based systems emerge from the classification algorithm, the pretraining phase or the training data will have to be investigated and further explored in the future. as a preliminary analysis, our results show that training sets play a relevant role in this, as already highlighted in previous works @xcite . for some phenomena, such as body image stereotypes or feminism-related statements, different training sets make the classifier behave very differently, in a way that we were able to quantify through our approach. moreover, the notebooks through which we built the suite are made available and the tests are easily editable and adaptable to specific data or linguistic aspects to be investigated. a future direction of this work might be to expand the package integrating other linguistic resources, such as emotion or sentiment lexica. concerning linguistic capabilities, for fairness other stereotypes from a wider range of datasets could be more thoroughly explored and formalised into templates. it would be also interesting to analyse whether classification that takes into account the broader discourse context @xcite is less prone to biases. suites for other languages could be built as well, given that datasets for abusive language detection are available in many languages beyond english @xcite . as suggested in @xcite , proposing a contribution within the machine learning domain responsibly and consciously means foremost acknowledging our own biases. in particular, we are referring to the implementation of hand-coded templates, that we generalized within the check-list framework starting from real-user examples. the selection and the way in which the tests have been built certainly shaped the results. surely, this paper is not a complete or comprehensive work: for example, a direct interaction with the targeted users and the different stake-holders affected could have enriched the perspective and the insights retrieved. furthermore, it is important to be aware that any solely technological solution will be partial, as not considering the broader social issue that is the source of these biases means simplifying and "fixing" only on the surface @xcite . regardless, we strongly believe that abusive language classifiers need a robust value-sensitive evaluation, in order to assess unintended biases and avoid, as far as possible, explicit harm or the amplification of pre-existing social biases, trying to ultimately build systems that contributes in a beneficial way to the society and all its citizens.