Recalibrating classifiers for interpretable abusive content detection (2020)

Abstract: We investigate the use of machine learning classifiers for detecting online abuse in empirical research. We show that uncalibrated classifiers (i.e. where the ‘raw’ scores are used) align poorly with human evaluations. This limits their use for understanding the dynamics, patterns and prevalence of online abuse. We examine two widely used classifiers (created by Perspective and Davidson et al.) on a dataset of tweets directed against candidates in the UK’s 2017 general election. A Bayesian approach is presented to recalibrate the raw scores from the classifiers, using probabilistic programming and newly annotated data. We argue that interpretability evaluation and recalibration is integral to the application of abusive content classifiers.

## introduction computational tools for automatically detecting and categorizing abusive online content are now widely used for content moderation, to enforce and monitor regulatory and legal standards, and to study the dynamics of online abuse @xcite @xcite . these tools enable abusive content to be assessed quantitatively, scalably and efficiently. recent research has drawn attention to several biases with existing classifiers and the datasets they are trained on, such as racial biases @xcite , and evidence that they may be more attuned to detecting abuse against certain targets than others @xcite . other research shows that existing tools can be fooled by 'obfuscatory' content, in which small changes are made so that the abuse is 'masked', even though it is clearly discernible to humans (e.g. changing 'niggas' to 'n!gg@z') @xcite . equally, many classifiers struggle with contextual statements, irony, humour and con-tent that is non-abusive but 'incivil' @xcite . evaluating the explainability of classification systems for online abuse detection has become an important focus of research @xcite @xcite . explainable classifications can help to ensure systems are accountable, social biases are identified and addressed, and that model performance and generalisability is improved @xcite @xcite . in practice, explainability often requires statistical modelling to uncover the complex interactions between different input features that led to a result @xcite . it may even require complete rethinking of how classifiers are developed, given the difficulties of post-hoc rationalisation and the potential for poor explanations to confuse end users @xcite . a related but previously under-researched problem in abuse classification is whether the scores returned by classifiers meaningfully encode differences in the likelihood that content is abusive. this issue can be seen as a problem of interpretation. in contrast with explainability this does not involve showing why a particular classification is given but, rather, ensuring that the classification itself is presented in understandable terms @xcite . to our knowledge only one piece of research has investigated this problem. the perspective team at jigsaw calibrated the scores of their toxicity classifier using isotonic regression (perspectivescorenorm). the full details of the method are not published, but we understand that this calibration is primarily intended to ensure that, even as the production models are updated, the threshold of 0.8 remains a useful cutoff which for content moderation. ensuring that the scores from abusive content classifiers are interpretable would pose several benefits. first, the actual probabilities returned by models can be used for empirical analyses, reducing the information lost from only using a categorical label decided by a threshold. this is crucial in cases where a lot of content lies near the cutoff, and is the primary motivation behind this work. second, well-calibrated scores could help host platforms to curate and filter content. for instance, companies may only want their adverts near content that has a 99.99% chance of not being hateful. well-calibrated models could be used to ensure this. third, users who want to understand why their content has been taken down (or not) may want to review the scores. if they are poorly calibrated it could generate distrust and confusion. we investigate the use of machine learning classifiers for detecting and analysing online abuse in empirical research. we present three contributions. first, we show that the scores from uncalibrated abusive content classifiers align poorly with human evaluations. second, we present a method for recalibration which uses probabilistic programming, which also gives an indication of the confidence in the recalibration. third, we show that not using a calibrated classifier can severely impact empirical analysis through a case study of abuse directed against mps in the 2017 general election. all of our code and data is made available for other researchers to use. foot_0 ## research design we evaluate the toxicity classifier from perspective and the hate speech classifier provided by davidson et al. @xcite . both classifiers, and the datasets they were trained on, have been extensively researched in machine learning and computational social sciences (e.g. @xcite @xcite and the perspective classifier is widely used for content moderation. better understanding of their limitations and flaws will help to inform responsible use of them , and support development of better systems in the future. to examine the classifiers, we study tweets directed at candidates on twitter in the run up to the 2017 uk general election. we collected all mentions and replies to the 2,620 candidates from 16 may to 8 june 2017, creating a dataset of 8.93 million tweets. we apply both classifiers to the dataset, from which we construct two samples to be annotated. ## analysis of online abuse in the 2017 uk election to illustrate the importance of classifier calibration, we look at the temporal dynamics of abuse directed at two successful candidates in the 2017 uk election, ivan lewis and diane abbott. we show that recalibration is important for understanding the dynamics of abuse, such as when abuse 'events' take place, especially for candidates with a low volume of tweets. in this section we focus only on perspective's toxicity classifier. figure 4 plots the number of toxic tweets directed at lewis, using different thresholds (set at 0.5, 0.6, 0.7 and 0.8). as expected, the estimated prevalence of toxic tweets directed at lewis depends on where this arbitrary threshold is set-he received only two tweets with @xmath0.8, but 10 with @xmath1.7, a 5-fold increase. more concerningly, the thresholds give a very different view of when he receives abuse. in our dataset, this is a problem for all candidates who receive few tweets. more broadly, this problem will occur for any segments of a dataset (e.g. groups, individuals or time periods) which have few entries. an alternative to applying a threshold is to directly analyse the classification probabilities. this is far more desirable as it is less biased and leads to less information loss. it can be achieved by summing the toxicity probabilities within each unit of time (e.g. every hour). the principle behind this is that if there are 50 tweets each with 0.2 probability of toxicity then it should be likely that ∼10 will be toxic -and summing the probabilities best capture this. however, as the second panel of figure 4 shows, the uncalibrated scores substantially overemphasize non-toxic tweets, and using them for this purpose would inflate the estimated prevalence of abuse. the probabilities can only be used if the classifier is calibrated, allowing for far more flexible and insightful social scientific analysis. lewis received only 3,700 tweets during the run up to the 2017 election. in contrast, diane abbott (fig. 5 ) received 126,000. for candidates that receive many tweets, such as abbot, the recalibrated probabilities show similar dynamics compared with using a 0.8 threshold (recommended by perspective) and thus could be a reasonable choice (although they would still be biased by exactly where the threshold is set). however, for smaller samples this would lead to far less reliable results and could severely distort analyses. ## discussion abusive content classifiers are increasingly being used for empirical analysis. yet we show that they should be deployed with caution as their scores are often not interpretable. although they are usually on an ordinal scale (i.e. a higher value means there is a higher chance of abuse), they are not interval (i.e. a score that is twice as great is twice as likely). if the scores do not meaningfully encode differences then content with a score that is twice as high is not necessarily twice as likely to be abusive-nor can it be interpreted as the 'strength' of abuse is twice as great. the probabilistic programming method we have presented addresses this problem, ensuring that classifiers better reflect human interpretations. note that this procedure does not improve the 'performance' of classifiers, as measured by metrics such as auroc, but is important because it makes them far more useable. we propose that evaluation of interpretability (and recalibration) should be integral to abusive content classifier creation and application. one simple way of ensuring this is: (1) researchers apply their chosen classifier to the dataset they are analysing, (2) uniformly sample across the probabilities, (3) annotate the content based on the original guidelines, (4) evaluate the classifier scores using probablistic programming and (5) recalibrate them as needed. this process could be used for any similar nlp classification task, such as classification of incivil or aggressive language.