On Cross-Dataset Generalization in Automatic Detection of Online Abuse (2020)

Abstract: NLP research has attained high performances in abusive language detection as a supervised classification task. While in research settings, training and test datasets are usually obtained from similar data samples, in practice systems are often applied on data that are different from the training set in topic and class distributions. Also, the ambiguity in class definitions inherited in this task aggravates the discrepancies between source and target datasets. We explore the topic bias and the task formulation bias in cross-dataset generalization. We show that the benign examples in the Wikipedia Detox dataset are biased towards platform-specific topics. We identify these examples using unsupervised topic modeling and manual inspection of topics’ keywords. Removing these topics increases cross-dataset generalization, without reducing in-domain classification performance. For a robust dataset design, we suggest applying inexpensive unsupervised methods to inspect the collected data and downsize the non-generalizable content before manually annotating for class labels.

## introduction the nlp research community has devoted significant efforts to support the safety and inclusiveness of online discussion forums by developing automatic systems to detect hurtful, derogatory or obscene utterances. most of these systems are based on supervised machine learning techniques, and require annotated data. several publicly available datasets have been created for the task @xcite . however, due to the ambiguities in the task definition and complexities of data collection, cross-dataset generalizability remains a challenging and understudied issue of online abuse detection. existing datasets differ in the considered types of offensive behaviour and annotation schemes, data sources and data collection methods. there is no agreed-upon definition of harmful online behaviour yet. several terms have been used to refer to the general concept of harmful online behavior, including toxicity @xcite , hate speech @xcite , offensive @xcite and abusive language @xcite . still, in practice, every dataset only focuses on a narrow range of subtypes of such behaviours and a single online platform @xcite . for example, @xcite annotated tweets for three categories, racist, offensive but not racist and clean, and @xcite collected discussions from yahoo! finance news and applied a binary annotation scheme of abusive versus clean. further, since pure random sampling usually results in small proportions of offensive examples @xcite , various sampling techniques are often employed. @xcite used words and phrases frequently found in offensive messages to search for potential abusive tweets. @xcite and @xcite started from random sampling, then boosted the abusive part of the datasets using specific search procedures. hosseinmardi et al. @xcite used snowballing to collect abusive posts on instagram. due to this variability in category definitions and data collection techniques, a system trained on a particular dataset is prone to overfitting to the specific characteristics of that dataset. as a result, although models tend to perform well in cross-validation evaluation on one dataset, the cross-dataset generalizability remains low @xcite . in this work, we investigate the impact of two types of biases originating from source data that can emerge in a cross-domain application of models: 1) task formulation bias (discrepancy in class definitions and annotation between the training and test sets) and 2) selection bias (discrepancy in the topic and class distributions between the training and test sets). further, we suggest topicbased dataset pruning as a method of mitigating selection bias to increase generalizability. this approach is different from domain adaptation techniques based on data selection @xcite in that we apply an unsupervised topic modeling method for topic discovery without using the class labels. we show that some topics are more generalizable than others. the topics that are specific to the training dataset lead to overfitting and, therefore, lower generalizability. excluding or down-sampling instances associated with such topics before the expensive annotation step can substantially reduce the annotation costs. we focus on the wikipedia detox or wikidataset, (an extension of the dataset by @xcite ), collected from english wikipedia talk pages and annotated for toxicity. to explore the generalizability of the models trained on this dataset, we create an out-of-domain test set comprising various types of abusive behaviours by combining two existing datasets, namely waseemdataset @xcite and fountadataset @xcite , both collected from twitter. our main contributions are as follows: ## biases originating from source data we focus on two types of biases originated from source data: task formulation and selection bias. task formulation bias: in commercial applications, the definitions of offensive language heavily rely on community norms and context and, therefore, are imprecise, application-dependent, and constantly evolving @xcite . similarly in nlp research, despite having clear overlaps, offensive class definitions vary significantly from one study to another. for example, the toxic class in the wiki-dataset refers to aggressive or disrespectful utterances that would likely make participants leave the discussion. this definition of toxic language includes some aspects of racism, sexism and hateful behaviour. still, as highlighted by @xcite , identity-based abuse is fundamentally different from general toxic behavior. therefore, the toxic class definition used in the wiki-dataset differs in its scope from the abuserelated categories as defined in the waseem-dataset and founta-dataset. @xcite converted various category sets to binary (offensive vs. normal) and demonstrated that a system trained on one dataset can identify other forms of abuse to some extent. we use the same methodology and examine different offensive categories in outof-domain test sets to explore the deviation in a system's performance caused by the differences in the task definitions. regardless of the task formulation, abusive language can be divided into explicit and implicit @xcite . explicit abuse refers to utterances that include obscene and offensive expressions, such as stupid or scum, even though not all utterances that include obscene expressions are considered abusive in all contexts. implicit abuse refers to more subtle harmful behaviours, such as stereotyping and micro-aggression. explicit abuse is usually easier to detect by human annotators and automatic systems. also, explicit abuse is more transferable between datasets as it is part of many definitions of online abuse, including personal attacks, hate speech, and identity-based abuse. the exact definition of implicit abuse, on the other hand, can substantially vary between task formulations as it is much dependent on the context, the author and the receiver of an utterance @xcite . selection bias: selection (or sampling) bias emerge when source data, on which the model is trained, is not representative of target data, on which the model is applied @xcite . we focus on two data characteristics affecting selection bias: topic distribution and class distribution. in practice, every dataset covers a limited number of topics, and the topic distributions depend on many factors, including the source of data, the search mechanism and the timing of the data collection. for example, our source dataset, wikidataset, consists of wikipedia talk pages dating from 2004-2015. on the other hand, one of the sources of our target dataset, waseem-dataset, consists of tweets collected using terms and references to specific entities that frequently occur in tweets expressing hate speech. as a result of its sampling strategy, waseem-dataset includes many tweets on the topic of 'women in sports'. @xcite showed that different data sampling methods result in various distributions of topics, which affects the generalizability of trained classifiers, especially in the case of implicit abuse detection. unlike explicit abuse, implicitly abusive behaviour comes in a variety of semantic and syntactic forms. to train a generalizable classifier, one requires a training dataset that covers a broad range of topics, each with a good representation of offensive examples. we continue this line of work and investigate the impact of topic bias on cross-dataset generalizability by identifying and changing the distribution of topics in controlled experiments. the amount of online abuse on mainstream platforms varies greatly but is always very low. founta et al. (2018) found that abusive tweets form 0.1% to 3% of randomly collected datasets. @xcite showed that depending on the platform the prevalence of abusive language can range between 0.001% and 8%. despite various data sampling strategies aimed at increasing the proportion of offensive instances, the class imbalance (the difference in class sizes) in available datasets is often severe. when trained on highly imbalanced data, most statistical machine learning methods exhibit a bias towards the majority class, and their performance on a minority class, usually the class of interest, suffers. a number of techniques have been proposed to address class imbalance in data, including data re-sampling, cost-sensitive learning, and neural network specific learning algorithms @xcite @xcite . in practice, simple re-sampling techniques, such as down-sampling of over-represented classes, often improve the overall performance of the classifier @xcite . however, re-sampling techniques might lead to overfitting to one of the classes causing a trade-off between true positive and true negative rates. when aggregated in an averaged metric such as f-score, this trade-off is usually overlooked. ## datasets we exploit three large-scale, publicly available english datasets frequently used for the task of online abuse detection. our main dataset, wiki-dataset @xcite , is used as a training set. the out-of-domain test set is obtained by combining the other two datasets, founta-dataset @xcite and @xcite . training set: the wiki-dataset includes 160k comments collected from english wikipedia discussions and annotated for toxic and normal, through crowd-sourcing foot_0 . every comment is annotated by 10 workers, and the final label is obtained through majority voting. the class toxic comprises rude, hateful, aggressive, disrespectful or unreasonable comments that are likely to make a person leave a conversation foot_1 . the dataset consists of randomly collected comments and comments made by users blocked for violating wikipedia's policies to augment the proportion of toxic texts. this dataset contains 15,362 instances of toxic and 144,324 normal texts. ## topic analysis of the wiki-dataset we start by exploring the content of the wikidataset through topic modeling. we train a topic model using the online latent dirichlet allocation (olda) algorithm @xcite as implemented in the gensim library @xcite with the default parameters. latent dirichlet allocation (lda) @xcite is a baysian probabilistic model of a collection of texts. each text is assumed to be generated from a multi- ## topic distribution of the test set we apply the lda topic model trained on the wikidataset as described in section 4 to the out-of-domain test set. as before, each textual instance is assigned a single topic that has the highest probability. observe that category 3 is the least represented category of topics across all classes, except for the normal class in the wiki-dataset. specifically, there is a significant deviation in the topic distribution between the wiki-normal and the founta-normal classes. this deviation can be explained by the difference in data sources. normal conversations on twitter are more likely to be about general concepts covered in category 1 or identity-related topics covered in category 2 than the specific topics such as writing and editing in category 3. other than waseem-racist, which has 67% overlap with category 2, all types of offensive behaviour in the three datasets have more overlap with the general topics (category 1) than identity-related topics (category 2). for example, for the waseem-sexist, 50% of instances fall under category 1, 35% under category 2 and 15% under category 3. topic 1, which is a mixture of general topics, is the dominant topic among the waseem-sexist tweets. out of the topics in category 2, most of the sexist tweets are matched to topic 2 (focused on sexism and homophobia) and topic 12 (general personal insults). test subset dataset/class all cat.#1 cat.#2 cat.#3 out-of-domain -toxic founta-abusive 0.94 0.94 0.96 0.91 founta-hateful 0.62 0.65 0.62 0.43 waseem-sexist 0.26 0.29 0.26 0.17 waseem-racist 0.35 0.37 0.36 0.20 out-of-domain -normal founta-normal 0.96 0.95 0.97 0.99 note that given the sizes of the positive and negative test classes, all other common metrics, such as various kinds of averaged f1-scores, can be calculated from the accuracies per class. in addition, we report macro-averaged f-score, weighted by the sizes of the negative and positive classes, to show the overall impact of the proposed method. results: the overall performance of the classifier on the out-of-domain test set is quite high: weighted macro-averaged f 1 = 0.90. however, when the test set is broken down into the 20 topics of the wiki-dataset and the accuracy is measured within the topics, the results vary greatly. for ex-ample, for the instances that fall under topic 14, the explicitly offensive topic, the f1-score is 0.99. for topic 15, a wikipedia-specific topic, the f1-score is 0.80. for the class founta-abusive, the classifier achieves 94% accuracy. 12% of the founta-abusive tweets fall under the explicitly offensive topic (topic 14), and those tweets are classified with a 100% accuracy. the accuracy score is highest on category 2 and lowest on category 3. for the founta-hateful class, the classifier recognizes 62% of the tweets correctly. the accuracy score is highest on category 1 and lowest on category 3. 8% of the founta-hateful tweets fall under the explicitly offensive topic (topic 14), and are classified with a 99% accuracy. for the founta-normal class, the classifier recognizes 96% of the tweets correctly. unlike the founta-abusive and founta-hateful class, for the founta-normal class, the highest accuracy is achieved on category 3. 0.1% of the founta-normal tweets fall under the explicitly offensive topic, and only 26% of them are classified correctly. the accuracy of the classifier on the waseem-sexist and waseem-racist classes is 0.26 and 0.35, respectively. this indicates that the wiki-dataset, annotated for toxicity, is not well suited for detecting sexist or racist tweets. this observation could be explained by the fact that none of the coherent topics extracted from the wiki-dataset is associated strongly with sexism or racism. nevertheless, the tweets that fall under the explicit abuse topic (topic 14) are recognized with a 100% accuracy. topic 8, which contains abuse mostly directed towards jewish and muslim people, is the most dominant topic in the racist class (32% of the class) and the accuracy score on this topic is the highest, after the explicitly offensive topic. the racist class overlaps the least with category 3 (see ## impact of data size, class and topic distribution on generalizability our goal is to measure the impact of various topics on generalization. however, modifying the topic distribution will impact the class distribution and data size. to control for this, we first analyze the impact of class distribution and data size on the classifier's performance. then, we study the effect of topic distribution by limiting the training data to different topic categories. impact of class distribution: the class distribution in the wiki-dataset is fairly imbalanced; the ratio of the size of wiki-toxic to wiki-normal is 1:10. class imbalance can lead to poor predictive performance on minority classes, as most of the learning algorithms are developed with the assumption of the balanced class distribution. to investigate the impact of the class distribution on generalization, we keep all the wiki-toxic instances and randomly sample the wiki-normal class to build the training sets with various ratios of toxic to normal instances. figure 1 shows the classifier's accuracy on the test classes when trained on subsets with different class distributions. observe that with the increase of the wiki-normal class size in the training dataset, the accuracy on all offensive test classes decreases while the accuracy on the founta-normal class increases. the classifier assigns more instances to the the normal class resulting in a lower true positive (accuracy on the offensive classes) and a higher true negative (accuracy on the normal class) rates. the drop in accuracy is significant for the waseem-sexist, waseem-racist and waseem-hateful classes and relatively minor for the founta-abusive class. note that the impact of the class distribution is not reflected in the overall f1-score. the classifier trained on a balanced data subset (with class size ratio of 1:1) reaches 0.896 weighted-averaged f1-score, which is very close to the f1-score of 0.899 resulted from training on the full dataset with the 1:10 class size ratio. however, in practice, the designers of such systems need to decide on the preferred class distribution depending on the distribution of classes in the test environment and the significance of the consequences of the false positive and false negative outcomes. impact of dataset size: to investigate the impact of the size of the training set, we fix the class ratio at 1:1 and compare the classifier's performance when trained on data subsets of different sizes. we randomly select subsets from the wiki-dataset with sizes of 10k (5k toxic and 5k normal instances) and 30k (15k toxic and 15k normal instances). each experiment is repeated 5 times, and the averaged results are presented in figure 2 . the height of the box shows the standard deviation of accuracies. observe that the average accuracies remain unchanged when the dataset's size triples at the same class balance ratio. this finding contrasts with the general assumption that more training data results in a higher classification performance. impact of topics: in order to measure the impact of topics covered in the training dataset, we compare the classifier's performance when trained on only one of the three categories of topics described in section 4. to control for the effect of class balance and dataset size, we run the experiments for two cases of toxic-to-normal ratios, 3k-3k and 3k-27k. each experiment is repeated 5 times, and the average accuracy per class is reported in figure 3 . for both cases of class size ratios, shown in figures 3a and 3b, we notice that the classifier trained on instances belonging to category 3 reaches higher accuracies on the offensive classes, but a significantly lower accuracy on the founta-normal class. the benign part of category 3 is overwhelmed by wikipedia-specific examples. therefore, utterances dissimilar to these topics are labelled as toxic, leading to a high accuracy on the toxic classes and a low accuracy on the normal class. this is an example of the negative impact of topic bias on the detection of offensive utterances. in contrast, the classifiers trained on categories 1 and 2 perform comparably across test classes. the classifier trained on category 2 is slightly more effective in recognizing founta-hateful utterances, especially when the training set is balanced. this observation can be explained by a better representation of identity-related hatred in category 2. ## removing platform-specific instances from the training set we showed that a classifier trained on instances from category 3 suffers a big loss in accuracy on the normal class. here, we investigate how the performance of a classifier trained on the full wikidataset changes when the category 3 instances (all or the benign part only) are removed from the training set. to assess the impact of removing wikipediaspecific examples on in-domain classification, we train a model on the training set of the wiki-dataset, with and without excluding category 3 instances, and evaluate it on the full test set of the wiki-dataset. we observe that the in-domain performance does not suffer from removing category 3 from the training data ( ## discussion in the task of online abuse detection, both false positive and false negative errors can lead to significant harm as one threatens the freedom of speech and ruins people's reputations, and the other ignores hurtful behaviour. although balancing the class sizes has been traditionally exploited when dealing with imbalanced datasets, we showed that balanced class sizes may lead to high misclassification of normal utterances while improving the true positive rates. this trade-off is not necessarily reflected in aggregated evaluation metrics such as f1-score but has important implications in real-life applications. we suggest evaluating each class (both positive and negative) separately taking into account the potential costs of different types of errors. furthermore, our analysis reveals that for generalizability, the size of the dataset is not as important as the class and topic distributions. we analyzed the impact of the topics included in the wiki-dataset and showed that mitigating the topic bias improves accuracy rates across all the out-of-domain positive and negative classes. our results suggest that the sheer amount of normal comments included in the training datasets might not be necessary and can even be harmful for generalization if the topic distribution of normal topics is skewed. when the classifier is trained on category 3 instances only (figure 3 ), the normal class is attributed to the over-represented topics, leading to high misclassification of normal texts or high false positive rates. in general, when collecting new datasets, texts can be inspected through topic modeling using simple heuristics (e.g., keep topics related to demographic groups often subjected to abuse) in an attempt to balance the distribution of various topics and possibly sub-sample over-represented and less generalizable topics (e.g., high volumes of messages related to an incident with a celebrity figure happened during the data collection time) before the expensive annotation step. ## conclusion our work highlights the importance of heuristic scrutinizing of topics in collected datasets before performing a laborious and expensive annotation. we suggest that unsupervised topic modeling and manual assessment of extracted topics can be used to mitigate the topic bias. in the case of the wikidataset, we showed that more than half of the dataset can be safely removed without affecting either the in-domain or the out-of-domain performance. for future work, we recommend that topic analysis, augmentation of topics associated with offensive vocabulary and targeted demographics, and filtering of non-generalizable topics should be applied iteratively during data collection.