Generalisability of Topic Models in Cross-corpora Abusive Language Detection (2021)

Abstract: Rapidly changing social media content calls for robust and generalisable abuse detection models. However, the state-of-the-art supervised models display degraded performance when they are evaluated on abusive comments that differ from the training corpus. We investigate if the performance of supervised models for cross-corpora abuse detection can be improved by incorporating additional information from topic models, as the latter can infer the latent topic mixtures from unseen samples. In particular, we combine topical information with representations from a model tuned for classifying abusive comments. Our performance analysis reveals that topic models are able to capture abuse-related topics that can transfer across corpora, and result in improved generalisability.

## introduction with the exponentially increased use of social networking platforms, concerns on abusive language has increased at an alarming rate. such language is described as hurtful, toxic, or obscene, and targets individuals or a larger group based on common societal characteristics such as race, religion, ethnicity, gender, etc. the increased spread of such content hampers free speech as it can potentially discourage users from expressing themselves without fear, and intimidate them into leaving the conversation. considering variations of online abuse, toxicity, hate speech, and offensive language as abusive language, this work addresses the detection of abusive versus non-abusive comments. automatic detection of abuse is challenging as there are problems of changing linguistic traits, subtle forms of abuse, amongst others @xcite . moreover, the performance of models trained for abuse detection are found to degrade considerably, when they encounter abusive comments that differ from the training corpus @xcite @xcite . this is due to the varied sampling strategies used to build training corpus, topical and temporal shifts @xcite , and varied targets of abuse across corpora. since social media content changes rapidly, abusive language detection models with better generalisation can be more effective @xcite . to this end, a cross-corpora analysis and evaluation is important. topic models have been explored for generic cross-domain text classification @xcite @xcite , demonstrating better generalisability. moreover, they can be learnt in an unsupervised manner and can infer topic mixtures from unseen samples. this inspires us to exploit topic model representations for crosscorpora abuse detection. recently, @xcite have "retrained" bert @xcite over large-scale abusive reddit comments to provide the hatebert model which has displayed better generalisability in cross-corpora experiments. furthermore, @xcite show that combination of topic model and bert representations leads to better performance at semantic similarity task. taking these studies into account, we investigate if combining topic representation with contextualised hatebert representations can result in better generalisability in cross-corpora abuse detection. cross corpora evaluation on three common abusive language corpora supports and demonstrates the effectiveness of this approach. besides, we bring some insights into how the association of unseen comments to abusive topics obtained from original training data can help in cross-corpora abusive language detection. the rest of the paper is organised as follows: section 2 describes the architecture of the combination of topic model and hatebert. section 3 presents our experimental settings. an analysis of the results obtained is present in section 4, and section 5 concludes the paper. ## combining topic model and hatebert in this work, we leverage the topically driven neural language model (tdlm) @xcite to obtain topic representations, as it can employ pre-trained embeddings which are found to be more suitable for short twitter comments @xcite . the original model of tdlm applies a convolutional neural network (cnn) over wordembeddings to generate a comment embedding. this comment embedding is used to learn and extract topic distributions. @xcite show that transfer learning via sentence embeddings performs better than word-embeddings on a variety of tasks. hence, we modify tdlm to accept the transformer based universal sentence encoder (use) @xcite embeddings extracted from input comments, instead of the comment embeddings from cnn. the modified model is denoted as u-tdlm hereon. refer to appendix a.1 for the architecture of u-tdlm and also to @xcite . u-tdlm is trained on the train set from the source corpus and is used to infer on the test set from a different target corpus. the topic distribution per comment c is given by t @xmath0:k , where k is the number of topics. t c is passed through a fully connected (fc) layer to obtain transformed representation t c . besides, we first perform supervised fine-tuning of hatebert foot_0 on the train set of the source corpus. the vector corresponding to the [cls] token in the final layer of this fine-tuned hatebert model is chosen as the hate-bert representation for a comment. it is transformed through an fc layer to obtain the c vector. finally, in the combined model (hatebert+u-tdlm), the concatenated vector [t c ; c] is passed through a final fc and a softmax classification layer. the readers are referred to appendix a.2 for the architecture of the individual, and the combined models. ## results and analysis ## discussion and conclusion an in-corpus and cross-corpora evaluation of hate-bert and u-tdlm has helped us confirm our perspective on generalisation in the abusive language detection task. a contextualised representation model like hatebert can achieve great levels of performance on the abusive language detection task, only when the evaluation dataset does not differ from the training set. the performance of this model degrades drastically on abusive language comments from unseen contexts. topic models like u-tdlm, which express comments as a mixture of topics learnt from a corpus, allow unseen comments to trigger abusive language topics. while topic space representations tend to lose the exact context of a comment, combining them with hate-bert representations can give modest improvements over hatebert or at the least, retain the performance of hatebert. these results should fuel interest and motivate further developments in the generalisation of abusive language detection models. fc layer (size: 10) t c from u-tdlm (size:15) softmax layer t c ' comments (a) fc layer (size: 600) [cls] representation from fine-tuned hatebert (size: 768) softmax layer c comments (b) fc layer 1 (size: 10) fc layer 2 (size: 600) softmax layer t c from u-tdlm (size:15) [ t c ' ; c ] fc layer 3 (size: 400) [cls] representations from finetuned hatebert (size: 768) comments (c) figure 2: architecture of classifier for individual models: (a) u-tdlm, (b) hatebert, and the combined model (c) hatebert + u-tdlm; fc: fully connected.