IRC ologne at G erm E val 2021: Toxicity Classification (2021)

Abstract: In this paper, we describe the TH Köln’s submission for the Shared Task on the Identification of Toxic Comments at GermEval 2021. Toxicity is a severe and latent problem in comments in online discussions. Complex language model based methods have shown the most success in identifying toxicity. However, these approaches lack explainability and might be insensitive to domain-specific renditions of toxicity. In the scope of the GermEval 2021 toxic comment classification task (Risch et al., 2021), we employed a simple but promising combination of term-frequency-based classification and rule-based labeling to produce effective but to no lesser degree explainable toxicity predictions.

## introduction toxic language in online comments and discussions is an increasingly relevant problem @xcite . however, toxicity classification is a challenging task @xcite . there is no universally agreed on complete definition of toxicity. instead, toxicity is an umbrella term for a variety of problematic, negative phenomena @xcite . since, even for human annotators, it often is hard to explain why precisely a comment is or is not toxic, language models trained on an extensive dataset of labeled comments usually performed best in the task of identifying toxicity @xcite . however, these approaches lack explainability, and when the systems are employed to filter user-composed comments, the system should be able to indicate what aspect(s) of the comment lead to it being declared as toxic. more traditional approaches like support vector machines, linear models, or naive bayes classification do not perform as well, generally. nevertheless, tfidf-based classification has clear advantages in terms of ease of domain adaptability and explainability. our ap-proach employed in the germeval 2021 toxicity classification task @xcite combines these traditional classification methods with the snorkel framework @xcite . using labeling functions (lfs), we identify indicators for aspects of toxicity to enable explainable toxicity judgments and improve classification performance. depending on the definition, a wide range of aspects defines what constitutes toxic comments @xcite . as described by georgakopoulos et al. ( 2018 ), a toxic comment, aside from exhibiting verbal violence, can also be "a comment that is rude, disrespectful or otherwise likely to make someone leave a discussion". this definition is consistent with the definition of toxicity given in the overview paper, where toxic comments are deemed problematic because they discourage and hamper participation in discussions. @xcite . to better tackle the issue of toxic language in comments, we categorized the aspects of toxicity in comments into three categories. the first category of toxicity-defining aspects can be characterized as language aspects. these describe features like particular vocabularies or attributions that carry a toxic tonality. they range from different forms of hate speech such as racism @xcite , sexism (jha and mamidi, 2017), fanaticism and identity hate, to profane, offensive and aggressive language and incivility @xcite . of the toxic features listed in the task description, vulgar language, screaming, and insults are listed as toxic comment features @xcite . the second category can be described as toxic behavioral aspects, defined by their communicative intentions. they are composed of cyberbullying (chavan and shylaja, 2015), sexual predation @xcite , threats, and socalled spam messages @xcite . sarcasm and zynism, discrimination, discrediting, accusations, and threats are the aspects of toxic comments mentioned in @xcite that best fit this category. finally, the most latent type of toxic aspects can be grouped as inappropriate language. these highly context-or domain-specific aspects span from age-inappropriateness @xcite to general undesirable topics or off-topic messages. in one of the rare approaches to more explainable toxicity classification, @xcite tried to address the issue of poor explainability of language model classification techniques. however, their approach is decidedly different from ours. by assuming that a text is at least as toxic as its most toxic part, they focused their work on increasing the explainability of transformer-based classification. as previously mentioned, this is a rare exception since most recent approaches utilize deep learning and attention-based language models like bert @xcite . ## methodological approach our approach applies binary classification (bc) and data programming on a preprocessed version of the provided corpus. in this section, we give a brief overview of all aspects of this approach. ## conclusion to overcome the difficulties posed by germeval 2021's toxicity classification task, we combined traditional linear svc classification with labeling functions based on false negative and false positive classifications of the model. this combined approach is able to deliver explainable results and adaptability. despite the total coverage of the lfs in the training data of about 5% and although we discarded most of the developed labeling functions due to bad classification performance, data programming increased the classification's performance significantly. including the four best performing labeling functions, our final classification model increased the f1-score by almost 5 points. this increase indicates that the toxic attributes covered by the lfs have not been taken into account by the linear svc classifier. on the evaluation dataset, our approach reached an f1-score of 0,576. overall, the approach was a success. however, a more extensive dataset might have benefited our linear svc model's and our labeling functions' performance.