Muted: Multilingual Targeted Offensive Speech Identification and Visualization (2023)

Abstract: Offensive language such as hate, abuse, and profanity (HAP) occurs in various content on the web. While previous work has mostly dealt with sentence level annotations, there have been a few recent attempts to identify offensive spans as well. We build upon this work and introduce MUTED, a system to identify multilingual HAP content by displaying offensive arguments and their targets using heat maps to indicate their intensity. MUTED can leverage any transformer-based HAP-classification model and its attention mechanism out-of-the-box to identify toxic spans, without further fine-tuning. In addition, we use the spaCy library to identify the specific targets and arguments for the words predicted by the attention heatmaps. We present the model’s performance on identifying offensive spans and their targets in existing datasets and present new annotations on German text. Finally, we demonstrate our proposed visualization tool on multilingual inputs.

## introduction offensive language such as hate, abuse, and profanity (hap) occurs in various content on the web such as social media sites (e.g. twitter) and discussion forums (e.g. reddit). such content can be hurtful to the reader, and identifying and visualizing hap speech is necessary to understand and avoid harm. it increases interpretability and can be used to hide and provide a warning for offensive terms, and to avoid generating hate in large language models. while such visualizations exist, the focus has primarily been on english hap and on identifying offensive language on the sentence level @xcite . there are few works that explore spans and other languages @xcite warning: this paper contains offensive examples. figure 2 : german input ("politicians notoriously lie, not to say their entire lives"): <politiker, lügen @xmath0. @xcite but these do not identify and visualize the target of the offensive argument which is an important indicator regarding whether the offensive argument is harmful or not, as shown in @xcite . we propose identifying hate using existing approaches @xcite to display multilingual offensive arguments and their targets using heat maps as a means of showing their intensity. moreover, the spacy library @xcite can also be used to identify the specific target and argument from the predicted words. an example with @xmath1,@xmath2: muted : visualizing offensive spans and targets using attention heatmaps. a token-level attention score of a given sentence is obtained using the average attention across all heads of the last layer of the given hap classifier, and extracting the attention from the first token (often the cls vector). the score for a word is calculated as the maximum token-level attention score of its constituent tokens. finally, we display the predicted spans using the attention heatmap, and use spacy's dependency parser to identify the target and argument in the predictions. and german inputs in fig. 1 and fig. 2 , with the resulting visualizations. our contributions are as follows: identifying offensive content has been a popular area of research in recent years @xcite . one popular model that is available is hatebert @xcite which is a bert-based model finetuned on offensive speech from reddit comments. similar models exist in other languages such as dehate-bert @xcite in german. we present our own multilingual model for detecting offensive content which outperforms hatebert on offensive span selection. however, our notebooks demonstrating our approach for identifying the offensive target and argument can be used with any transformer-based offensive classifier. several demos on offensive text exist that perform on the span or sentence level, mostly in english @xcite . perhaps the most relevant demo is mudes @xcite . they identify offensive spans in input text by classifying each token as offensive or not, and support english, danish and greek. the ui is token-classification based, and can be used with their trained models and the datasets used in the paper (or any input text) to identify offensive spans which will be displayed in red. in contrast to other prior work, our heat map-based system can be used to visualize the offensive argument and target for any language for which a sentence level hate classifier is available. ## approach muted provides an intuitive visualization of existing hap classifiers by using attention maps to identify offensive text and their targets, as shown in fig. 3 . formally, for a transformer model (of l transformer layers and h attention heads) finetuned to classify whether a given input sentence x contains offensive language, we first obtain the attention outputs a l @xmath3, @xmath4, h] of the last transformer layer. we then compute the average attention across all heads, a ′ = 1 h h @xmath5, and extract the attention vector for the first token (e.g., the cls token for bert @xcite models), a ′ 0 ∈ r 1×|x| . based on a threshold, we obtain the set of tokens t with the highest attention score, which can be intuitively viewed as the tokens that contribute most to the classification decision. we convert the token-level attentions into word-level attentions by assigning a word the maximum attention of any of its constituent tokens. we provide the word-level attention visualization in the form of heat maps, and mark the target and the argument of the offensive span in the sentence (see the system output in fig. 3 ). our system can be used to visualize any transformer-based model that is trained to classify if a given sequence has hap content or not, irrespective of the language. in this work, we present the piccolo-hap classifier foot_0 , a tiny 4-layer xlm-roberta @xcite ) model (with 153 million parameters) finetuned on the hap detection task for 6 languages (english, german, japanese, spanish, french and portuguese). specifically, we distil the self-attention relations of an in-house xlm-roberta base model on a taskagnostic (general purpose) manner into a 4-layer architecture, as proposed in @xcite . we finetune this general purpose language model on the hap classification objective, using open-source multilingual annotated datasets @xcite @xcite @xcite @xcite @xcite originating from social media data, as well as internally an-notated samples from cc100 @xcite and scraped news data from the internet in the six languages mentioned above. for non-english data, we also translate english datasets @xcite to the language required. we finetune the model on a total of 1.7 million sentences, with the majority of data being in english. ## experiments we compare our model to a random baseline, as well as open-source toxicity classifiers (monolingual and multilingual). first, we evaluate a random selection of spans as target and arguments in the sentence. specifically, each span in the sentence is marked as hap with a probability of 0.50. we also use three off-the-shelf english hate-bert models @xcite , each finetuned on either hateval (basile et al., 2019), offenseval @xcite or abuseval @xcite . these models were made available by the hatebert authors foot_1 , and we have not finetuned them ourselves. we also compare our multilingual model to another open-source multilingual classifier available on huggingface, multilingual toxicity classifier plus [mtc+] 4 , and two german (monolingual) classifiers, dehatebertde foot_2 (aluru et al., 2020) and german toxicity classifier plus (v2) foot_3 . ## system demonstration we have jupyter notebooks and a front-end ui where users can load their models, and obtain visualizations for inputs in any language. ## conclusion we present a method for identifying and visualizing offensive arguments and their targets using the attention of the sentence-based offensive classifier to create a heat map. our multilingual model outperforms existing popular approaches on multiple datasets in english and german. we provide a notebook and user interface to run any multilingual transformer classifier on sentences and visualize the heat map as well as @xmath6,@xmath7. in the future, we would like to add a classifier to indicate harm of @xmath8,@xmath9. we would also like to extend our demo to provide warnings and hide the offensive content to users.