Annotating Targets of Toxic Language at the Span Level (2022)

Abstract: In this paper, we discuss an interpretable framework to integrate toxic language annotations. Most data sets address only one aspect of the complex relationship in toxic communication and are inconsistent with each other. Enriching annotations with more details and information is however of great importance in order to develop high-performing and comprehensive explainable language models. Such systems should recognize and interpret both expressions that are toxic as well as expressions that make reference to specific targets to combat toxic language. We therefore created a crowd-annotation task to mark the spans of words that refer to target communities as an extension of the HateXplain data set. We present a quantitative and qualitative analysis of the annotations. We also fine-tuned RoBERTa-base on our data and experimented with different data thresholds to measure their effect on the classification. The F1-score of our best model on the test set is 79%. The annotations are freely available and can be combined with the existing HateXplain annotation to build richer and more complete models.

## introduction communication through social media has exploded in the last decades. the ease of posting opinions and the relative anonymity of posters has also unleashed problematic communication that can take many different forms: offensive language, hate speech, discriminatory language, abusive language, cyberbullying, etc., which can be all captured under the umbrella term toxic. such communication is often very complex and involves different values and perspectives. a comprehensive interpretation of such communication requires different aspects to be detected and combined, among which expressions that make a judgement or suggest negative implications and expressions that refer to targets such as a specific group of people or an individual belonging to such a group. an explainable system that can act as an automated moderator should be capable of "understanding" such phrases, reason over their content and bring specific aspects to posters' attention to explain what is wrong with a post and why it has to be, for example, removed by moderators @xcite ). an explainable model not only produces the desired outputs, but also explains why such output are produced. the natural language processing community has started many initiatives to automatically detect and classify toxic language and created a plethora of datasets @xcite . however, these data sets often address only one of the above-mentioned aspects. furthermore, they use slightly different terminologies and definitions for annotation and their annotation guidelines lack compatibility, which makes it difficult to combine their annotations. another problem is that annotation is often done at a global level, such as the whole sentence instead of specific phrases and tokens. some recent initiatives have started to annotate specific spans within the text itself @xcite but this is limited to toxic spans only. although previous studies did annotate the target community (e.g., women, muslims, immigrants, etc.) at the post level, none of these studies marked the words that describe or refer to such a group. being able to detect these phrases is, however, crucial to reason over who is targeted and how they are referenced. furthermore, annotating references to targets separately from toxic spans makes it possible to also process larger contexts of communication, among which conversations where references to targets and toxic expressions may be dispersed over multiple posts. by building a framework where target spans are annotated, it is possible to train explainable models that not only tell which community groups are targeted in a piece of text, but also indicate which words and phrases this decision is based on. this will make the model and its decision more understandable for end users. in this study, we describe a crowd-annotation task to annotate such target spans. we used this framework to add target spans to the hatexplain data set @xcite ) and tested classification models by fine-tuning with different quality selections of data. our contributions are as follows: the paper is organized as follows. in section 2, we summarize the related work on target annotations and position our work. in section 3, we describe the resources that we use to sample data in order to obtain sufficiently diverse annotations from each source. section 4 explains our annotation framework and the crowd-annotation task that we designed, while in section 5 we describe the results of the annotation. in section 6, we report on the language models we fine-tuned with our data and explain the results. we conclude and discuss future work in section 7. please beware that this paper may contain some examples of hateful content. this is strictly for the purpose of enabling this research and we seek to minimize the number of examples wherever possible. please be aware that this content may be offensive and cause you distress, which is certainly not the intention of the authors of this article. ## related work the number of studies on the automatic identification of hate speech and other forms of toxic language has rapidly increased in recent years. several definitions for toxic language have been proposed and many different annotation schemes have been designed and applied. part of these annotation studies focuses on the target community that has been victimized by such language and acknowledges that the description of these targets is relevant in different ways for the automatic detection of toxic language. early studies @xcite , @xcite presented this task as a binary task labeling data as hateful or not. in these studies, only toxic expressions targeting people were considered hateful. for example, according to the annotation guidelines of @xcite , an expression should be labeled as hate speech only if all of the three following conditions are met: (1) there is a deliberate attack. (2) the attack is on a specific group of people. (3) the motive for this attack regards aspects of the group's identity. although the identity of the target group is decisive in determining whether an expression is considered hate speech or not, no details on this were annotated. another widely used annotation scheme (see e.g. @xcite , @xcite ) was developed by @xcite who addressed the need for identifying more specific information about the target communities and therefore introduced several annotation layers as follows: (1) determine whether the message is offensive. (2) if the message is offensive, determine whether it is targeting people or not. (3) if the message is targeting people, determine (a) whether the message is targeting an individual, or (b) whether the message is targeting a group or member of a group considered a unity due to the same ethnicity, gender or sexual orientation or any other common characteristic, or (c) whether the message is targeting other entities like an organization, a situation or an event. finally and most recently, several studies @xcite , @xcite , @xcite , @xcite ) have taken the target annotations one step further by providing the group aspect on which basis it was targeted ( e.g. gender, race, national origin, disability, religion, sexual orientation, etc.) and by mentioning the specific target communities (e.g. africans, immigrants, muslims, homosexuals, politicians, etc.) this information would allow further research into differences in the framing of specific target communities and the building of classifiers that avoid bias in hate speech detection @xcite or permit researchers to delve into issues related to such bias. although all of these studies considered target detection in hate speech as challenging and important, none of them annotated target spans at the token level. our work builds on the already existing annotations of the hatexplain data set @xcite ) by adding such span annotations that refer to a target/ target community. in combination with the annotations already present in hatexplain, this allows us to train systems to detect both the phrases making reference to the targets as well as inferring the group aspect of these targets together with pointing at the phrases that represent the insulting content or judgement expressed about them. ## source data overview and sampling hatexplain is the first hate speech data set that covers many aspects of toxic language @xcite . each post in this data set has been annotated from three different perspectives: 1) the three main classes: hate speech, offensive or normal 2) the target community (i.e., the community that has been the victim of hate speech/ offensive language in the post) 3) rationales that are the parts of a post based on which annotators have decided to label it as such. the annotations were carried out at the word and phrase levels except for the target information which was done at the utterance level. according to @xcite , the data was collected from twitter and gab with a total of 9,055 and 11,093 samples, respectively. 1 for this study, we added target spans to the already existing annotations in this data set. this means that, for each sample targeting a target community, we wanted to determine which tokens in that sentence referred to that target community. for this reason, we selected only those samples that a) were instances of offensive language or hate speech b) targeted only one target community c) at least 2 out of 3 annotators agreed on its target label and d) had more than two and fewer than 61 tokens. we had extracted the distribution graphs of sentences per number of words and noticed that there are very few sentences that had more than 60 words in our data set. also, the more words a sentence has, the more complex it becomes. in addition sentences with fewer than 3 words seem to have not enough and useful information for analysis. that is why we selected only sentences whose number of words was within the range described. the reason why we chose sentences with only one target group was mainly to be make the task as easy and simple as possible for the crowd. nonetheless, we later found that there were still a number of sentences that targeted more than one target group even though they were annotated in hatexplain as having only one target group. this is also referred to and explained in 5.4. as a result, a total of 6,445 samples were selected. from these, we selected 3,480 samples that were representative of different target communities and data sources, which constituted about 54% of the full sample set. the number of selected samples per target community are shown in to prepare the data set for the crowd-annotation task described in the next section, we sorted the data per target community and created separate batches of 30 samples. each batch contained 29 samples from hatexplain and 1 attention check sentence randomly selected from a list of 60 sentences. these attention check sentences were manually generated and annotated, and were both targeting and toxic. attention check sentences were included to inspect the crowd-annotators' annotation and make sure that they did not simply annotate the data samples haphazardly or even skip them. the total number of batches amounted to 120. the first 25 batches only contained "hispanic" and homosexual" target communities. from batch 26 onward, however, each batch contained a mix-ture from different target communities and sources (gab and twitter) to reduce imbalanced data bias. ## crowd annotation task for the crowd-annotation task, annotators were provided with batches of 30 toxic sentences presented one-by-one, where each sentence was followed by a list of tokens with tick-boxes. they were asked to read each sentence carefully before determining who the target was and select the tokens that referred to that target. at the start, they received an instruction page with examples. they were instructed to choose all the words -including the pronouns -that referred to the target no matter how many times they appeared in the sentence. also, complete noun phrases including any adjectives had to be selected. below are a few examples that demonstrate how sentences should be annotated: target spans: you, bangladeshi, guys 2. just because we don't let groups of creepy moroccan rapists into the club doesn't make us racist. target spans: creepy, moroccan, rapists 3. they do no shit other than going to the mosque. target spans: they 4. fuck the ignorant assholes from america, uk, and all other countries who try and soil the wonderful ties between all democratic nations. target spans: the, ignorant, assholes, from, america, uk, and, all, other, countries it is worth noting that there are certain words and phrases that are toxic and specifically used for referring to a target community, such as "spic" in 7, which generally refers to hispanics, or "nigger" in 6, which refers to black people. these words should be also annotated as target words although they are toxic as well, because they are ultimately used to address specific target communities. we used lingoturk @xcite as the platform to design our task, which is a free and available crowdsourcing client/ server experiment management system. furthermore, we used prolific @xcite , as the online platform for recruiting crowdworkers. the advantage of prolific over other crowdsourcing platforms, such as amazon mechanical turk, is that it provides more prescreening options, such as nationality, first language, ethnicity, political affiliation, socio-economic status, etc. also, prolific is not only limited to us participants. when recruiting annotators, we clearly described the aim of the study to them and explained what they had to do in detail. no specific sensitive information about annotators was stored. we also informed them beforehand that they should be aware of the inappropriate content of the sentences and they were not supposed to participate in this study if they were not comfortable with being exposed to such a language. since the study was closely related to one's cultural understanding of the context and there were a lot of slang words and phrases used, we recruited only participants that met the following criteria: in order to determine the optimal number of annotators to recruit for each batch, we ran a test batch with 15 annotators and then extracted 10 random subsets, once with 5 and once with 10 annotators. mathtools following the crowdtruth framework (dumitrache et al., 2018), we used the media unit quality score (uqs) to analyze the collected results from different sets of annotators. uqs expresses the overall worker agreement over a so-called media unit. in our case, each token was considered to be a media unit with the binary classification as either targeting (1) or non-targeting (0). in order to calculate the uqs, one needs to first calculate the average cosine similarity between all worker vectors, weighted by the worker quality (wqs) and annotation quality (aqs). for more details on how each of these scores is calculated, please refer to @xcite . the advantage of using uqs in comparison to other metrics for calculating the inter-annotator agreement is that crowdtruth interprets both the disagreement among the annotators and the ambiguity of the token. the quality of an annotation is considered as the interaction between the quality of the annotator in terms of how often she/ he agrees with others as well as the complexity of the input data and set of annotation categories. we calculated the uqs for the complete set with 15 annotators and each of its subsets including 10 and 5 annotators, respectively. next, we took the average of the obtained results over all subsets. by doing so, we could test in which cases and with what number of annotators the results were more consistent. in as can be seen in to select 10 annotators within the prolific platform, the above pre-screening criteria were applied to the total pool of annotators. after running each batch, we analyzed the data to make sure the annotation quality was good enough and annotators acted according to our instructions. in order to do so, we compared the performance of each annotator to that of other participants, validated the attention check sentences, and considered the time taken on the whole for each annotator to finish the task. we also validated the annotations of some other randomly selected sentences. finally, we checked whether the data provided by each participant corresponded with their prolific id and if they had entered a completion code showing that they had completed the whole task. if annotators failed any of the above-mentioned criteria, their submissions were rejected and another annotator was recruited in their place. we added the ids of rejected annotators to our blocklist after each batch, which would exclude them from the next batches. in the next section, the results will be described in more detail. ## automatic classification after having obtained the labels for each token and having determined the best uqs threshold, we low uqs high uqs you ( 206) nigger (357) they ( 90 ) faggot ( 265 ) the ( 77) bitch ( 211 ) to test this, we fine-tuned a pretrained language model for a token classification task to predict whether each token was targeting or non-targeting. in @xcite , the performances of a number of language models for detecting toxic spans in a sentence were compared. the best-performing model (roberta-base) had the highest f1-score on the test set with a value of 68.41%. therefore, we chose roberta-base as our pretrained model. for fine-tuning, we converted the data to the iob (inside-outside-beginning) format, which is widely used in token classification tasks @xcite . we created a separate test set consisting of 20% of the whole data, but ensured that it was representative of all target communities and data sources. the test set was generated by setting the uqs threshold to 50, as this threshold had previously resulted in the highest agreement when the crowd annotations were compared with the expert annotations. for the training, on the other hand, we generated three different training sets with uqs thresholds of 50, 60, and 70, to test the effects on the predictions. all other hyperparameters and arguments remained the same in all three cases. furthermore, we selected 10% of the training data as the validation set. the training set, test set, and validation set included each 2227, 696, and 557 samples (sentences), respectively. arguments and hyperparameters used for the training are as follows: batch @xmath0; @xmath1; learning @xmath2; weight de-@xmath3.01. to prepare the data for fine-tuning our models, they were tokenized using autotokenizer from hugging face 2 . during fine-tuning, evaluation was done at the end of each epoch. we batched our data with a data collator while using padding to make them all the same size. each pad was padded to the length of its longest sample. we padded not only the inputs, but also the labels. we evaluated our model and its predictions on the test set with accuracy, precision, recall, and f1-score. after the predictions had been made, we needed to do some postprocessing. we picked the predicted index (with the maximum logit) for each token, converted it to its string label and ignored wherever we put a -100 label. we repeated the training procedure with the three training sets, each generated with a different uqs threshold as described earlier. overall, our model showed a good performance predicting the target spans. the scores for the dominant class "non-targeting" (0) were higher than the scores for the "targeting" class. the weighted f1 scores ranged from 74 to 79% , which is significantly higher than the results for the toxic span detection task in @xcite although the tasks, data and annotations are different across these tasks. the best results were again obtained when the uqs threshold was set to 50. ## conclusion we presented an extension to the hatexplain data set with annotations for target spans using crowd-annotation. the extended data set will enable the community to train and test models that recognize not only toxic language, but also the referents that are targeted. this is essential for future systems that need to comment on "wrong" behaviour in possibly interactive settings, discussing who has been targeted by what aspect and what toxic comments are used against the targeted person or community. we provided the guidelines and instructions with clear examples of what we meant by target in a toxic sentence. we collected expert-annotated data for two of the batches with reasonable agreement among annotators. we obtained crowd annotations for target tokens in 3,480 sentences that targeted one target community. we also analyzed frequent patterns observed in the annotations and provided a statistical overview of the collected annotations. we fine-tuned three roberta-base language models with our data and investigated how changing the uqs threshold would affect the results. our best model resulted in an f1-score of 79% on the test set, which was higher than other works in the field of toxic span classification. all the required information regarding the data and models is available on our github repository foot_3 . in future work, we will extend the data to multiple languages as well as to richer and longer contexts, such as in conversational settings, where toxic expressions and targets can be mentioned sparsely. we want to explore other language models and compare their results by changing the hyperparameters and training arguments. also, we are keen to compare the predictions of these models to the crowd-annotations and perform some error analysis.