S afe W eb UH at S em E val-2023 Task 11: Learning Annotator Disagreement in Derogatory Text: Comparison of Direct Training vs Aggregation (2023)

Abstract: Subjectivity and difference of opinion are key social phenomena, and it is crucial to take these into account in the annotation and detection process of derogatory textual content. In this paper, we use four datasets provided by SemEval-2023 Task 11 and fine-tune a BERT model to capture the disagreement in the annotation. We find individual annotator modeling and aggregation lowers the Cross-Entropy score by an average of 0.21, compared to the direct training on the soft labels. Our findings further demonstrate that annotator metadata contributes to the average 0.029 reduction in the Cross-Entropy score.

## introduction while the web space is inundated with derogatory textual content, the subjectivity of their interpretation frequently necessitates a system capable of capturing reader disagreements. the learning-with-disagreement (le-wi-di) task involves learning annotators' disagreements based on how they categorize a text @xcite . recent research has found that almost every annotation task contains a wide range of disagreements @xcite . the subjective and biased nature of the raters, among other elements of natural language comprehension, make it crucial to learn disagreements through annotations @xcite . in this study, we compare two strategies of disagreement learning: disagreement targeted learning of soft labels, and annotator-specific learning with post aggregation, using bert model. furthermore, we utilize annotator-specific metadata, to capture annotators' disagreements in disparaging content. since the advent of social media, which has flooded the web with massive amounts of content, the number of offensive text, such as hate speech, misogyny, sexism, and abusive content has also increased significantly. several studies were carried out to battle this problem, such as, @xcite williams studied online hate-speech in tweets, triggered by the murder of lee rigby, a london-based drummer @xcite . @xcite formulated the cyber-bullying in social media as an nlp task @xcite . similar works are conducted in @xcite @xcite . however, tasks related to the detection of social phenomena, like offensiveness, and toxicity are often subjective in nature @xcite . a recent survey among american adults stated that according to half of the participants, "it is hard to know what others might find offensive", and the majority of them acknowledged there were disagreements in what is perceived as sexist or racist (pew, accessed: 2022-12-03). to this end, we aim to develop a system that can capture subjective disagreement in derogatory text. the four datasets in the le-wi-di task come with the annotator-specific labels, with aggregated hard labels (majority voting) and soft labels (average of the labels). although a system for modeling disagreements should be trained to estimate soft labels, it is not clear, whether direct training on the soft label or aggregating on the annotator labels is a better approach. hence, our first research question (q1): can annotator-specific classification models, and post hoc aggregation outperform the direct approach of regression on soft labels in disagreement modeling? additionally, we explore the annotator metadata which explains how an annotator labeled other related text, and we pose the question (q2): can annotator metadata improve the disagreement modeling? to address these questions, we compare bert-based disagreement-targeted learning (regression) and post-aggregation learning (classification) and explore different strategies for incorporating annotator metadata to model the disagreement. however, due to the inconsistency of the annotators and lack of metadata, we limit our comparisons to two datasets only. our work has several important implications. to begin, our model's ability to capture conflicts makes it applicable to the modeling of controversial social phenomena and public opinions. hence, it can be used to model ambiguity in textual ambiguity. furthermore, our explorations of incorporating annotator metadata can help understanding readers' perception and outlook in different context. finally, enhancing transparency and accountability among the raters can be performed as a mean to quality control in multi-rater annotation process. the code for implementing our work is available here: https://github.com/sadat1971/lewi-di-semeval-23 2 dataset and task description semeval'23 task 11 has four datasets that deal with derogatory text. while the three datasets are in english, armis is in arabic. along with soft and hard labels, each dataset contains some metadata. they are described below in brief. the multidomain agreement (md) dataset comes with tweets from three domains: blm, election and covid-19 @xcite . a total of 819 annotators were used to label all the tweets using amt. a random combination of five annotators was chosen to label each tweet for offensiveness. the train set contains 6,592 tweets, the dev set from the practice phase has 1,104 tweets, and the test set from the evaluation phase contains 3,057 tweets. the hs-brexit dataset contains tweets related to brexit, and annotation from six annotators (a target group of three muslim immigrants in the uk and a control group of three) @xcite . each of them labeled a tweet for hate speech, which is the target class of the task. they also annotated tweets for being offensive and aggressive. the train, dev, and test set have 784, 168, and 168 tweets respectively. misogyny and sexism are labeled in the armis dataset, rated by three annotators (moderate female, liberal female, and conservative male) @xcite . there are 657, 141, and 145 tweets in the train, dev, and test sets, respectively. the convabuse dataset captures dialogues between a user and two conversational agents, and at least two annotators annotated the conversation for abusiveness @xcite . the dataset also provides labels for a conversation being sexist, explicit, implicit, intellectual, racist, transphobic, homophobic, and the target of the abuse. the train, dev, and test set have 2,398, 812, and 840 tweets respectively. ## system description for the textual data, we use a pretrained language representation model, called bert @xcite . since bert is trained on the english data only, to handle the armis task, we use arabic-bert @xcite . figure 1 shows the system description. to address q1, we compare two techniques-post aggregation and disagreement targeted learning, and we also investigate the effect of metadata to address q2. the performance is measured by f1-score and cross-entropy (ce) score. ## experimental set-up all of our models use "bert-base-uncased" version of bert ("bert-base-arabic" in armis). we deploy a two-layered fully-connected network for fine-tuning in both regression and classification tasks. we choose the hyper-parameters from all the combinations, by three-fold cross-validation in the practice phase, and on the released validation set in the evaluation phase. the hidden size and dropout rate are chosen from {32, 64, 128, 256}, and {.1, .3, .5}. the learning rate is chosen from {5e-4, 1e-5, 5e-5, 1e-6}. we keep the batch size small due to the gpu limitations and choose from {8, 16}. since bert models quickly overfit on the data, we kept the epoch size between 2 and 4. however, for "arabic-bert-base", the performance was unstable, and we train upto 10 epochs. for all cases, adamw is used as optimizer @xcite . for all our experiments, pytorch version 1.11.0 is used @xcite . to evaluate the result of capturing disagreement, we use the cross-entropy score provided by the competition. if the target soft label is t , and predicted soft label is p , for a dataset of size d, the cross-entropy (ce) is computed as: we further report the f1-score (micro) on the hard label to evaluate the model performance on the majority-voted final prediction task. ## result and discussion we further investigate why the post-agg approach works better at capturing disagreement. since the dis-learning approach does not take into account individual annotators, it mainly approximates the "intensity" of a text being derogatory. conversely, the post-agg approach considers each annotator separately and learns their annotation pattern, which is aggregated afterward. consequently, dis-learning has to depend only on textual data, making its job harder than post-agg. however, in a realistic case, the annotators may not be consistent (as in the md and convabuse datasets), or a large number of models are needed to be trained, rendering the post-agg technique infeasible. therefore, the post-agg approach is better suited for modeling disagreement if a small number of annotators are consistent across the dataset. hence, q1 is addressed. next, the results reveal that performance is enhanced when annotator metadata is utilized as opposed to when it is not ( in the md, hs-brexit, armis, and convabuse datasets, our results were ranked 7th, 9th, 11th, and 12th, respectively. overall, we ranked 9th in the ce score category and 8th in the f1-score category. error analysis finally, we focus on the error analysis of this study. we find that both our approaches often make mistakes in prediction for the texts that do not use slang or curse words but are still voted by the majority as offensive. for ex-ample, three of the five annotators annotated the following sentence as offensive (soft label 0.60): #tonybobulinski #maga2020 #maga #changey-ourvotetotrump #bidencrimefamily #bidenhar-ris2020 #bidencares #laptopfromhell joe is going down. <url>. however, our model predicts the soft label as 0.15. similarly, tweets that contain curse words but do not necessarily exhibit offensiveness, are sometimes mistaken by our model as hate speech. for example, the tweet: astounding words from the prolific and talented -<@xmath0, however, our model predicts the soft label as 0.85, due to the presence of profane language in one of the hashtags. ## related works though the majority of ai learning still operates under the assumption that a single interpretation exists for each item, research is growing to build learning methods that do not rely on this assumption @xcite ). rater's disagreement is a familiar phenomenon in natural language processing @xcite . the disagreement may take place because of the annotator error or interface problem @xcite , explicit or implicit ambiguity @xcite , item difficulty @xcite , and subjectivity @xcite . notwithstanding, the simpler task such as pos tagging @xcite to subjective tasks like sentiment analysis, semantic role assignments also involve raters' disagreement @xcite . hence, researchers argued for taking disagreement into account during the labeling process and retaining the implicit ambiguity @xcite ). to this end, we explore the learning-with disagreement task for derogatory text. the previous version of this competition was launched in 2021, where the organizers used nl and image-classification task to address for disagreement in the labeling @xcite . the winning team used the sharpness-aware minimization technique (sam) and a special nn layer called softmax crowd-layer with bert as baseline model @xcite . while the sam architecture was mainly used for cifar-10 (image classification), the crowdlayer architecture aims to map the label with each individual annotator. since the current competition only involves text, we fine-tune a bert model and use the annotator metadata to capture the disagreement. ## conclusion because of the proliferation of social media content, the internet has become a breeding ground for derogatory text. however, due to the differences in human perception and opinion, often there is no unanimous consensus among the annotators about the text being derogatory or not. hence, it is imperative to store the soft labels and capture annotator disagreement in the modeling process. our work compares the direct training on the soft label with the annotator-specific model and postaggregation. we find that with the presence of consistent annotators, it might be helpful to take the latter approach. in addition, integrating annotator metadata has been proved to be beneficial in our experiments. our work has a wide variety of potential future research directions, such as: