Dragonfly_captain at S em E val-2023 Task 11: Unpacking Disagreement with Investigation of Annotator Demographics and Task Difficulty (2023)

Abstract: This study investigates learning with disagreement in NLP tasks and evaluates its performance on four datasets. The results suggest that the model performs best on the experimental dataset and faces challenges in minority languages. Furthermore, the analysis indicates that annotator demographics play a significant role in the interpretation of such tasks. This study suggests the need for greater consideration of demographic differences in annotators and more comprehensive evaluation metrics for NLP models.

## introduction which voices should an algorithm be trained to replicate? in natural language processing, detecting and handling disagreements among annotators is a critical task. annotators may disagree due to various reasons, including differences in personal beliefs, cultural background, or subjective interpretations. disagreements in annotated data may lead to biased models, which can have negative implications, such as amplifying hate speech or discrimination. however, in the current supervised learning paradigm, annotation disagreements are often resolved by aggregating the opinions of majority groups, effectively excluding minority perspectives. therefore, it is essential to develop methods that can capture and address disagreements in annotated data. this paper reports analysis of participating the semeval-2023 task 11, learning with disagreement, to promote a better understanding of disagreements among annotators and their impact on natural language processing models. ## background the use of crowdsourcing for annotating data has become a popular solution for the expensive and time-consuming process of labeling large amounts of data in nlp models. however, it also brings its own set of challenges, with annotator disagreement being a primary concern @xcite . annotator disagreement can have various causes, but subjective judgment and language ambiguity are common culprits @xcite . failure to consider the subjective and ambiguous nature of some instances can result in inaccurate predictions. ## understand disagreement from data the semantic evaluation workshop 2023 task 11 collected a benchmark of four textual datasets with different characteristics @xcite . the hs-brexit dataset is a new dataset of tweets on abusive language on brexit annotated for hate speech, aggressiveness, and offensiveness by two distinct groups. the armis dataset is a dataset of arabic tweets annotated for misogyny detection by annotators with different demographics characteristics. the convabuse dataset is a dataset of english dialogues annotated by at least three experts in gender studies using a hierarchical labeling scheme. the multidomain agreement dataset is a dataset of english tweets from three domains annotated for offensiveness by 5 annotators via amt, with a focus on pre-selecting tweets that are potentially leading to disagreement. overall, the datasets provide a multiplicity of labels for each instance, and particular attention was paid to selecting instances that have the potential for disagreement among annotators, as shown in the hs-brexit @xcite consists of tweets related to hate speech on brexit, annotated for hate speech, aggressiveness, and offensiveness by the same six annotators from two distinct groups: a target group of three muslim immigrants residing in the uk and a control group of three other non-muslim immigrants. splitting the annotation results by target group and control group in the given training set, i ran a t-test to identify whether there are significant disagreement among muslim immigrants and other individuals. the null hypothesis is there is no difference in group means, while the alternate hypothesis is there is some difference. the t-test p-value (8.86e-27) is smaller than 0.05, so we have strong evidence to show that the target group and control group hold different stance in identifying hate speech on brexit in the given text. also the t-statistic is -10.91 which reflects the control group perceives more hate speech than the target group of muslim immigrants. conabuse @xcite is a dataset of english dialogues between expert users and conversational agents. it is important to note that this data is collected during experimental setups, which may be significantly different from real-life conversations, such as those that take place on social media platforms. armis @xcite ) is a collection of arabic tweets annotated for misogyny detection. the annotations were provided by annotators with different demographic characteristics, including "moderate female", "liberal female", and "conservative male". like the hs-brexit dataset, the annotators represent compound features of their gender and political views. additionally, this dataset presents its unique challenge of the arabic language. to investigate how different demographics of annotators may identify misogyny/sexism differently in the armis dataset, three t-tests were conducted to compare the set of annotations between each pair of annotators. the results revealed that the opinions of the moderate female and the conservative male were not significantly different from each other (p-@xmath0.318). however, both groups were significantly different from the liberal females' identification of misogyny(p-@xmath1.14e-06 and 0.0001)). specifically, the liberal female annotators identified much less misogyny in the dataset compared to the other two groups. however, only three annotators couldn't show whether the distinction of these three people will also apply to other moderate female, liberal female, conservative male groups that they were representing in the task. @xcite contains english tweets from three different domains: black lives matter (blm), election in 2020, and covid-19. the dataset has been annotated for offensiveness via amazon mechanical turk (amt). compared with previous datasets, md-agreement has a more diverse annotator pool. ## experiments the experiments involves using roberta-base @xcite to predict soft agreement label in regression setup with evaluation using crossentropy loss. given that subjective disagreement tasks lack a definitive "truth," the primary metric for evaluating the model's performance will be a soft evaluation using cross-entropy. this metric assesses the degree to which the model's predicted probabilities align with the agreement level among annotators. although my results were in the middle of the leaderboard foot_0 , i did not implement any further modifications to the basic model. instead, i focused on understanding how the disagreement prediction works across datasets. this allowed me to gain insights into the nature of the disagreement in each dataset and identify potential areas for improvement in future work. based on the results, it shows that the basic roberta model performs best on the convabuse task with a cross-entropy score of 0.347, and worst on the armis task with a score of 0.688. this suggests that the convabuse task is relatively easier task while armis is more challenging. also becasue i used roberta base model that is pretrained on english corpus. the ranking also follows this pattern. the best performed convabuse task is the lowest rank which shows convabuse is the easist task that advanced language techniques can easily learn to identify agreement/disagreement pattern. further, because convabuse data is collected from experimental setup which also make it easier task than identifying agreement/ disagreement in complex real social context. ## conclusion in conclusion, this study highlights the importance of considering disagreement in nlp tasks and the impact of annotator demographics on the interpretation of such tasks. my results suggest that machine learning models can perform well on experimental datasets but may struggle with minority languages. as a future direction, it is important to investigate more effective approaches for addressing disagreement in subjective tasks in complicated societal context, as well as to explore ways to improve the diversity and representativeness of the annotator pool.