Arguably at S em E val-2023 Task 11: Learning the disagreements using unsupervised behavioral clustering and language models (2023)

Abstract: We describe SemEval-2023 Task 11 on behavioral segregation of annotations to find the similarities and contextual thinking of a group of annotators. We have utilized a behavioral segmentation analysis on the annotators to model them independently and combine the results to yield soft and hard scores. Our team focused on experimenting with hierarchical clustering with various distance metrics for similarity, dissimilarity, and reliability. We modeled the clusters and assigned weightage to find the soft and hard scores. Our team was able to find out hidden behavioral patterns among the judgments of annotators after rigorous experiments. The proposed system is made available.

## introduction in the field of natural language processing (nlp), it has been assumed that a given natural language expression (such as a sentence) has a single, clear and unambigous meaning in a particular context. however, this assumption is now being recognized as an idealization or simplification that does not always hold in practice. for example, the sentence "i saw her duck" could mean that someone observed a woman's pet duck, or that someone watched a woman quickly lower her head to avoid hitting it on a low object. in these cases, the interpretation of the sentence depends on the context and the particular meanings of the words used. the learning with disagreement shared task @xcite aims to provide a testing framework for machine learning models that can learn from disagreements in natural language interpretation. our approach involves developing a model that can take into account multiple possible interpretations of a single sentence and identify the most likely one based of available context and information. we have utilized unsupervised learning approach and experimented using various clustering techniques and different types of similarities of annotations and metadata to bring up associations and relationships among various groups of annotators. this could tell us how a specific group of annotators think given a particular scenario. ## background this shared task provides a testing framework for learning from disagreements using datasets containing information about disagreements for interpreting language. the focus of this task is on subjective tasks, where training with aggregated labels is less effective. there are four textual datasets with different characteristics, including social media and conversation genres, english and arabic languages, and tasks related to misogyny, hate speech, and offensiveness detection. these datasets also use different annotation methodologies, including experts, specific demographic groups, and amt-crowd, and all provide multiple labels for each instance. the le-wi-di dataset is a collection of four existing datasets that have been harmonized into a common json format that emphasizes their commonalities. these datasets include the hs-brexit dataset @xcite , which is a new dataset of tweets on abusive language related to brexit that has been annotated for hate speech, aggressiveness, and offensiveness by six annotators from two distinct groups. the armis dataset @xcite is a dataset of arabic tweets annotated for misogyny and sexism detection by annotators with different demographic characteristics. the convabuse dataset @xcite is a dataset of english dialogues between users and conversational agents, annotated for abuse by at least three experts in gender studies. the multidomain agreement dataset @xcite is a dataset of english tweets from three domains, annotated for offensiveness by five annotators via amt. we have gone through some of the related works dealing with subjective tasks in nlp domain.a survey of natural language processing for social media" by @xcite . this survey paper provides an overview of nlp techniques for social media analysis, including sentiment analysis, emotion detection, and opinion mining. another paper by @xcite examines the state-of-the-art approaches for detecting abusive language in online user content, including hate speech, cyberbullying, and harassment. "offensive language detection: a review" by @xcite . this review paper summarizes the current research on offensive language detection, including different types of offensive language and the challenges associated with detecting them. "a survey on sentiment analysis in social media" by @xcite . this survey paper reviews the current research on sentiment analysis in social media, including the challenges of analyzing informal and unstructured text in social media platforms. ## system overview we have focused on behavioral segregation of annotations that had been provided to us in data-set. using this information we then experimented with various unsupervised clustering techniques and then passed the output to bert transformer @xcite ) that gave us the predictions for each group. we have observed that assigning weights to each cluster worked well in our case because that indicated the contributing nature of a particular group. organisers had mentioned that there are two groups of annotators, hence we were curious to know the unity of each group. after doing various experiments we had found that muslim annotators had a less bias so we have kept the cluster with muslim annotators with more weights i.e 2 x ( control group ). this helped in providing higher soft-score and better hard-score.