xiacui at S em E val-2023 Task 11: Learning a Model in Mixed-Annotator Datasets Using Annotator Ranking Scores as Training Weights (2023)

Abstract: This paper describes the development of a system for SemEval-2023 Shared Task 11 on Learning with Disagreements (Le-Wi-Di). Labelled data plays a vital role in the development of machine learning systems. The human-annotated labels are usually considered the truth for training or validation. To obtain truth labels, a traditional way is to hire domain experts to perform an expensive annotation process. Crowd-sourcing labelling is comparably cheap, whereas it raises a question on the reliability of annotators. A common strategy in a mixed-annotator dataset with various sets of annotators for each instance is to aggregate the labels among multiple groups of annotators to obtain the truth labels. However, these annotators might not reach an agreement, and there is no guarantee of the reliability of these labels either. With further problems caused by human label variation, subjective tasks usually suffer from the different opinions provided by the annotators. In this paper, we propose two simple heuristic functions to compute the annotator ranking scores, namely AnnoHard and AnnoSoft, based on the hard labels (i.e., aggregative labels) and soft labels (i.e., cross-entropy values). By introducing these scores, we adjust the weights of the training instances to improve the learning with disagreements among the annotators.

## introduction annotated datasets are fundamental for any machine learning model. traditional supervised machine learning models are heavily based on welllabelled datasets. fine-tuning and validation could not exist without them either. the cost of manual labelling using domain experts is generally high to ensure the annotation quality. with the recent popularity of crowd-sourcing, the cost has become lower. however, the reliability of annotators remains a question. even with domain experts, disagreements are commonly observed in the multiple annotations for the same task due to several circumstances. for example, the task is complex with many factors. it may require annotators' expertise, such as predicting mental health issues @xcite , or subjective such as detecting emotions or other polarised opinions @xcite . the model trained with unreliable labels can potentially increase the cost by human postvalidation. in this paper, we focus on a specific sub-problem of learning with disagreements in mixed sets of annotators from different backgrounds, and it could be raised by crowd-sourcing or a mixture of the crowd and expert labels. we propose a training strategy to weigh the instance by computing the ranking score of the annotator. more specifically, our method is proposed to trust more instances with higher annotator ranks during the training stage. hence, we propose two functions to compute annotator ranking scores: annohard and annosoft. as suggested in the naming, one is computed using the hard labels, and the other uses the soft labels. this paper follows the task definition to consider cross entropy values as the soft labels and the majority voting labels as the hard labels. by participating in this shared task, we made an initial attempt using heuristic methods and explored the performance of introducing annohard and annosoft to various learning algorithms. a list of main contributions is summarised as follows: ## background in this section, we specifically focus on the prior works driven by the dependent scores of annotators. as the annotator is the origin of the disagreements, these works have demonstrated the design of training strategies to address this factor. subjective tasks (e.g., detecting hate speech or offensive languages) generally have different definitions among different communities @xcite . developing an automatic system based on the hard labels would substantially impact a specific community. to overcome this issue, @xcite proposed grouping the annotators into two groups using the average polarization index (p-index). however, polarity is a task-specific factor. @xcite proposed computing the two annotators' agreements using f1 scores between them and label confusion probabilities for part-of-speech (pos) tagging. the method was extended to dependency parsing in @xcite . we aim to propose a method to increase the feature and task generality to the dedicated problem of disagreements. machine learning models are trained by taking the penalties for misclassifications into account. a loss function is applied to map the distance between the current output and the ground truth. each training instance could possibly affect the prediction result @xcite . introducing instance weights to cost-sensitive classifiers is a popular solution. it intends to increase or decrease the weights of some instances in nlp tasks to enrich the model performance @xcite @xcite . incorporating the annotator ranking scores, we propose annohard to tackle the annotator's agreement with the aggregative labels and annosoft using the cross entropy values and the probabilities of the aggregative labels over the annotators. in this shared task, we are given four datasets for individual classification tasks in various scenarios: different languages (i.e., arabic and english) and text formats (i.e., short texts and dialogues). we used only the provided datasets during the practice and evaluation phases and followed the official splits. no additional training data was introduced to boost the performance. the detail of the datasets can be found in section 4. ## methods subjective tasks rely heavily on the annotators' personal opinions and their ability to perform a specific task. we focus on a particular case of a mixed set of annotators without any pre-knowledge about the annotator (e.g., background and expertise). in this case, not all instances share the same set of annotators. for example, instance #1 is annotated by annotators 1, 2, and 3; instance #2 is annotated by annotators 2, 3, and 4; and instance #3 is annotated by annotators 3, 6, 7, and 8 etc. for simplicity, we represent each instance using the term frequency-inverse document frequency (tf-idf) or pre-trained word embeddings. the detail of the features we selected for the experiments can be found in section 4.3. to incorporate the performance of the different sets of annotators, the first step is to compute each annotator's ranking score on a specific task (section 3.1). second, we combine the ranking scores of all annotators working on the same training instance. then, we adjust the training instance weights by the sum of the annotator ranking scores (section 3.2). ## experiments the system was developed using the le-wi-di dataset @xcite , which includes 4 sub datasets: armis (almanea and poesio, 2022), convabuse (cercas @xcite , hs-brexit @xcite and md-agreement @xcite . the statistics of the train or development split for each dataset can be found in ## limitations and future directions due to the time limitation, this system was developed using traditional machine learning algorithms or an mlp. the proposed solution is limited to a annohard is sensitive to the workload of the annotators, as stated in the previous section. it sometimes fails to reflect the annotator's ability to perform a specific task with a light workload. hence, it is limited to a sizeable amount of data labelled by an annotator. an alternative strategy might be introducing a constraint about the workload of the annotators to the ranking function. in this case, the annotators labelling more instances following the majority would be encouraged, whereas those with fewer labelled instances would be downgraded. on the other hand, model training was carried out using hard labels, introducing a variable of the sum of the annotator ranking scores for each instance. the feature and algorithm selection was based on the hard labels as well. we could train the model and select the best candidate from these combinations using the soft labels. given these limitations in the proposed method, we hope to encourage the community to explore further the idea of using the interpretability linking to the heuristics. ## conclusions in this paper, we proposed two heuristic functions to compute the annotator ranking scores. we used cost-sensitive learning algorithms and introduced a cost variable for each instance into the training step. the variable is computed by the sum of the ranking scores of the participating annotators for each instance. we discussed the advantages of using random forest as the learning algorithm for the short text datasets and the limitations of the proposed methods. compared to a typical classifier, we observed slight improvements in the soft and hard evaluation with the proposed function annosoft.