Federated Learning for Exploiting Annotators’ Disagreements in Natural Language Processing (2024)

Abstract: The annotation of ambiguous or subjective NLP tasks is usually addressed by various annotators. In most datasets, these annotations are aggregated into a single ground truth. However, this omits divergent opinions of annotators, hence missing individual perspectives. We propose FLEAD (Federated Learning for Exploiting Annotators’ Disagreements), a methodology built upon federated learning to independently learn from the opinions of all the annotators, thereby leveraging all their underlying information without relying on a single ground truth. We conduct an extensive experimental study and analysis in diverse text classification tasks to show the contribution of our approach with respect to mainstream approaches based on majority voting and other recent methodologies that also learn from annotator disagreements.

## introduction artificial intelligence (ai) and in particular natural language processing (nlp) are dominated by data-driven approaches that often require datasets with human judgments @xcite . the difficulty of annotating data is magnified in nlp due to the inherent ambiguity of text @xcite and the subjectivity concerned in the evaluation of its meaning, which often depends on the interpretation of individual annotators @xcite . in other words, the annotation may be a reflection on some private state caused by emotions, sentiments, hate, or opinions of the author @xcite . this subjectivity is present in many nlp tasks such as sentiment analysis @xcite , offensive language detection @xcite and hate speech analysis @xcite , to name a few. the participation of more than one human annotator per data instance is a common strategy to mitigate the ambiguity and subjectivity of language @xcite . then, each item is adjudicated a gold label. this implies that a ground truth exists, which does not usually fit the real practice of text annotation in which disagreements are frequent among annotators @xcite @xcite . there are several approaches to adjudicate a gold label overcoming the disagreement among annotators @xcite : (1) approaches which simply aggregate crowd annotations into (typically, one) gold label for each instance; (2) approaches which assume a gold label for each item but consider disagreement to filter or weigh items when the true label is uncertain; (3) approaches for directly learning a classifier from crowd annotations; and (4) approaches that train a classifier by combining both hard labels and soft labels obtained from crowd annotations. in this paper, we argue that integrating disagreement into the learning process brings about clear benefits, as the data perspectivism paradigm advocates @xcite . instead of relying on a single aggregated label, we can exploit all annotator' opinions by leveraging disagreement among annotators. to this end, we propose flead @xcite for exploiting annotators' disagreements), a methodology that separately model each annotator and consolidates all these annotator-specific models into a global model that integrates all the annotators perspectives. our solution relies on federated learning to model each annotator's behavior and to summarize them into a global model. hence, our methodology does not rely on a single ground truth, but instead exploits the information provided by each annotator. our fl-based methodology requires datasets with the labels of different annotators, which is not a common practice in nlp, as the perspectivist data manifesto foot_0 highlights. the emergence of this paradigm has led to the construction of datasets where individual annotator information is provided. nonetheless, the availability of resources is mainly skewed to the english language. thus, as an additional contribution of this paper we have created and annotated the multilingual sentiment analysis dataset sentimp for english, spanish, and greek. finally, we evaluate our methodology on this and other datasets from subjective nlp tasks, and we compare it with other approaches taking into account all the annotators' information @xcite . the results highlight the benefits of our approach with respect to dominant paradigms and previous work on modelling disagreement. to better understand the behavior of the flead methodology, we perform an extensive analysis, including targeted ablations on the components of our proposed methodology. ## related work the quality of supervised learning models in nlp depends on the quality of the annotation of the datasets. this paradigm requires a human interpretation of annotation guidelines and text content that may cause disagreements among the annotators @xcite @xcite . in many cases, the disagreements are considered to be noise, and they tend to be filtered out by adjudicating a single gold label to each item. however, the use of disagreement as learning signal has been proved useful in nlp and other areas of artificial intelligence @xcite . @xcite went a step further defining and providing a taxonomy of different types of disagreements among annotators, including an offensive language classification case study on the md-agreement dataset @xcite . there are various strategies to learn from crowd annotations. the prevalent method in the literature consists of the aggregation of the annotators judgments into a single label @xcite , for example via a majority vote. depending on the type of the annotation, other heuristics to reduce the noise of the annotations are the following: weighting labels according to probability distributions @xcite ; re-annotation @xcite ; filtering hard items @xcite ; adapting the original labels to probabilistic ones based on the labels of all annotators @xcite @xcite ; adding a specific layer in an end-to-end model to learn the individual behavior of the annotators @xcite @xcite ; or adding information about the annotators and labels into the model @xcite . in our case, and building on the success of language models, we propose a single model that learns from all the non-aggregated annotations, without altering the language model or the individual labels. however, our proposed flead methodology can be applied to any context with multiple annotations, regardless of the task and the underlying learning model. most similar to our methodology are the approaches presented by @xcite , who built their proposal upon different aggregation strategies: (1) ensemble, where a model is learned for each annotator and models predictions are aggregated at the end; (2) multi-label, in which all possible labels are processed by a single model, effectively converting the problem into multi-label classification; and (3) multi-task, where the labels of each annotator are considered as an independent classification task. in all cases, the last step is based on a majority vote, which resembles the traditional practice of adjudicating a gold label. we show an overview of the baselines and our proposed methodology, which we will explain in more detail in the following section, in figure 1 . 2022 ). on the right, illustration of the flead methodology with three clients and a central server. for simplicity, in the figure we assume three annotators: a, b, and c. each annotator is shown in a different color and each data instance x i refers to a classification training example and its corresponding annotator labels l i . finally, each client is depicted by a language model. ## methodology the flead methodology is built upon fl with the objective of learning from the disagreement among annotators, building a global model that integrates the perspectives of each of them. while the fl-based methodology is flexible to be applied to other tasks, in this paper our goal is to build a text classification model. in the following we formally define fl (section 3.1) and present the details of the flead methodology (section 3.2). ## experimental framework in this section we present the text classification experimental setup to test our flead methodology. ## experimental results in this section we present the results with the aim of evaluating our flead methodology in a multi-annotation context with both the standard single label evaluation protocol and an additional setting in which disagreement between annotators is taken into account in the evaluation. we use standard text classification evaluation metrics (accuracy and macro-f1) in section 5.1, and metrics specifically designed for disagreement between annotators in section 5.2. ## analysis in this section, we analyze the source of the flead methodology improvement through ablation studies in section 6.1, the effect of the disagreement between annotators in the gain of performance of each approach in section 6.2, and the effect of the learning model used in section 6.3. we also perform a qualitative study in section 6.4 with human annotators that supports the operation of the flead methodology. ## limitations we have identified five main limitations of our proposal and evaluation: ## conclusions in this paper, we proposed a text classification method to leverage the information from all annotators separately. to this end we put forward flead, a methodology based on fl and considering each annotator that participate in the annotation of a dataset as a federated client. thus, the labels of each annotator are independently learned and aggregated in a global or final model. in general, our methodology shows promising results and prove that fl can be used beyond protecting data privacy, in this case to learn from the disagreements among annotators in subjective tasks. finally, we performed an in-depth evaluation and analysis to understand the different component of our methodology, with the following conclusions: (1) the results on several multilingual datasets of subjective text classification tasks show that leveraging information from all the annotators is indeed beneficial and enhances the classification performance; (2) our ablation analysis highlights that the improvements are largely due to the fl operation, in addition to other side benefits that our methodology offers; (3) the qualitative analysis shows that the external annotators generally agree more with the errors made by our flead-based model, in comparison to the models trained on a single ground truth.