Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It’s Best to Relate Perspectives! (2023)

Abstract: Many annotation tasks in natural language processing are highly subjective in that there can be different valid and justified perspectives on what is a proper label for a given example. This also applies to the judgment of argument quality, where the assignment of a single ground truth is often questionable. At the same time, there are generally accepted concepts behind argumentation that form a common ground. To best represent the interplay of individual and shared perspectives, we consider a continuum of approaches ranging from models that fully aggregate perspectives into a majority label to “share nothing”-architectures in which each annotator is considered in isolation from all other annotators. In between these extremes, inspired by models used in the field of recommender systems, we investigate the extent to which architectures that predict labels for single annotators but include layers that model the relations between different annotators are beneficial. By means of two tasks of argument quality classification (argument concreteness and validity/novelty of conclusions), we show that recommender architectures increase the averaged annotator-individual F1-scores up to 43% over a majority-label model. Our findings indicate that approaches to subjectivity can benefit from relating individual perspectives.

## introduction there is inherent subjectivity in many annotation tasks in natural language processing @xcite @xcite . recent work has criticized the widely adopted approach of viewing variation in human labeling behavior as "noise" @xcite , advocating against approaches that disregard the richness of human annotations and perspectives by aggregating them into a single label. in fact, it has been argued that disagreement should not be regarded * authors contributed equally to this work. as a problem, but rather as a chance to end up with more user-adaptable classifiers, giving voice to minorities as well @xcite . in this paper, we hypothesize that machine learning can best address variation in human labeling by both accounting for subjective perspectives of individuals and for more objective concepts that build a common ground between annotators. a prime example of the interplay of individual and shared perspectives is the understanding and assessment of argumentation and, in particular, of argument quality @xcite . given the subjectivity that many concepts of argument quality face, it has been generally shown that its annotation often results in only fair to moderate interannotator agreements @xcite @xcite . for example, @xcite determined a global value of argument quality by asking annotators "if they would recommend a friend to use that argument as is in a speech supporting/contesting the topic". even in more well-defined aspects of argument quality, such as the sufficiency of an argumentative conclusion, @xcite observed "hard cases" in which labeling depends on subjective interpretations of keywords without being able to agree on a single ground truth. however, alongside "hard cases" of irreconcilable individual perspectives, in assessing argument quality we regularly also observe uncontroversial cases such as in human annotation of the validity and novelty of conclusions @xcite or the concreteness of arguments @xcite . in order to verify our hypothesis, we investigate the impact of different architectural design choices on the ability of models to predict the labels of single annotators. to this end, we look at a continuum of model architectures between such that fully suppress annotation variation by learning from an aggregated label and such that are specifically tailored to each annotator. along this continuum, we focus in particular on models that include components that involve the perspectives of single annotators @xcite , or are able to find and relate similar annotation behavior shared among different annotators using a recommender approach @xcite . figure 1 gives a first overview of the continuum of concepts and models, which we explain in detail in section 3. drawing on two tasks of argument quality, namely argument concreteness and validity/novelty of conclusions, we show that architectures inspired by recommender systems perform best in learning from disagreement. we attribute this to the identification of patterns in annotation behavior across individual annotators while being faithful to single choices. our main contributions are: extending previous work that proposed to take subjectivity into account by predicting the degree of expected label variation by @xcite , our study presents the first approach in the field of argument mining to learn directly from the individual human labels. beyond the specific contributions mentioned above, our work might encourage future research in argument mining and other fields to step away from systems that suppress valid perspectives by relying on a single aggregated label, and instead moving on towards systems that cover the multi-faceted spectrum of opinions. ## methodology we aim at the comparison of different paradigms that can be applied in order to model a classifier for predicting individual labeling decisions in subjective annotation tasks. to this end, we define a model spectrum ranging from a purely annotatorspecific model design (denying that there is anything learnable which is shared among the annotators, i.e., the objective grounding) to a purely annotator-agnostic model design which only considers the majority vote (denying that there is anything learnable which is individual for an annotator, i.e., the subjective grounding). we also include models that exploit the characteristics of both sides of the coin. these hybrid models combine model components that are shared for all annotators as well as model components that are specific for each annotator. in this way we can explore different architectural design choices along the above mentioned continuum, varying the degree to which the shared labeling behavior of certain groups of users is modeled in the architecture. the proposed model spectrum complements existing taxonomies of learning from disagreement. @xcite distinguish four categories of approaches, one of them being "learning directly from crowd annotations". within this category, our proposed spectrum allows to further differentiate how models process individual annotations. figure 1 shows our broad bandwidth of approaches, which covers five different approaches to classification: a majority vote model, per-annotator models, and three hybrid approaches, namely a model with annotator-specific classification heads, a recommender system with a shared text encoder, and a recommender system with annotatorseparated text encoders. these will be described in more detail in the following subsections. ## results & evaluation we provide results for the five models considered along the spectrum of architectures as outlined in figure 1 . we evaluate these architectures on two levels using the previously described two-step calculation of model scores. in section 5.1, we use the annotator-average scores in order to evaluate models across individual predictions (suitable for use cases in which a global indicator of performance is desired). to have a more fine-grained examination (suitable for use cases aiming to maximize argument quality for an individual user) we evaluate against the non-averaged annotator-level scores in section 5.2. see appendix b and c for full results and additional recommender configurations. ## qualitative analysis: case study the evaluated models show different behaviors that result from how they incorporate individual annotators. these differences can best be illustrated by discussing controversial examples. we consider in particular an example from the concreteness dataset in which the full range of possible labels is provided by annotators. the majority model, by definition, predicts the same label for each annotator. the predicted value, intermediate concreteness, is plausible in this case at face value: examining further examples from the dataset shows a tendency for short texts to be annotated as less concrete. however, for two annotators this is an example of high concreteness, despite its brevity. thus, there is no clear majority label in this controversial case, so this prediction misses three out of five annotators. sharerec shows the same prediction pattern as the majority model. this is in line with the model's discussed tendency to predict uniform labels per example. the model stresses the common ground based on its shared text representation and picks a plausible label. the perannnotator and sepheads models show more diversity in their predictions. this makes sense, as their architectures contain components for more variation isolated from other annotators via separate classification heads or models. this example underlines, however, that variation does not necessarily lead to more accurate representation of annotators overall. both models, again, only manage to predict two out of five labels correctly. thus, they are not more accurate than the uniform predictions. seprec, in contrast to all other models, is very close to predicting the actual distribution of labels. for the one annotator it misses, the tendency of the label (less concrete) is correct. seprec is thus the only model to predict the class "high concreteness" correctly for two annotators. in this example, the model picks up a peculiarity of argument concreteness in public participation related to traffic planning: while generally short argumentative units tend to be less concrete, they might be very concrete to some annotators. for example, annotators might have specific knowledge about local contexts, such as that there are certain bike lanes in a city controversial example: "bike lane blocked by cars" ## conclusion & future work in this work, we have proposed a general framework to investigate the performance of different architectures on subjective annotation tasks and similar cases where different legitimate perspectives exist on the appropriate labeling decision across annotators. we have, in particular, highlighted architectures predicting labels for individual annotators along a continuum from fully annotator-specific architectures to architectures that rely only on aggregated annotations that completely disregarded individual annotators. our main focus has been on hybrid architectures along these extreme ends of the continuum that model commonalities in annotation behavior across individual annotators. regarding different aspects of argument quality (concreteness, conclusion validity, and conclusion novelty), we have shown that such hybrid architectures are best suited to classify the range of opinions. our results show that recommender architectures that use an annotator-shared llm for encoding the text excel in this setting in terms of maximizing the f 1 -score averaged over all individual annotators up to 43% over a majority-label model. nevertheless, further analysis going beyond this averaged score has emphasized the importance of having an annotator-tailored text encoding to capture the different reading nuances in "hard cases", resulting in a variance of predicted classes. our work provides a starting point for including human label variation into subjective tasks in the field of argument mining, such as the assessment of argument quality, without denying generally accepted objective concepts of argumentation. still, further work is needed to fully understand the reasons for disagreement.