Is Your Perspective Also My Perspective? Enriching Prediction with Subjectivity (2022)

Abstract: Although argumentation can be highly subjective, the common practice with supervised machine learning is to construct and learn from an aggregated ground truth formed from individual judgments by majority voting, averaging, or adjudication. This approach leads to a neglect of individual, but potentially important perspectives and in many cases cannot do justice to the subjective character of the tasks. One solution to this shortcoming are multi-perspective approaches, which have received very little attention in the field of argument mining so far. In this work we present PerspectifyMe, a method to incorporate perspectivism by enriching a task with subjectivity information from the data annotation process. We exemplify our approach with the use case of classifying argument concreteness, and provide first promising results for the recently published CIMT PartEval Argument Concreteness Corpus.

## introduction the analysis of arguments and especially their properties is challenging and often subjective, which renders the creation of suitable language resources for argument mining difficult @xcite . uniform annotation often requires intensive training, and this costly approach has been shown to regularly result in at most moderate agreement among annotators @xcite @xcite . alternative approaches such as crowd-sourcing share this problem, especially for demanding tasks like argument quality @xcite . although the lack of consensus might clearly indicate that the annotation task is either ambiguous @xcite , too complex @xcite , or influenced by variables such as demographics and individual bias @xcite , the established procedure is to aggregate the individual judgments into a single ground truth at the end of the annotation process (by majority vote, averaging, or adjudication). learning from aggregated ground truth has several drawbacks. minority voices are ignored, however valuable they may be, and only those in line with the mainstream are heeded @xcite . this rises also a fairness concern, as certain sociodemographic groups and their perspectives may be underrepresented @xcite . finally, it is questionable whether the assumption of a single truth, i.e., that there is only one correct label for an example, holds at all for subjective tasks @xcite . therefore, the question of multi-perspective approaches arises @xcite . @xcite introduced the paradigm of data perspectivism in order to "integrate the opinions and perspectives of the human subjects involved in the knowledge representation step of ml processes". one example for perspectivist data is argumentation @xcite . however, many popular algorithms require a single ground truth to which the model can adapt. in this paper, (i) we thus introduce a method that combines collaborative and subjective viewpoints by complementing an aggregated label with a subjectivity score. more specifically, perspectifyme proposes to add the prediction of how perspectivist an input is as an additional sub-task. providing this information can for example help a human decide when to rely on their own perspective. (ii) to exemplify our approach, we draw on a recently published perspectivist dataset for argument concreteness in public participation processes @xcite . we provide several baselines based on our proposed method for this subjective task. while these are certainly extendable, they already show promising results for automatic classification by concreteness. (iii) to the best of our knowledge, we are the first to automatically classify arguments in an explicitely perspectivist manner. ## related work basile et al. (2021) provide a nice summarization of the previous work towards perspectivist machine learning, dividing the field in two groups. the first aims at building unified ground labels that involve perspectivism by either only keeping instances on which a statistically significant majority agrees @xcite , by computing a weighting according to annotator reliability @xcite @xcite , by replicating or weighting instances using provided labels or disagreement measures @xcite , or by participatory consensus building @xcite . the second group incorporates the perspectivism into the core machine learning workflow by either training an ensemble of models that rely on different ground truths @xcite , by soft loss learning @xcite @xcite , or by utilizing multi-task learning @xcite @xcite @xcite . our approach ties into the latter idea by transforming the original problem into multiple subtasks. however, multi-task learning approaches for multi-perspectivist tasks have primarily aimed at improving model performance. to do so, the aggregated ground truth is learned along with the distribution of individual labels. instead, we focus on outputting an indication of how perspectivist the model predictions are (namely, by adding a subjectivity score) to help interpret the results. the only previous studies that specifically address argument mining are, to the best of our knowledge, two recently published non-aggregated datasets: qt30nonaggr @xcite and the cimt parteval argument concreteness corpus @xcite . ## use case: argument concreteness in public participation public participation is a means regularly used by democratic authorities to involve citizens in policymaking processes @xcite . the manual evaluation workflow often includes reading the contributions, detecting duplicates, identifying arguments and opinions, and thematically clustering content before drawing conclusions from the input @xcite . one solution to reduce the workload of human evaluators is machine learning @xcite . although there is a general consensus that such important democratic processes cannot be fully automated, automating sub-tasks such as topic classification or argument detection and analysis can support the evaluation. argument mining for public participation has received considerable attention @xcite @xcite @xcite . while works such as @xcite and @xcite have already addressed the evidence and verifiability of propositions, there has been no attempt to automatically classify their concreteness. predicting the concreteness of propositions can assist a human analyst to speed up the evaluation by ranking them, since less concrete ideas tend to be more laborious to evaluate @xcite . the cimt parteval argument concreteness corpus @xcite provides argumentative text units (atu) in german extracted from mobility-related public participation processes. each atu consists of one to several sentences, consecutive in the original document, and a tag that describes the argumentative function (major positions: proposed courses of action and policy options or premises: attacking/supporting reasons). in total, the dataset contains 1, 127 atus, 614 of which are major positions and 513 are premises. these atus have been categorised into three different degrees of concreteness: ## perspectifyme previous work has incorporated perspectivism through distributions over individual labels. however, such distributions may be of limited use when provided to a human as a direct output, e.g. in human-machine interactions. in particular, providing such a diversity of perspectives that might apply (from the annotators' point of view -not necessarily from the point of view of the particular user) can be too complex and potentially confusing. for items that trigger a subjective perception, it might make more sense (e.g., in a use case like ours) to inform the user about this and let them decide whether to make their own assessment or to go along with the collaborative opinion. task label support sub-task t h : concreteness high 709 (62.9%) intermediate 336 (29.8%) low 82 (7.3%) sub-task t s : subjectivity objective 478 (42.4%) rather objective 244 (21.7%) rather subjective 275 (24.4%) subjective 130 (11.5%) ## discussion the evaluation of public participation can be supported by machine learning in a human-machine interaction. not only machine prediction, but also pointing out cases where the user might potentially disagree can help with good evaluation practice. perspectives can differ for a variety of reasons. first, it is due to the task itself, which is subjective. in addition, personal biases of the analyst may also contribute, such as their professional background (e.g., in our application case, whether they studied urban planning or administrative sciences). furthermore, process-related demands on the evaluation may require the analyst to adjust their view. all these factors argue for a perspectivist approach. as exemplified, our method can be integrated into workflows by adding a model for the sub-task of predicting subjectivity. while t h reflects the prevailing opinion of the crowd, t s can indicate how different coders' perceptions were when rating the unit -a valuable piece of information that is lost in non-perspectivist approaches. however, a potential barrier to applying our method to further use cases is the need for a non-aggregated dataset. the publication of annotations on an individual level is not yet common @xcite . we found that objective atus (regarding their concreteness) can already be filtered out with an f 1 score between 0.73 and 0.80, depending on the granularity level (cf. concerning the original task of classifying the concreteness of arguments, the degree of concreteness (hard label) could be predicted with an accuracy of 0.80 and an f 1 of 0.67, which can already be helpful for supporting the manual evaluation of public participation processes. ## conclusion & future work we introduced perspectifyme, a simple method to include perspectivism in machine learning workflows. using argument concreteness as an example, we have shown that our baseline approaches can assess the subjective perception of atus. in future work, we plan to apply advanced multitask learning models as previous work has shown that they can lead to an increase in performance @xcite . furthermore, we have tailored the transformation of the spectrum of annotations into a subjectivity score specific to the use case at hand. it would be of great interest to develop a more general (task-independent) algorithm.