Towards a Perspectivist Turn in Argument Quality Assessment (2025)

Abstract: The assessment of argument quality depends on well-established logical, rhetorical, and dialectical properties that are unavoidably subjective: multiple valid assessments may exist, there is no unequivocal ground truth. This aligns with recent paths in machine learning, which embrace the co-existence of different perspectives. However, this potential remains largely unexplored in NLP research on argument quality. One crucial reason seems to be the yet unexplored availability of suitable datasets. We fill this gap by conducting a systematic review of argument quality datasets. We assign them to a multi-layered categorization targeting two aspects: (a) What has been annotated: we collect the quality dimensions covered in datasets and consolidate them in an overarching taxonomy, increasing dataset comparability and interoperability. (b) Who annotated: we survey what information is given about annotators, enabling perspectivist research and grounding our recommendations for future actions. To this end, we discuss datasets suitable for developing perspectivist models (i.e., those containing individual, non-aggregated annotations), and we showcase the importance of a controlled selection of annotators in a pilot study.

## introduction the question of "what makes an argument good" is at the core of computational argumentation (ca), the area of natural language processing (nlp) dealing with the mining, assessment, and generation of arguments @xcite . while rooted in well-established theories, argument quality (aq) still exhibits a high degree of subjectivity in perception. this degree may vary across quality aspects; for example, evaluating an argument's benefit in agreement-seeking discussions is considered less subjective than assessing its effectiveness @xcite . the ca community is aware of the variance in annotators' perception @xcite @xcite . not least, this is documented by the generally moderate inter-annotator agreement in aq annotations -a widely accepted condition that authors commonly attribute to the subjective nature of the task (e.g., @xcite @xcite ). 1 @xcite explicitly question whether an aggregated ground truth is suitable to model aq. meanwhile, the nlp community has started to undergo a fundamental change in the way it deals with subjective tasks. while aggregated ground truth and an according model alignment were long standard, more recent work calls for this course to be reconsidered @xcite @xcite : rather than eradicating any existence of annotator disagreement, the perspectivist turn embraces the coexistence of perspectives @xcite @xcite . this transformation implies the acceptance of variations in data annotation (through non-aggregated datasets) as well as the consideration of heterogeneity in modeling and evaluation @xcite @xcite . we postulate that the perspectivist turn in nlp lends itself as a natural solution to face the issue of subjectivity in modeling aq. not only does it have a better shot at promoting diversity and fairness in aq assessment, such as allowing for valid but minority voices @xcite . it is also likely to be more robust in modeling perceptions of aq across (changing) societies (e.g., today's minority groups may become tomorrow's majority) and target audiences. yet, the perspectivist turn so far had only minimal impact on aq. presumably, one reason for the limited modeling of perspectives in aq is the lack of datasets designed for this purpose. preference has been given to aggregated annotations, whereas individual labeling decisions were often not communicated (e.g., @xcite @xcite . while a solution may be new datasets, the annotation of argumentation phenomena is highly complex and costly. we therefore deem it essential to first gain an overview of the options that existing datasets already offer for developing perspectivist models. this is the goal of the paper at hand. we provide a systematic literature review of 103 aq datasets and their properties. 2 crucially, to support the perspectivist turn, our collection includes meta-information about annotators and the availability of non-aggregated annotations. while only 24 datasets come with the latter, 14 of them seem relevant to the perspectivist turn. in a pilot study, we conduct a statistical analysis of the disagreement patterns in four of them. we conclude by highlighting the opportunities of available datasets and discuss challenges, for example a lack of transparency and socio-demographic diversity. contributions (1) we release an extensive database with 32 types of meta-information about 103 aq datasets. (2) we review the multitude of annotated aq categories (what is annotated) and consolidate them in an overarching taxonomy to foster comparability and interoperability. (3) we perform a comprehensive meta-analysis of annotators (who annotates) across the datasets, uncovering a lack of transparency and socio-demographic diversity, promoting bias in aq datasets and models. (4) we deep-dive into the 24 datasets with non-aggregated labels both qualitatively and quantitatively, and discuss their potential for a perspectivist turn in aq. ## datasets: annotations & annotators the 98 identified papers introduce 103 aq datasets in total. a complete list including the mapped aq categories is in appendix e. in what follows, we provide an overview of the quantitative properties overall quality rationality interactivity construct iveness civility alternative forms of c. del. norms com. good dialectical quality rhetorical quality logical quality deliberative quality overall arg. quality cogency local acceptab. local relevance local sufficiency effective ness appropri ateness clarity credibility arrangem. em. appeal reason ableness global acc. global relevance global sufficiency 20 40 60 80 100 figure 1: frequency and distribution of aq categories (major and sub-categories) as assigned to datasets, grouped by the four major categories and overall aq. ## towards perspectivist argument quality assessment developing perspectivist models requires the existence of multiple assessment perspectives. among the 103 datasets we found, only 24 come with non- ## discussion imbalance of aq datasets we identified a substantial number of datasets, covering nearly the entire taxonomy of quality categories. the two dialectically-driven dimensions (reasonableness and deliberative norms) are less represented, likely due to the reduced focus on dialogical argumentation in ca and the more recent attention to deliberation. unfortunately but not unexpectedly, we found a very uneven representation of language. together with the sparsity of languages comes a lack of cultural diversity. as arguments are perceived differently across cultures @xcite , this gap should be closed. ## conclusion a critical first step in developing perspectivist models for aq assessment is suitable data. we identify several datasets as a learning ground for selected aq categories in a non-aggregated way, while noting current shortcomings. future datasets should (1) cover a diverse set of perspectives with respect to a reference population, (2) collect annotatorspecific attributes, and (3) maintain an adequate size of total and individually labeled items. the listed desiderata indicate that the perspectivist turn in aq assessment requires resources to be invested in annotation quality and quantity. in this context, recent research on the potential of active learning methods for subjective nlp tasks may be relevant @xcite . filling the resource gap will unlock the potential of the perspectivist turn in aq assessment advocated in this paper. with suitable resources, future work can start leveraging the machine learning toolbox developed thus far in the perspectivist community (e.g., @xcite . our extensive meta-information database will facilitate aq research in general. one example is instruction-following llms, in which access to annotation guidelines is crucial @xcite . we make a significant contribution to this by expanding the number of publicly available guidelines, provided as a central listing in our database.