Cross-Domain Argument Quality Estimation (2023)

Abstract: Argumentation is one of society’s foundational pillars, and, sparked by advances in NLP, and the vast availability of text data, automated mining of arguments receives increasing attention. A decisive property of arguments is their strength or quality. While there are works on the automated estimation of argument strength, their scope is narrow:They focus on isolated datasets and neglect the interactions with related argument-mining tasks, such as argument identification and evidence detection. In this work, we close this gap by approaching argument quality estimation from multiple different angles:Grounded on rich results from thorough empirical evaluations, we assess the generalization capabilities of argument quality estimation across diverse domains and the interplay with related argument mining tasks. We find that generalization depends on a sufficient representation of different domains in the training part. In zero-shot transfer and multi-task experiments, we reveal that argument quality is among the more challenging tasks but can improve others. We publish our code at https://github.com/fromm-m/acl-cross-domain-aq .

## introduction the argumentation process is one of the cornerstones of society, as it allows the exchange of opinions and reaching a consensus together. fueled by advances in natural language processing, recent years have witnessed the advent of argument mining (am), i.e., the field of automated discovery and organization of arguments. am is helpful over various scenarios, reaching from legal reasoning @xcite @xcite to supporting the decision-making process of politicians @xcite @xcite @xcite . thus, there is a flurry of works on identification of arguments from text @xcite @xcite and retrieval of them @xcite @xcite @xcite . since arguments often have to be weighed against each other, a central property of arguments is their argument quality (aq) or convincingness, i.e., their (perceived) strength. while the ancient greeks @xcite already discussed the constituents of strong arguments, automated estimation is a relatively uncharted field. due to the high subjectivity of argument strength @xcite @xcite @xcite , obtaining high-quality annotations is challenging, cf. section 1. in this light, a legitimate question is the reliability and robustness of the existing approaches for estimating aq and their applicability in real-life scenarios. existing aq benchmark datasets are often restricted to a single domain @xcite or/and make different assumptions about factors impacting the aq. thus, enabling transfer between sources and datasets appears especially appealing, but existing works @xcite @xcite cease to provide detailed studies thereupon. in this work, we thus investigate for the first time the automatic evaluation of the quality of arguments from a holistic perspective, bringing together various aspects. first, we evaluate whether aq models can generalize across datasets and domains, a crucial feature for deployment in the diverse environments encountered in relevant real-world applications. next, we investigate the hypothesis of whether models for related argument mining tasks inherently learn the concept of argument strength without being explicitly trained to do so by evaluating their zero-shot performance for estimating aq. a in summary, our contributions are as follows: • as far as we know we are the first to study the generalization capabilities of aq prediction models across different datasets and aq notions. ## generalization across argument quality corpora high-level applications such as argument retrieval @xcite @xcite @xcite and autonomous debating systems @xcite require reliable argument quality (aq) models to select strong arguments among the relevant ones. the research community has identified this gap and proposed and evaluated different automated models for aq estimation @xcite @xcite . however, aq is often captured differently due to its high subjectivity, e.g., absolutely as a continuous score or relative to other arguments by pairwise comparison. consequently, many publications also introduced their own corpus with individual annotation schemes capturing different notions of aq. while they have compared multiple aq estimators against each other within a single corpus, there is a lack of cross-corpora empirical evaluations. thus, the robustness of predictions across datasets remains largely unexplored, which poses a severe challenge for reliable real-world applications integrating diverse data sources. to assess the generalizability capability of aq estimation models, we designed a series of experiments across all four major aq datasets to answer the following research questions: ## zero-shot-learning in argument mining in this section, we investigate whether explicit argument quality (aq) corpora are a necessity or whether the task of aq can also be solved by transferring from other related argument mining tasks such as argument identification (aid) or evidence detection (ed), in contrast to the relatively new task of automatic aq estimation, other argument mining (am) tasks already offer a broad range of large datasets that cover different domains and annotation schemes. moreover, the agreement between the annotators is higher on the other tasks, as aq is highly subjective @xcite @xcite @xcite . therefore, a successful transfer from related tasks to the target task of aq would represent a significant advance in the field. to this end, we investigate the zero-shot capability of am models across different corpora and different am tasks. to the best of our knowledge, we are the first to compare am task similarity by providing a first study on how individual tasks can benefit from each other. in particular, we aim to answer the following guiding research questions: while not a primary focus of this work, for completeness, we also provide experimental results for the reverse direction of transferring from aq estimation to the other tasks. ## conclusion we see this work as a fundamental step towards a more holistic view of argument quality (aq). we have shown that for good generalization across individual aq corpora, a match between the source and target domain of the arguments is essential. in contrast, diversity in aq concepts does not hinder generalization but rather enriches it. the target domain has a minor impact with a sufficiently broad coverage of different domains and adequate size. this insight is directly applicable to practical applications: the advantages of different aq notions allow the direct integration of different data sources, which is a prerequisite for handling the input from different domains encountered, e.g., by general-purpose argument retrieval engines. moreover, we were able to elucidate the relationship between aq's and other argument mining (am) tasks, such as evidence detection (ed) and argument identification (aid). our zero-shot transfer experiments showed that the concepts learned for one of the tasks are sufficient to solve the other to some extent without explicitly being trained for it. by comparing the results obtained, we con-clude that aid and ed are more closely related to each other than to aq and are per se also easier to transfer to. the multitask experiment further emphasized this, where aq could gain less from the other tasks than vice-versa. thus, an important open question is how to enable a more successful transfer to aq, extending beyond the three tasks we studied in this work.