Claim Optimization in Computational Argumentation (2023)

Abstract: An optimal delivery of arguments is key to persuasion in any debate, both for humans and for AI systems. This requires the use of clear and fluent claims relevant to the given debate. Prior work has studied the automatic assessment of argument quality extensively. Yet, no approach actually improves the quality so far. To fill this gap, this paper proposes the task of claim optimization: to rewrite argumentative claims in order to optimize their delivery. As multiple types of optimization are possible, we approach this task by first generating a diverse set of candidate claims using a large language model, such as BART, taking into account contextual information. Then, the best candidate is selected using various quality metrics. In automatic and human evaluation on an English-language corpus, our quality-based candidate selection outperforms several baselines, improving 60% of all claims (worsening 16% only). Follow-up analyses reveal that, beyond copy editing, our approach often specifies claims with details, whereas it adds less evidence than humans do. Moreover, its capabilities generalize well to other domains, such as instructional texts.

## introduction the delivery of arguments in clear and appropriate language is a decisive factor in achieving persuasion in any debating situation, known as elocutio in aristotle's rhetoric @xcite . accordingly, the claims composed in an argument should not only be grammatically fluent and relevant to the given debate topic, but also unambiguous, selfcontained, and more. written arguments therefore often undergo multiple revisions in which various aspects are optimized @xcite . extensive research has been done on the automatic assessment of argument quality and the use of large language models on various text editing tasks. yet, no work so far has studied how to ac- original claim optimized claim 1 optimized claim 2 previous claim should humans be allowed to explore diy gene editing? debate topic optimized claim 3 humans should be allowed to explore diy gene editing. this technology could be weaponized. this technology could be weaponized and harmful to human beings. this technology could be used by criminals to create and weaponize bio-mechanisms. this technology could be weaponized, so it is important to safeguard it from being weaponized. figure 1 : examples of different optimized versions of an original claim found on the debate platform kialo. all optimizations were generated by the approach proposed in this paper, using the debate topic as context. we start by defining the task of claim optimization in section 3, and adjust the english-language claim revision dataset of @xcite for evaluation. the new task requires complementary abilities: on the one hand, different types of quality issues inside a claim must be detected, from grammatical errors to missing details. if not all quality aspects can be improved simultaneously, specific ones must be targeted. on the other hand, improved claim parts need to be integrated with the context of the surrounding discussion, while preserving the original meaning as far as possible. figure 1 shows three exemplary optimizations of a claim from the debate platform kialo. the first elaborates what the consequence of weaponization is, whereas the second rephrases the claim to clarify what weaponizing means, employing knowledge about the debate topic. the third renders the stance of the claim explicit. we observe that different ways to optimize a claim exist, yet the level of improvement differs as well. to account for the multiplicity of claim optimization, we propose a controlled generation approach that combines the capabilities of large language models with quality assessment (section 4). first, a fine-tuned generation model produces several candidate optimizations of a given claim. to optimize claims, we condition the model on discourse context, namely the debate topic and the previous claim in the debate. the key to selecting the best optimization is to then score candidates using three quality metrics: grammatical fluency, meaning preservation, and argument quality. such candidate selection remains understudied in many generative tasks, particularly within computational argumentation. in automatic and manual evaluation (section 5), we demonstrate the effectiveness of our approach, employing fine-tuned bart @xcite for candidate generation. our results stress the benefits of quality assessment (section 6). incorporating context turns out especially helpful for making shorter claims-where the topic of the debate is difficult to infer-more self-contained. according to human annotators, our approach improves 60% of all claims and harms only 16%, clearly outperforming standard fine-tuned generation. to gain further insights, we carry out a manual annotation of 600 claim optimizations and identify eight types typically found in online debate communities, such as elaboration and disambiguation (section 7). intriguingly, our approach covers similar optimization types as in human revisions, but we also observe limitations (section 7). to explore to what extent it generalizes to other revision domains, we also carry out experiments on instructional texts @xcite and formal texts @xcite , finding that it outperforms strong baselines and state-of-the-art approaches. in summary, the contributions of this paper are: quality assessment has become a key topic in computational argumentation research @xcite . various quality dimensions exist in argumentation theory, as surveyed by @xcite and assessed computationally in various works @xcite . many of them relate to quality aspects we consider in this work, from clarity and organization @xcite to the general evaluability of arguments @xcite , potential fallacies in their reasoning @xcite , and the appropriateness of the language used @xcite . recently, @xcite tackled the question whether an argumentative claim is in need of revision, whereas @xcite investigated where to best elaborate a discussion. while @xcite leverage claim generation for a refined assessment of argument quality, we are not aware of any prior work that actually optimizes arguments or their components in order to improve quality. as shown in figure 1 , there can be several ways to optimize a given text. our key idea is to select the best optimization among diverse candidates generated by a language model. prior generation work on candidate selection hints at the potential benefits of such setup, albeit in other tasks and domains. in early work on rule-based conversational systems, @xcite introduced dialogue quality metrics to optimize template-based systems towards user satisfaction. @xcite and @xcite chose the best templates for generation, and @xcite used syntactic features to rank candidates in grammar correction. recently, @xcite proposed a reference-less metric trained on manual evaluations of grammar correction system outputs to assess generated candidates, while @xcite utilize pre-trained language models to select the best candidate in textual style transfer tasks. in generation research on computational argumentation, candidate selection remains largely understudied. most relevant in this regard is the approach of @xcite which reframes arguments to be more trustworthy (e.g., less partisan). it generates multiple candidates and selects one based on the entailment relation scores to the input. extending this idea, we select candidates based on various properties, including argument quality. understanding the editing process of arguments is crucial, as it reveals what quality dimensions are considered important. for wikipedia, @xcite proposed a fine-grained taxonomy as a result of their multi-label edit categorization of revisions @xcite . the taxonomy focuses solely on the editing actions performed, such as inserting, deleting, and paraphrasing. in contrast, @xcite identified various semantic intentions behind wikipedia revisions, from copy editing to content clarifications and fact updates. their taxonomy defines a starting point for our research. not all covered intentions generalize beyond wiki scenarios, though. wikipedia-based corpora have often been used in the study of editing and rewriting, including paraphrasing @xcite , grammar correction @xcite , bias neutralization @xcite , and controllable text editing @xcite . similarly, wikihow enabled summarization @xcite and knowledge acquisition @xcite . however, neither of these includes argumentative texts. instead, we thus rely on the corpus of @xcite , which consists of revision histories of argumentative claims from online debates. whereas the authors compare claims in terms of quality, we propose and study the new task of automatically optimizing claim quality. moreoever, we see the revision types they distinguish (clarification, grammar correction, linking to external sources) as too coarse-grained to represent the diversity of claim optimizations. we refine them manually into eight optimization types, allowing for a more systematic analysis. @xcite also found low correlations between the revision types and 15 common argument quality dimensions @xcite , suggesting that they are rather complementary. primarily, they target the general form a well-phrased claim should have and its relevance to the debate. for the analysis of argumentative text rewriting, @xcite incorporated both argumentative writing features and surface changes. to explore the classification of essay revisions, they defined a two-dimensional schema, combining the revision operation (e.g., modify, add, or delete) with the component being revised (e.g., reasoning or evidence). moreover, afrin and litman (2018) created a small corpus of between-draft revisions of 60 student essays to study whether revision improves quality. however, these works do not uncover the reasoning behind a revision operation and are more geared towards analysis at the essay level. ## task and data this section introduces the proposed task and presents the data used for development and evaluation. ## approach we now present the first approach to automatic claim optimization. to account for the variety of possible optimizations, multiple candidate claims are generated that are pertinent to the context given and preserve the claim's meaning. then, the best candidate is selected based on quality metrics. both steps are detailed below and illustrated in figure 2 . ## experiments this section describes our experimental setup to study how well the claims from section 3 can be improved using our approach from section 4. we focus on the impact of candidate selection. ## results and discussion apart from evaluating the applicability of large generative language models to the task of argumentative claim optimization in general, our experiments focus on two questions: (1) does the use of explicit knowledge about text and argument quality lead to the selection of better candidates? (2) does the use of contextual information make the generated candidates more accurate and relevant to the debate? 6.1 overall claim optimization performance automatic evaluation the bart+max meaning ablation supports the intuition that the candidates with highest meaning preservation scores are those with minimal changes, if any (72% of the candidates remain identical to the input). such identical outputs are undesirable, as the claims are not optimized successfully, which is also corroborated by the low weight parameter (β = 0.01) found for the meaning preservation metric when optimizing autoscore (see section 5). ## analysis to explore the nature of claim optimization and the capabilities of our approach, this section reports on (a) what types of optimizations exist, (b) how well our approach can operationalize these, and (c) how well it generalizes to non-argumentative domains. ## conclusion with this paper, we work towards the next level of computational argument quality research, namely, to not only assess but also to optimize argumentative text. applications include suggesting improvements in writing support and automatic phrasing in debating systems. we presented an approach that generates multiple candidate claim optimizations and then selects the best one using various quality metrics. in experiments, combining fine-tuned bart with such candidate selection improved 60% of the claims from online debates, outperforming several baseline models and candidate selection strategies. we showcased generalization capabilities on two out-of-domain datasets, but we also found some claim optimization types hard to automate. in future work, we seek to examine whether recent large language models (e.g., alpaca) and endto-end models (where generation and candidate se-lection are learned jointly) can further optimize the quality of claims. as our approach so far relies on the availability of large claim revision corpora and language models, techniques for low-resource scenarios and languages should be explored to make claim optimization more widely applicable.