Assessing the Sufficiency of Arguments through Conclusion Generation (2021)

Abstract: The premises of an argument give evidence or other reasons to support a conclusion. However, the amount of support required depends on the generality of a conclusion, the nature of the individual premises, and similar. An argument whose premises make its conclusion rationally worthy to be drawn is called sufficient in argument quality research. Previous work tackled sufficiency assessment as a standard text classification problem, not modeling the inherent relation of premises and conclusion. In this paper, we hypothesize that the conclusion of a sufficient argument can be generated from its premises. To study this hypothesis, we explore the potential of assessing sufficiency based on the output of large-scale pre-trained language models. Our best model variant achieves an F1-score of .885, outperforming the previous state-of-the-art and being on par with human experts. While manual evaluation reveals the quality of the generated conclusions, their impact remains low ultimately.

## introduction the quality assessment of natural language argumentation is nowadays studied extensively for various genres and text granularities, from entire news editorials @xcite to arguments in online forums @xcite to single claims in social media discussions @xcite . the reason lies in its importance for driving downstream applications such as writing support @xcite , argument search @xcite , and debating technologies @xcite . @xcite organized quality dimensions of arguments into three complementary aspects: logic, rhetoric, and dialectic. logical quality refers to the actual argument structure, that is, how strong an argument is in terms of the support of a claim (the argument's conclusion) by evidence and other reasons (the premises). the first reason why education and preventative measures should receive a greater budget is the potential improvements in health system. i believe that decreasing the number of patients can lead hospitals and healthcare centers to be managed effectively which will result in better treatments for current patients. therefore, society should be educated and became aware of health issues so that the potential precautions on the way of illnesses can be taken instead of trying to provide treatment for the increasing number of patients. the second reason why governments should allocate more budget on prevention from illness and providing health education is the welfare of the society. in my opinion, there is nothing more important than health in a human's life and the happiness and welfare come with health. therefore, a government's role should be providing means that lead its citizens to learn how to prevent from potential illness that can cause misery in people's lives. for example, the marketing campaign of ministry of health in turkey which aimed smoking problem among the youth increased the well-being of those who quit smoking and adapted a better lifestyle after the campaign. figure 1 : two example arguments from a persuasive student essay, one classified as sufficient, the other as insufficient in the corpus of @xcite . a key dimension of logical quality is sufficiency, capturing whether an argument's premises together make it rationally worthy of drawing its conclusion @xcite ). consider, for example, the two arguments on health education in figure 1 , taken from the argument-annotated essay corpus of @xcite . while the upper one was deemed sufficient by human experts @xcite ), the lower one was not, likely because the second premise tries to reason from a single example. a reliable computational assessment of argument sufficiency would allow systems to determine those arguments that are well-reasoned. as detailed in section 2, previous approaches to argument sufficiency assessment model the task as a standard text classification problem and tackle it with convolutional neural networks @xcite or traditional feature engineer-ing @xcite . in the focused domain of persuasive student essays, @xcite obtained a macro f 1 -score of .827, not far away from human performance in their setting (.887). however, to further improve the state of the art, we expect the integration of knowledge beyond what is directly available in the text at hand is needed. in particular, we observe that existing work neither explicitly considers an argument's premises and conclusions, nor a property of their relationship. we hypothesize that only a sufficient argument makes it possible to infer the conclusion from the premises. consequently, comparing the stated conclusion of an argument with one that is (automatically) generated from the premises could help the model to distinguish sufficient arguments from insufficient ones. this hypothesis raises the question of whether the knowledge encoded in large-scale pre-trained language models can be leveraged, a direction nearly unexplored so far in argument quality assessment. in this paper, we study whether generating a conclusion from an argument's premises benefits the computational assessment of the argument's sufficiency. in particular, we first enrich the argument with structural annotations, highlighting which parts are the premises and which part is the conclusion. we propose in section 4 to then mask the conclusion in order to learn to re-generate it using fine-tuned bart @xcite . combining the generated conclusion with the original argument and its annotations, our approach learns to distinguish sufficient from insufficient arguments using a modified roberta model @xcite . starting from ground-truth argument structure, we subsequently evaluate conclusion generation and sufficiency assessment on the merged annotations of the corpora of @xcite and @xcite , as described in section 3. our generation experiments indicate that fine-tuning bart leads to better conclusions, which are on par with human-written conclusions in terms of sufficiency, likeliness, and novelty (section 5). to quantify the impact on sufficiency assessment, we explore various combinations of premises, original conclusion, and generated conclusion in systematic ablation tests, and we compare them to the state of the art and a human upper bound (section 6). our sufficiency experiments reveal that, even on the plain input text of an argument, roberta already improves significantly over the state of the art. the addition of structural annotations and the generated conclusion lead to further improvements, although the benefit of generation ultimately remains limited, possibly due to the generally limited importance of knowing the conclusion on the given data. finally, we discuss the results of our approaches in section 7 in light of their implications for the field. the main contributions of this paper are: foot_0 ## related work computational argumentation research has assessed various dimensions of argument quality. @xcite provide a theory-based taxonomy of 15 logical, rhetorical, and dialectical quality dimensions and of the work in natural language processing done in these directions. we focus on the (local) sufficiency dimension, which is key to logical cogency, representing that an argument's conclusion can rationally be drawn from its premises, given that these are acceptable and relevant @xcite . few approaches tackled sufficiency computationally so far. aside from wachsmuth and werner (2020) who assess it as one of the 15 dimensions above using traditional text-focused feature engineering, we are only aware of the work of @xcite who extend the argumentannotated essay corpus of @xcite with binary sufficiency annotations. on this basis, the authors compare a support vector machine using lexical, syntactic, and length features to a convolutional neural network (cnn) with word vectors, the latter achieving the best result with a macro f 1 -score of .827. in our experiments, we use their dataset and replicate their experiment settings, in order to compare to the cnn. unlike @xcite , we rely on a transformer-based architecture, namely we adapt roberta @xcite to assess sufficiency. approaches to argument quality assessment using such architectures are still limited, mostly focusing on a holistic view of quality @xcite , although a few approaches used transformers for some of the dimensions of @xcite , such as @xcite , or somewhat related dimensions in light of quality improvement @xcite . in contrast to the standard use of transformers for text classification, we leverage the structure of arguments for their assessment. @xcite provided evidence that mining the argumentative structure of persuasive essays helps to better assess four essay-level quality dimensions of argumentation. similarly, we use annotations of the premises and conclusions of arguments for the sufficiency assessment, but we target the arguments. moreover, we explore to benefit of conclusion generation for the assessment. the idea of reconstructing an argument's conclusion from its premises was introduced by @xcite , but their approach focused on the inference of a conclusion's target. the actual generation of entire conclusions has so far only been studied by @xcite . the authors presented the first corpus for this task along with experiments where they adapted bart @xcite ) from summarization to conclusion generation. while they trained bart to directly generate a conclusion based on premises, we generate conclusions that fit the context of an entire argument. to this end, we leverage and finetune bart's inherent denoising capabilities obtained during pretraining to replace a mask token in an argument. ## data to study our hypothesis that a sufficient argument's conclusion can be generated from its premises, we need data that is annotated for both argument structure and sufficiency. in this section, we describe how we employ existing corpora for this purpose. ## approach this section describes our two-step approach to assess the sufficiency of a given argument through conclusion generation. first, we generate a conclusion from the argument's premises using a pretrained language model finetuned on the task of replacing the masked conclusion of an argument. second, the generated conclusion is used to assess the argument's sufficiency by experimenting with eight modified versions of the original input argument (section 6). an overview of the approach is shown in figure 2 . in the following, we detail how we train the models for the two steps. ## evaluation of conclusion generation to study our hypothesis, we need to ensure that the generated conclusions are meaningful and fit in the context of a given argument, so they can be helpful in sufficiency assessment. in this section, we therefore evaluate the quality of the generated conclusions, both automatically and manually.