Data Augmentation for Improving the Prediction of Validity and Novelty of Argumentative Conclusions (2022)

Abstract: We address the problem of automatically predicting the quality of a conclusion given a set of (textual) premises of an argument, focusing in particular on the task of predicting the validity and novelty of the argumentative conclusion. We propose a multi-task approach that jointly predicts the validity and novelty of the textual conclusion, relying on pre-trained language models fine-tuned on the task. As training data for this task is scarce and costly to obtain, we experimentally investigate the impact of data augmentation approaches for improving the accuracy of prediction compared to a baseline that relies on task-specific data only. We consider the generation of synthetic data as well as the integration of datasets from related argument tasks. We show that especially our synthetic data, combined with class-balancing and instance-specific learning rates, substantially improves classification results (+15.1 points in F_1 -score). Using only training data retrieved from related datasets by automatically labeling them for validity and novelty, combined with synthetic data, outperforms the baseline by 11.5 points in F_1 -score.

## introduction recently, there has been interest in developing approaches that can automatically generate conclusions from textual premises @xcite . many of these systems rely on language models that are fine-tuned to the task of generating argument conclusions. as the space of possible conclusions that can be generated from a textual premise is a priori not constrained, it is key for a system to understand whether a conclusion candidate is adequate. in particular, models that can predict the quality of conclusions are needed to guide a generation system towards generating suitable argumentation conclusions. while there has been work on identifying dimensions that characterize argument qual-ity @xcite , there are very few models that actually operationalize the (automatic) scoring of the quality of a conclusion. @xcite have analyzed whether the notion of "sufficiency" of an argument can be predicted, reaching an accuracy of 90% with transformerbased language models. @xcite have relied on the notions of "validity" and "novelty" in their manual evaluation of conclusion quality -"validity" meaning that the conclusion is justified based on its premise and "novelty" that the conclusion contains novel content which is related to the premise. they have shown that there is a weak correlation between the automatically computed similarity between generated conclusion and reference conclusion, as measured by the bertscore, on the one hand, and the criteria of manually rated validity and novelty on the other hand. one key problem is that it is difficult to obtain sufficient training data for such tasks, which is a necessary basis for training reliable models for these tasks. in this paper, we focus on predicting the validity and novelty of argument conclusions. we propose a multi-task classification approach that jointly predicts validity and novelty in a single model that exploits synergies between both tasks. our main goal is to explore to what extent data augmentation can contribute to overcome the scarcity of manually labeled argument quality data. we propose and experimentally investigate two types of approaches. on the one hand, we investigate the impact of a synthetic data generation approach that modifies existing training data by generating 'altered copies' of its instances, e.g., by shifting or extending content between premise and conclusion in view of novelty, or by paraphrasing or negating parts of the argument in view of validity. further, we augment the data labeled for novelty and validity by considering datasets from related argument mining tasks. in particu-lar, we consider data from the explagraph-dataset by @xcite , the ibm-argumentqualitydataset by @xcite and the student-essays-dataset, annotated for sufficiency of the conclusion by @xcite . we describe how training data from these related tasks is mapped into a form that can be used to enhance the performance of our classifier for validity and novelty prediction. we experimentally evaluate the impact of both data augmentation strategies, showing that the generation of synthetic data outperforms a baseline system trained with only taskspecific data by 15.1 points in f 1 -score (38.3 vs. 23.2). even when only using datasets from related tasks as training data, we improve results over the baseline by 11.5 points. our main contributions are: ## related work the task of automatic generation of arguments has received increasing attention in the last years @xcite . in particular, research has considered the generation of a conclusion given a (textual) premise @xcite @xcite . these approaches rely on language models that are fine-tuned to the task of conclusion generation. the generation of conclusions can be seen as a search in the output space of a language model conditioned on the textual premise. in the manual evaluation of approaches generating conclusions, @xcite and @xcite found that (generated) conclusions are often either not justified given their premise, or are often just a plain copy or paraphrase of the premise, hence lacking novelty. they conclude that validity and novelty are two main properties a conclusion should fulfill and that stand in a tradeoff relation to each other. a key question is thus how to guide the search or generation process towards i) conclusions that represent a legitimate inference from the premises, meaning that the conclusions are valid, and ii) conclusions that are not simple paraphrases of the premises, i.e., they are novel or informative. having operationalized and thus automatically computable quality dimensions is key to generating high-quality conclusions. while there is previous work that identifies quality criteria for arguments @xcite , it has been shown that the annotation of such quality criteria is highly subjective @xcite . also, little work has been done on automatically rating the quality criteria for arguments. an exception is work by @xcite who -following @xcite -studied the operationalization of the criterion of sufficiency. sufficiency measures whether the premises provide enough evidence for accepting or rejecting the conclusion, and is hence a criterion closely related to our notion of validity. in this paper, we are concerned with developing a computational model that can jointly predict the validity and novelty of conclusions. given that manually annotated data is scarce, relying on the manual studies by @xcite , we consider how task-specific datasets can be augmented with synthetic data and how to repurpose data from related argument mining tasks. our work is thus related to and encouraged by data augmentation approaches in general. one example is the field of code-mixed languages, which often lacks available annotated training data. here, @xcite showed how to create synthetic instances of code-mixing language by merging sentences from different languages with the help of syntactic parse trees. another task that has been shown to profit from automatically generated synthetic training data is grammatical error correction. here, it has been shown that creating additional training data by corrupting error-free sentences leads to performance gains @xcite . finally, it has been shown that, by generating synthetic negative instances, one can bootstrap classifiers, e.g., to rate the output of a language model converting knowledge graph triples into natural language @xcite . building on prior evidence that generation of synthetic data can improve classifier performance, we investigate a clone&mutate technique that can artificially create new training instances of every class. ## methods in this section, we present our methods for tackling the task of predicting validity and novelty as a classification task. we describe the learning objective and how we generate and modulate additional training data using data augmentation techniques. figure 1 shows our proposed architecture. ## datasets this section presents the four datasets we use in our work. as a baseline, we rely on the relatively small dataset provided by @xcite , in which conclusions were explicitly annotated for validity and novelty (henceforth task-internal data). we further rely on task-external data: the explagraphs dataset by @xcite , the ibm debater datasets by @xcite and the annotated essays dataset by @xcite (sorted by their relatedness to the validitynovelty-classification task in descending order). appendix a shows examples from each dataset. ## experiments and evaluation in this section, we present our experimental results with the goal of testing the following hypotheses: ## conclusion predicting the validity and novelty of a given conclusion based on its premise is a challenging task. using 750 class-unbalanced training instances annotated with validity and novelty does not provide enough evidence for tuning a large language model. augmenting the task-internal training data to 10,000 instances using task-external and synthetic data increases the valnov-score up to 38.0. using task-internal and synthetic data to balance the training data increases this score to 38.3. how-ever, the results achieved by data augmentation techniques are still very modest, showing that massive training data and modern language models alone are not sufficient for solving the task. while valid but non-novel instances can, to a large part, be detected using paraphrase recognition tests, many instances require logical inference and commonsense knowledge to properly classify validity and novelty. none of these capabilities are supported in the subsymbolic approach we chose in this work. in future work, we aim to investigate the impact of incorporating commonsense knowledge and deeper logical reasoning into the task of validity and novelty prediction.