Overview of the 2022 Validity and Novelty Prediction Shared Task (2022)

Abstract: This paper provides an overview of the Argument Validity and Novelty Prediction Shared Task that was organized as part of the 9th Workshop on Argument Mining (ArgMining 2022). The task focused on the prediction of the validity and novelty of a conclusion given a textual premise. Validity is defined as the degree to which the conclusion is justified with respect to the given premise. Novelty defines the degree to which the conclusion contains content that is new in relation to the premise. Six groups participated in the task, submitting overall 13 system runs for the subtask of binary classification and 2 system runs for the subtask of relative classification. The results reveal that the task is challenging, with best results obtained for Validity prediction in the range of 75% F1 score, for Novelty prediction of 70% F1 score and for correctly predicting both Validity and Novelty of 45% F1 score. In this paper we summarize the task definition and dataset. We give an overview of the results obtained by the participating systems, as well as insights to be gained from the diverse contributions.

## introduction an important challenge within the field of argument mining is the assessment of the quality of an argument. in recent years, several systems have emerged that make mined arguments accessible to an end user, either via search engines @xcite , debate summarization systems @xcite , dialogue systems @xcite , or by other means. in order to establish confidence and trust on the side of the user, the ability to distinguish high-quality arguments from low-quality ones is important. @xcite investigated the notion of quality for argumentation and proposed 3 dimensions along which the quality of arguments can be rated: cogency, reasonableness, and effectiveness, introducing corresponding subcategories for each dimension. however, there have not been many attempts to operationalize the notion of quality so far, e.g., by an exact definition of a metric that assesses the quality or by means of an automatic procedure to determine the quality. exceptions are datasets manually labeled with coarse scores denoting overall quality, which have been used for supervised learning @xcite or attempts to determine single subdimensions, such as sufficiency (a subdimension of cogency) @xcite . motivated by this gap, the authors of this paper decided to propose a new shared task and submitted it to the 9th argmining workshop. instead of tackling the entire wide field of argument quality or isolating a single quality aspect, we focus on the conclusion in the context of its argument, and assess its quality in the validity and novelty prediction shared task. this task consists of the prediction of these two important conclusion quality dimensions. following @xcite , we define validity as the degree to which the conclusion is justified with respect to the given premise, and novelty as the degree to which the conclusion contains premise-related content that is not explicitly stated in the premise. the two notions stand in a trade-off to each other as it is straightforward to maximize one of them at the expense of the other. copying or paraphrasing parts of the premise as a conclusion will yield high validity but no novelty. expanding a concept of the premise with commonsense knowledge as a conclusion can potentially yield high novelty but may not satisfy validity. previous research in @xcite and @xcite has indeed shown that it is difficult to generate conclusions that satisfy both criteria, which require proper inference based on and expanding the premise. we divide the task of predicting validity and novelty into two subtasks. the first subtask consists in the binary prediction of whether a conclusion is valid resp. novel or not. the second subtask is framed as a comparative task, tasking systems to predict which of two given conclusions is more valid resp. more novel compared to the other, or whether they form a tie. the best achieved f 1 score for binary prediction of both validity and novelty is 45.16, by @xcite , and the best achieved f 1 score averaging the scores for relative validity and novelty is 41.5, by the team nlp@uit 2 -while the best scores for predicting validity and novelty as single prediction targets yield substantially higher results, with up to 74.6 points f 1 score for validity, and 70 points f 1 score for novelty. this large contrast shows that the joint objective is challenging. judging from the properties of the high-scoring systems for the individual quality aspects, we conclude that this challenging task requires strong text understanding capabilities, as well as (symbolic) background knowledge, which our received submissions are addressing, by taking a first step towards tackling this fundamental task for many downstream applications in computational argumentation. in the following, we describe the task as organized in the context of the 9th argument mining workshop. we describe the datasets used, as well as the different systems that have participated in the task. we provide an overview of the results the systems have obtained and make explicit the lessons we can learn from the shared task results, so that these observations can guide the community in their future choice of methods to address this and related tasks. ## related work argument quality within the growing field of computational argumentation, an important concern is to assess the quality of arguments. in their seminal work, @xcite established important dimensions for rating the quality of arguments. they proposed three quality dimensions: cogency (related to logics), effectiveness (relating to rhetoric) and reasonableness (relating to dialetics) -which they sub-divided into 11 finegrained quality aspects. in a recent survey, vecchi et al. (2021) extend the notion of argument quality to account for their function in deliberative pro-2 no description paper was submitted for this system. several works have proposed computational models to rate the quality of arguments. while @xcite and @xcite target rather coarse overall quality scores based on single quality labels, other systems were designed to assess specific quality aspects, such as convincingness @xcite , relevance @xcite or cogency, the logical coherence of an argument @xcite . while these works assess the quality of an argument as a whole, @xcite focused on the quality of the premises of an argument, in terms of their sufficiency, asking whether an argument's premises provide enough evidence for accepting or rejecting its claim or conclusion. they provide annotations of argument sufficiency on the argument essays @xcite and develop a classifier that achieves 84% accuracy for detecting (in)sufficiently supported arguments as a whole, including their (in)sufficient premises. @xcite revisited this quality criterion (sufficiency) in a new task formulation: conclusion generation from (in)sufficient premises, where the aim is to determine the sufficiency of a premise by examining the quality of the generated conclusion including the premise. ## submissions and results in total, we received 13 submissions, from six participating teams foot_1 for subtask a, and an additional submission each for subtask b from two teams that participated in subtask a. in addition we provide baselines for both subtasks, by fine-tuning the roberta-base-language model @xcite on the shared task training data, once to predict validity and and once novelty independently of each other (more details in appendix a). note that some teams did not provide a system description paper. we nevertheless include their results and short descriptions based on the teams' submission information. ## discussion the results for the submitted systems suggest that the prediction of validity seems to be an easier task compared to the prediction of novelty, as many submitted system reach higher scores on validity than on novelty prediction for both subtasks (computed mean scores of 62.74 for validity vs. 52.97 points f 1 -scores for novelty, across all system submissions in subtask a). most of the submitted systems (clteaml, axis@eduni-1, css, harshad) rely on large pre-trained languages models (e.g. gpt-3, roberta, bart) that are i) fine-tuned on task-specific data, ii) pre-trained on related tasks (nli, argrel), iii) or are used as generators conditioned on selected prompts, as well as combinations of these. systems relying on large language models achieve strong results in terms of validity prediction. the fact that the best results are achieved with the huge gpt-3 system, pretrained with a massive amount of textual data, and relying on prompts to condition the generation without being fine-tuned for the specific task is remarkable. pre-training on the related task of nli has been shown, in many submissions, to be beneficial for the task, whereas argument relation classification was not found to be similarly effective (see results for clteaml). further, multi-task-learning, aimed towards exploiting interactions between both quality labels has been shown to improve performance, having a joint instead of separate prediction of the two target labels, which corroborates their complementary nature. ## conclusion in this paper we have described the shared task on validity and novelty prediction that has been carried out as part of the 9th argument mining workshop. six groups participated in the task, submitting 15 system runs overall, with a preference for the more course-grained first subtask with binary labels for validity and novelty. the results suggest that validity is easier to predict compared to novelty. large language models which are few-shot prompted or fine-tuned on the provided task-specific data, especially by applying transfer-learning from natural inference tasks, perform reasonably well on the task of predicting validity. however, such systems have a notably worse performance on predicting novelty. systems that complement large pre-trained language models with external commonsense or world knowledge, by contrast, perform much better for novelty. this suggests that the recognition of novel content is challenging, requiring deeper understanding and inference involving background, common sense or even domain-specific knowledge. figure 2 : error heatmap of each prediction and submitted system. the x-axis lists the submitted systems and the y-axis instances grouped by topic. a topic marked with out does not occur in the other splits of the dataset, in-topics are also included in the validation-split. red and dark areas represent misclassified instances validity. figure 3 : error heatmap of each prediction and submitted system. the x-axis lists the submitted systems and the y-axis the instances group by topic. a topic marked with out does not occur in the other splits of the dataset, in-topics are also included in the validation-split. red and dark areas represent misclassified instances novelty.