Which Side Are You On? A Multi-task Dataset for End-to-End Argument Summarisation and Evaluation (2024)

Abstract: With the recent advances of large language models (LLMs), it is no longer infeasible to build an automated debate system that helps people to synthesise persuasive arguments. Previous work attempted this task by integrating multiple components. In our work, we introduce an argument mining dataset that captures the end-to-end process of preparing an argumentative essay for a debate, which covers the tasks of claim and evidence identification (Task 1 ED), evidence convincingness ranking (Task 2 ECR), argumentative essay summarisation and human preference ranking (Task 3 ASR) and metric learning for automated evaluation of resulting essays, based on human feedback along argument quality dimensions (Task 4 SQE). Our dataset contains 14k examples of claims that are fully annotated with various properties supporting the aforementioned tasks. We evaluate multiple generative baselines for each of these tasks, including representative LLMs. We find, that while they show promising results on individual tasks in our benchmark, their end-to-end performance on all four tasks in succession deteriorates significantly, both in automated measures as well as in human-centred evaluation. This challenge presented by our proposed dataset motivates future research on end-to-end argument mining and summarisation. The repository of this project is available at https://github.com/HarrywillDr/ArgSum-Datatset.

## introduction crafting arguments for a debate requires a wealth of knowledge, meticulous logical thinking and a high level of command of language @xcite . recent trends treat this task as the automatic summarisation of a large body of debate-related material to alleviate the effort and time spent on brainstorming by stakeholders @xcite . most * corresponding email: hao.li-2@manchester.ac.uk of previous work split these processes into multicomponent systems, including context-dependent claim detection (cdcd) @xcite @xcite , claim stance classification (csc) @xcite @xcite , context-dependent evidence detection (cded) @xcite @xcite and argument summarisation (as) @xcite @xcite . however, thus far, these tasks have been approached in isolation without formulating them into an end-to-end process. this contradicts the natural sequence of actions in the debate preparation process @xcite , where outputs of previous tasks (e.g. finding relevant evidence) form the basis for the subsequent tasks (e.g. arranging claims and evidence in an argumentative essay). the only comprehensive automated debating work at present is ibm's project debater @xcite . however, it is composed of multiple independent components, potentially giving rise to cascading errors, whereby errors made in any of the preceding components might be propagated and amplified in subsequent components @xcite . currently, there is no dataset supporting the development and evaluation of end-to-end systems. furthermore, project debater lacks evaluative components; its argument quality component only considers how relevant potential candidate texts are to the debate topic. earlier works explored the assessment of the argument quality in various dimensions, like convincingness of arguments and evidence @xcite @xcite @xcite and sufficiency checking @xcite @xcite @xcite . recent advances in large-scale language yes yes no figure 1 : overview of proposed annotation pipeline, which includes four main tasks. task 1 identifies whether a snippet is an evidence for a given claim; task 2 selects the appropriate evidence for each claim to make it the most persuasive; task 3 generates a diversity of debate scripts for a given debate topic and stance then ranks them according to human preference, with the dimension of quality being measured in task 4. models (llms) such as gpt-3.5 * , gpt-4 (ope-nai, 2023), llama @xcite and bard @xcite , resulted in the achievement of seemingly human-like generated text according to evaluation dimensions such as fluency and richness @xcite . as such, the quality of their output falls short with respect to traditional argumentative quality criteria such as convincingness @xcite . at the same time, llms still exhibit deficiencies, such as the generation of false information @xcite . @xcite proposed a gpt-generated argumentative essay corpus with annotations by human experts to understand the lexical, syntactic and stylistic features of aigenerated content (aigc). however, they do not take into account fact-checking the generated texts and assessing the quality of essays in terms of argumentative properties. such evaluations are crucial in argument mining, as using false evidence to convince others can lead to untenable decisions. to address these shortcomings, we have developed an end-to-end dataset that cohesively unifies these primary tasks and provides a comprehensive view of debate preparation (illustrated in figure 1 ). we provide human annotations for each of the involved tasks as well as explore the performance of sota llms on these tasks in an end-to-end manner, facilitating design and evaluation of automated debate assistant systems, aligned with human preferences. our proposed dataset supports tasks that are broadly divided into two streams: (1) evidence detection (ed), (2) evidence convincingness ranking (ecr) and (3a) argument summarisation (as), are concerned with finding evidence to support the arguments and arranging them into a narra-tive. meanwhile, (3b) argument summarisation ranking (asr), learning to rank generated summaries from human preferences and (4) learning automated metrics for summary quality evaluation (sqe), are concerned with automating fine-grained evaluation of the generations from a (weak) human supervision signal. in particular, we design ed, ecr and sqe as classification tasks. as is formulated as a summarisation task, aimed at generating debate scripts. we conduct an extensive empirical evaluation of the selected baselines on the proposed tasks. in addition, we rely on human assessments to verify the quality of the generated tasks, which showed a positive correlation between automated evaluation and human preference. in summary, the main contributions of this paper are: ## related work one of the challenges in the field of argument mining is the lack of high-quality annotated arguments to serve as training and test data. some work has been done to alleviate the situation, focussing on different sub-tasks such as claim identification @xcite @xcite , stance identification @xcite @xcite , evidence identification @xcite , automatic identification of relational properties @xcite @xcite , argument summarisation and clustering @xcite @xcite , key point analysis @xcite , and debate speech analysis @xcite @xcite . others contributed to evaluation of argument quality, like convincingness ranking @xcite @xcite , sufficiency checking @xcite @xcite @xcite or factchecking @xcite . however, all of these work focus on only one of the abovementioned tasks, making them difficult to integrate due to the different data sources. similar to our work is the iam dataset @xcite , but there, only arguments at the sentence-pair level are considered, @xmath0, positions, @xmath1, instead of providing a document-level narrative to provide a comprehensive understanding; also, the resulting argument quality is not assessed. in addition, the proposed task focusses on long articles, whereas we focus on shorter, social-media style texts with non-standard structures. ## argument summarisation and evaluation dataset in this section, we discuss the process of data collection and annotation. the annotation process is divided into two main stages, each with two subtasks: stage 1 includes the classification of evidence and the ranking of the persuasiveness of the evidence; stage 2 is focussed on the quality of the generated summaries. ## baseline approaches sentence-pair classification: we formulate tasks 1, 2 and 4 as sentence-pair classification tasks, where we employed flan-t5 (chung et al., 2022), bert @xcite and roberta @xcite as baseline models. as an abstract representation of the task "does the evidence support the claim", the input is a concatenation of argument and evidence candidate, and the output is binary, i.e., {1, 0}, where the snippet is considered as evidence for the claim when they are judged to be the same pair. task 4 (sufficiency and factual error checking, where {1, 0} represent sufficient/factually wrong or not, respectively) follows the same setup. for task 2, we define it as a multiclass classification task, i.e., given an evidence pair and a claim, "the former is more convincing" is labelled as 1, "the latter is more convincing" is labelled as 0, and "both are equal" is labelled as 0.5. llms with contrastive learning: we formulate tasks 2 and 3 as a contrastive learning task. conceptually, we consider the evidence closest to the claim in the semantic embedding space as the most supportive candidate for it. for this purpose, we utilise siamese neural networks (including simcse-roberta @xcite , sentence-t5 @xcite ) with contrasting loss functions. the inputs are the concatenation of the topic and claim, and the evidence candidate, respectively. the output is the score reflecting how close they are to each other (more details in appendix f). summarisation with llms: inspired by previous work in argument summarisation @xcite , we employed flan-t5 @xcite and pegasus @xcite as summarisation systems. the former is a large model trained on more than a thousand instruction tasks, while the latter has pre-trained objectives tailored specifically for abstract summarisation. the input of the model is in the form of a concatenated @xmath2, topic, claim 1, evidence 1, evidence 2,..., claim 2, ....>. the output will be a single summary. project debater: the first end-to-end automatic debating system, project debater @xcite encompasses multiple components such as claim detection, evidence detection, argument quality assessment and narrative generation. employing it on our dataset, we can explore the limitations of a component-based sota system. ## experimental settings broadly speaking, we aim to investigate how well the generated data aligns with human standard and the performance of current representative llms in end-to-end debate preparation; we also provide a benchmark for future research. specifically, we ask: (i) how well do representative llms perform on tasks? (ii) how well does llm-generated data correlate with human judgement? (iii) how big is the impact of end-to-end pipeline inputs on model performance? (iv) how good is the baseline performance on each of the subtasks? to answer questions (i) and (ii), we conduct both manual evaluation and automated evaluation by first evaluating llm outputs by means of human evaluation and then assessing correlations with human judgement and learned metrics. for questions (iii) and (iv), we compare the performance of our proposed baseline approach with both golden input and pipeline input. to measure the performance on the imbalanced data distribution setting, we measure the matthews correlation coefficient (mcc) @xcite as well as macro-f1. the former progressively penalises over-confidence on the majority class, while the latter allocates the same weight to each class regardless of its cardinality in the dataset. for the summarisation task, we employ the widely used rouge metric @xcite , which measures performance by counting n-gram overlap. in addition, bleurt @xcite ) is used to measure the performance from a semantic similarity perspective. for the ranking task, we consider precision@k, mean average precision@k (map@k) @xcite and normalized discounted cumulative gain@k (ndcg@k) @xcite . map@k is concerned with precision at every rank up to k, which takes into account both the number of relevant documents retrieved and their order in the rank list, penalising systems that return relevant documents further down the list. meanwhile, 1 shows that human evaluations suggest that generated summaries are of high quality. overall, gpt4 showed the best performance, while alpaca had the worst performance which is consistent with the results of the bt model based on ranked quantification (shown in appendix c). specifically, all models perform well in the truth-ful, harmless, faithful and redundant categories, suggesting that llm outputs are comparable to humans along those dimensions. however, they still lack in dimensions that require logical understanding (helpful and informative). similarity to gpt-generated summaries correlates with argumentative quality assessments: the relative quality of the models is further represented in flant5-3b shows comparable overall performance with project debater; bigger llms could potentially lead to improved performance. task 2: evidence convincingness ranking as mentioned in the baseline setting description, task 2 has been formulated as both a classification and ranking task. comparing the performance of classification vs contrastive learning, the former outperforms the latter ( task 3 subtask a: argumentation summary here, flant5-3b outperformed all other models in all settings, with the remaining models exhibiting comparable performance levels (c.f., average rouge scores of all vs top-2 in task 3 subtask b: argumentation summary and ranking ## conclusion in this study, we built an end-to-end argumentative summary and evaluation dataset and established the benchmarks for each task. this process involves annotating the data by human experts mixed with llms and applying popular deep learning baseline methods to the new datasets. we open-source our data and benchmark code, enabling researchers to reproduce and enhance the results. in future, we hope to further expand the evaluation dataset to explore the potential of unprocessed real-world data as evidence and fact-checking for its quality.