Exploring the Potential of Large Language Models in Computational Argumentation (2024)

Abstract: Computational argumentation has become an essential tool in various domains, including law, public policy, and artificial intelligence. It is an emerging research field in natural language processing that attracts increasing attention. Research on computational argumentation mainly involves two types of tasks: argument mining and argument generation. As large language models (LLMs) have demonstrated impressive capabilities in understanding context and generating natural language, it is worthwhile to evaluate the performance of LLMs on diverse computational argumentation tasks. This work aims to embark on an assessment of LLMs, such as ChatGPT, Flan models, and LLaMA2 models, in both zero-shot and few-shot settings. We organize existing tasks into six main categories and standardize the format of fourteen openly available datasets. In addition, we present a new benchmark dataset on counter speech generation that aims to holistically evaluate the end-to-end performance of LLMs on argument mining and argument generation. Extensive experiments show that LLMs exhibit commendable performance across most of the datasets, demonstrating their capabilities in the field of argumentation. Our analysis offers valuable suggestions for evaluating computational argumentation and its integration with LLMs in future research endeavors.

## introduction argumentation is a powerful and indispensable tool in various domains such as legality @xcite , debating @xcite , and education @xcite . it plays a vital role in facilitating understanding between individuals by providing insights into different perspectives and their underlying reasons. additionally, argumentation serves as a means of communicating convincing opinions, enhancing the acceptability of positions among readers. as computational argumentation becomes a growing research field in natural language processing (nlp) @xcite @xcite @xcite , researchers have dedicated considerable efforts to two distinct directions @xcite @xcite . the first direction, argument mining, focuses on understanding unstructured texts and automatically extracting various argumentative elements @xcite @xcite . the other direction is argument generation, which aims to generate argumentative texts based on external knowledge @xcite or summarize key argument points. @xcite . unlike classical structure prediction nlp tasks like named entity recognition that typically take a single sentence as the input and extract token-level information, computational argumentation tasks require discourse-level comprehension. this requirement makes it challenging and laborious to gather a large volume of labeled data for training, hindering the progress of research in this field. fortunately, recent studies have shown that large language models (llms) @xcite @xcite have demonstrated impressive performance on a wide variety of nlp tasks @xcite @xcite @xcite in both zero-shot and few-shot settings. given their strong capability in understanding long contexts and generating natural language, it is ex-citing yet still questionable how well llms can perform computational argumentation tasks without any supervised training. in light of this, our objective is to investigate the performance of llms on diverse computational argumentation tasks. there are two main issues we aim to address in our study. firstly, although there are existing surveys about argument mining @xcite , the systematic study of the broader definition of computational argumentation including argument mining and argument generation is under-explored. to bridge this gap, we categorize current computational argumentation tasks into two primary classes, comprising six distinct categories. in addition, we establish a standardized format and evaluation metrics for fourteen openly available datasets. secondly, existing tasks and datasets either focus on argument mining or argument generation. to take a holistic approach, we propose a new task that integrates both argument mining and generation. this task is designed to generate counter speeches in response to debate speeches, which typically advocate a particular stance. we name them counter speech and supporting speech respectively in the remainder of our paper. this task requires the model to understand the argumentative structures in the supporting speech, meanwhile to generate the counter speech against the proposition. to facilitate the study, we construct a new document-to-document counterargument generation benchmark based on a debate database @xcite . to evaluate the performance of llms on computational argumentation tasks, we choose from both open-source and proprietary llms to conduct our main experiments, in zero-shot and fewshot settings. our results reveal that llms exhibit promising performance in both argument mining and argument generation tasks. while llms might fail to achieve exceptionally high scores on specific metrics such as rouge, we hypothesize that the strict nature of these metrics could potentially underestimate the true potential of llms, which are inherently generative in nature. human evaluation shows that llms are able to comprehend the core meaning of arguments and convey them effectively, even if the exact wording might not match. collectively, these findings highlight the strengths of llms in grasping and effectively conveying the essence of arguments, showcasing their potential beyond what traditional metrics may suggest. to summarize, our contributions include: ## background computational argumentation argumentation research has a long history @xcite , aiming to persuade through logical propositions and achieve agreement among parties @xcite . recently, computational argumentation has emerged as a significant field in nlp. the two main research directions are argument mining and argument generation, along with other directions such as persuasiveness of arguments @xcite and quality assessment of arguments @xcite . our work specifically focuses on argument mining and argument generation, where the detailed background can be found in appendix a. ## tasks and datasets in this work, we systematically review existing tasks and datasets of computational argumentation and organize them in figure 1 . to maintain a balance for different tasks and datasets, we restrict our assessment by randomly sampling 500 examples from each dataset. ## experiments in this section, we discuss our choices of models, methods, and evaluation metrics. ## results and discussion in this section, we discuss the main results and provide insights into the performance of various llms on argument mining, argument generation and counter speech generation. ## conclusion and broader impacts in this paper, we have made several significant contributions to the field of computational argumentation research. firstly, our efforts in organizing the diverse landscape of argumentation-related tasks and standardizing the format of related datasets are crucial for future research in designing domain-specific largescale models for argumentation. secondly, we for the first time systematically evaluate the performance of multiple computational argumentation tasks using llms in zero-shot and few-shot settings. traditional approaches for computational argumentation rely heavily on supervised fine-tuning that requires a large amount of labeled data, hindering the progress of research in this field. our exploration of low-resource settings addresses a gap in previous computational argumentation research, demonstrating the potential of llms in scenarios with limited training data. furthermore, we introduce a new counter speech generation benchmark that evaluates models' capability in both argument mining and argument generation. our extensive experimental results and analysis demonstrate the potential of llms in computational argumentation, while also highlighting existing limitations in evaluating computational argumentation tasks. overall, our paper provides important insights and valuable resources for researchers interested in the field of computational argumentation, which will potentially inspire further advancement in this exciting area.