ARIES : A General Benchmark for Argument Relation Identification (2024)

Abstract: Measuring advances in argument mining is one of the main challenges in the area. Different theories of argument, heterogeneous annotations, and a varied set of argumentation domains make it difficult to contextualise and understand the results reported in different work from a general perspective. In this paper, we present ARIES, a general benchmark for Argument Relation Identification aimed at providing with a standard evaluation for argument mining research. ARIES covers the three different language modelling approaches: sequence and token modelling, and sequence-to-sequence-to-sequence alignment, together with the three main Transformer-based model architectures: encoder-only, decoder-only, and encoder-decoder. Furthermore, the benchmark consists of eight different argument mining datasets, covering the most common argumentation domains, and standardised with the same annotation structures. This paper provides a first comprehensive and comparative set of results in argument mining across a broad range of configurations to compare with, both advancing the state-of-the-art, and establishing a standard way to measure future advances in the area. Across varied task setups and architectures, our experiments reveal consistent challenges in cross-dataset evaluation, with notably poor results. Given the models’ struggle to acquire transferable skills, the task remains challenging, opening avenues for future research.

## introduction argument mining was originally defined as the task of automatically identifying argument structures from unstructured natural language inputs @xcite . although argument mining research has been split into several subtasks in the literature such as segmentation, argument classification and argument relation identification @xcite it is the latter that represents the main challenge in argument mining due to its complexity. argument relation identification (ari) starts from the point where all the relevant argument sequences have been segmented, and its main objective is to identify argumentative relations between them building complete argumentative structures. therefore, it is the ari subtask that provides us with the argument structures from unstructured natural language. in addition, while outstanding results have been reported in the previous stages, results on ari are more limited, representing one of the most difficult tasks in natural language processing due to its implicitness, the lack of data, and the lack of solid baselines with which to compare. one of the main challenges in the area of argument mining, however, has always been to compare advances in different contexts, understanding these contexts as different annotation theories or argumentative domains. therefore, previous work reports different findings and advances, but it does that without providing a general picture of them and a comprehensive understanding of their findings for the argument mining community as a whole. a lack of a consistent cross-domain benchmark, as it has been done in many other areas of natural language processing (see glue @xcite , superglue @xcite , tweeteval @xcite , or @xcite among others), hinders our advances as a research community. although previous effort has been put in this direction, none of these previous work considers state-of-the-art nlp algorithms considering multiple language modelling approaches, and typically focused on specific tasks or domains @xcite . providing relevant results in the good direction, but limited in terms of generalisability. therefore, the definition of a general benchmark for state-of-the-art argument mining is something that remains unaddressed. this limitation, taking the success of the glue benchmark for natural language under-standing tasks @xcite into account, motivates the development of an argument miningspecific benchmark to comprehensively evaluate and measure the advances done in the area. in this paper, we present the argument relation identification evaluation strategy (aries), a robust, cross-dataset benchmark for evaluating existing and future contributions to the ari task. aries represents the first and most extensive benchmark to evaluate ari systems, thus providing a robust framework for comparative evaluation of argument mining systems. our main contribution is the formal definition of aries, including eight different corpora, three different natural language modelling approaches, and three different model architectures. furthermore, we carried out extensive experimentation, implementing the previous natural language modelling approaches and model architectures with different pre-trained language models. from our results, we do not only advance the state-of-the-art in ari, but also identify a concerning limitation of the generalisation capabilities of argument mining systems. this way, aries provides an ideal base on which to compare, propose, and implement argument mining systems addressing the ari task. ## related work the latest advances in natural language processing have been reflected in argument mining and especially in ari research, the most challenging part of it. these natural language processing advances have been gradually integrated into argument mining systems with (in order) the use of lstm networks @xcite , the transformer architecture @xcite , contrastive learning @xcite , generative language models @xcite , end-to-end architectures @xcite , or the most recent large language models (llms) @xcite . all these advances, however, are difficult to compare and contextualise due to a lack of a standardised set of evaluation baselines. some effort has been put with previous research on the definition of benchmarks for argument mining. initially proposed in @xcite , the authors define node, a natural language argument benchmark consisting of three datasets and 792 related proposition pairs. in this early benchmark, the authors pointed out the needs of standardising the evaluation of argument mining systems. following this direction, @xcite extended the previous benchmark with a total of ten datasets containing 35,918 related proposition pairs. both benchmarks exclusively focused on the classification of argument relations, a subset of the ari task in which the relation is assumed to be known, limiting their applicability in more general situations. recently in @xcite , the authors address this limitation by proposing a benchmark for argument mining (bam), in which all the argument mining subtasks are brought into consideration together. the bam framework, however, is proposed as a pipeline-like method combining different previously existing argument mining systems to cover the complete argument mining process @xcite , rather than a thorough analysis of state-of-the-art nlp modelling techniques and architectures. furthermore, its current version only contains argumentative information in scientific documents, making it a valuable resource for this domain but limiting its generalisability to other application domains. ## benchmark the main contribution of this paper is the definition of aries, a state-of-the-art benchmark for argument relation identification in datasets of different domain and nature, which can be used as a reference to advance and to relativise the real impact of new findings in this area. furthermore, aries also reflects on a wide variety of model architectures, providing more insight on the capabilities of state-of-the-art algorithms. this section provides an in-depth presentation of all the variables taken into account in the proposed aries benchmark. ## discussion to contextualise the results reported in the aries benchmark, providing a better understanding of their impact in the argument mining community, we compared the best model architecture observed in aries with the best performing and most recent identified previous work addressing ari in each of the datasets individually. works by morio et al. 2023 ) represent the best possible reference with which to compare aries, given the similarity in the way that the ari task is approached. the resulting comparison is depicted in as it can be observed, aries represents a significant jump in performance compared to previous works. our benchmark consistently outperforms the previously reported results in the most similar instances of the ari task considering the same eight selected datasets. the direct comparison, however, is difficult to do due to the high variability in which different authors address the task and interpret argumentative concepts. for example, @xcite does it with an end-to-end model, and although we selected the reported results assuming an oracle system for adu segmentation, the proposed models are not entirely focused on ari, considering other aspects of argument mining such as component classification. other works such as @xcite and @xcite consider an additional relation for ari, the rephrase between two argument propositions. thus making the ari a four-class classification problem instead of considering the three classes included in aries. therefore, this comparison needs to be understood as a motivation and a starting point towards a more consistent and unified way of evaluating argument mining systems rather than a direct comparison between works. while worse results in a simpler version of the task should be taken as concerning, worse results in a more complex version of it do not need to mean that the system is worse. with our benchmark, we expect that future contributions in argument mining can be better contextualised and evaluated, moving forward together as a community rather than reporting specific results for heterogeneous setups that are difficult to compare and understand from a broader viewpoint. furthermore, we clearly observed how in the cd evaluation of the different natural language modelling approaches and architectures, the performance consistently dropped to the point of being close to the majority baseline. thus limiting the usability of the resulting models in different domains than the ones included during training. although some work has investigated cross-domain and cross-language argument mining @xcite , this issue has never been systematically explored in-depth, leaving the door open to a new challenging direction: robust-ness in argument mining @xcite . considering the relevance of language and domain in natural language argumentation, developing robust systems is a main issue if we want to be able to effectively deploy them in real-world scenarios. for this purpose, aries represents a valuable resource, allowing not only to compare between different datasets, but also to measure the crossdataset robustness of the developed argument mining systems. finally, we would also like to mention that recently, @xcite conducted a thorough study comparing the performance of generative llms (i.e., decoder-based architectures) for arc. although some of the reported results might seem higher than the ones included in the aries benchmark, as noted in the beginning of this paper, relation classification assumes that the relation has already been identified and classifies it as an attack or a support, significantly simplifying the task. therefore, we excluded these results from our comparison, being a significantly different task highly dependant on a previous step. instead, ari represents a completely independent task embedding the main purposes of argument mining (i.e., identifying argument structures from unstructured natural language inputs). ## conclusion in this paper we presented aries, a global benchmark for the identification of natural language arguments. aries represents an effort to ease the understanding of argument mining contributions and their impact to the community. we achieve this by providing solid results comparing the three main modelling approaches in nlp (i.e., sequence and token classification, and sequence-to-sequence alignment) combined with the three main model architectures (i.e., encoder, decoder, and encoderdecoder). our benchmark goes all over eight different corpora, presenting new state-of-the-art results for the ari task, and setting a new reference for research in argument mining. furthermore, we pointed out the limitations of domain-specific argument mining systems, showing poor results in cross-dataset evaluation. this limitation raises the question of how useful argument extraction systems can be when deployed in the wild, given their limited generalisability, highlighting the need to investigate the robustness of argument mining systems. 5 ): @xcite . as future work, we foresee expanding the aries benchmark to more languages than english. although argument mining has been mostly researched in english, corpora in catalan @xcite , spanish @xcite , japanese @xcite , or chinese @xcite among others have been annotated and publicly released in the recent years. increasing the language richness in argument mining research can be beneficial, not only for implementing more robust models, but also to help us investigating the differences between relevant natural language argument features underlying different languages.