IAM : A Comprehensive and Large-Scale Dataset for Integrated Argument Mining Tasks (2022)

Abstract: Traditionally, a debate usually requires a manual preparation process, including reading plenty of articles, selecting the claims, identifying the stances of the claims, seeking the evidence for the claims, etc. As the AI debate attracts more attention these years, it is worth exploring the methods to automate the tedious process involved in the debating system. In this work, we introduce a comprehensive and large dataset named IAM, which can be applied to a series of argument mining tasks, including claim extraction, stance classification, evidence extraction, etc. Our dataset is collected from over 1k articles related to 123 topics. Near 70k sentences in the dataset are fully annotated based on their argument properties (e.g., claims, stances, evidence, etc.). We further propose two new integrated argument mining tasks associated with the debate preparation process: (1) claim extraction with stance classification (CESC) and (2) claim-evidence pair extraction (CEPE). We adopt a pipeline approach and an end-to-end method for each integrated task separately. Promising experimental results are reported to show the values and challenges of our proposed tasks, and motivate future research on argument mining.

## introduction debating has a long history and wide application scenarios in education field @xcite @xcite , political domain @xcite @xcite , legal actions @xcite @xcite , etc. it usually involves tons of manual preparation steps, including reading the articles, selecting the claims, identifying the claim stances to the topics, looking for the evidence of the claims, etc. since the machine has shown promising potential in processing large quantities of information in many other natural language processing tasks, it is also worthwhile to explore the methods for automating the manual process involved in debating. argument mining (am), as the core of a debating system @xcite , has received more attention in the past few years. several am tasks and datasets have been proposed to work towards automatic ai debate, such as: context dependent claim detection (cdcd) @xcite , claim stance classification (csc) @xcite , context dependent evidence detection (cded) @xcite , etc. all the above tasks are essential elements for am and they are mutually reinforcing in the debating preparation process. in this work, we aim at automating the debating preparation process as shown in figure 1 . specifically, providing with the debating topic and several related articles, we intend to extract the claims with their stances, and also the evidence supporting the claims. however, none of the existing works can facilitate the study of all these tasks at the same time. motivated by this, we introduce a comprehensive dataset named iam to support the research of these tasks. we create our dataset by first collecting over 100 topics from online forums and then exploring over 1k articles related to these topics. all the sentences in those articles are fully-annotated following a set of carefully defined annotation guidelines. given a specific topic, the annotators have to distinguish whether the given sentence is a claim to this topic and identify the relation between the selected claim and the topic (i.e., support or contest). then given the claims, the annotators have to browse the contexts to find evidence supporting the claims. with all the labeled information, researchers can work towards these primary argument mining tasks simultaneously. to better coordinate these individual tasks together, we propose two new integrated tasks: claim extraction with stance classification (cesc) and claim-evidence pair extraction (cepe). instead of treating the existing tasks (i.e., cdcd, csc, cded) as individual ones, the two proposed tasks can integrate the relevant primary tasks together, which are more practical and more effective in the debating preparation process. the cesc task can be divided into two subtasks: the claim detection task and the stance classification task. intuitively, we conduct experiments on the cesc task with a pipeline approach to combine the two subtasks. as the two subtasks are mutually reinforcing each other, we also adopt an end-to-end classification model with multiple labels (i.e., support, contest, and no relation). the cepe task is composed of the claim detection task and the evidence detection task. similar to the annotation procedure, we apply a pipeline method to tackle this problem by first detecting the claims given the topics and then identifying the corresponding evidence of each claim. we also use a multi-task model to extract both claims and evidence as well as their pairing relation simultaneously. we conduct extensive experiments on our dataset to verify the effectiveness of our models and shed light on the challenges of our proposed tasks. to summarize, our contributions are as follows. ## related work in recent years, there is a tremendous amount of research effort in the computational argumentation research field @xcite , such as argument components identification @xcite @xcite , argument classification and clustering @xcite , argument relation prediction @xcite , argument pair extraction @xcite @xcite , argument quality assessment @xcite @xcite @xcite , etc. meanwhile, researchers have been exploring new datasets and methods to automate the debating preparation process, such as project debater @xcite , etc. bilu et al. (2019 ) work on the argument invention task in the debating field to automatically identify which of these arguments are relevant to the topic. li et al. (2020) explore the role of argument structure in online debate persuasion. levy et al. (2014) introduce a dataset with labeled claims and work on the task of contextdependent claim detection (cdcd). bar-haim et al. (2017) modify aharoni et al. (2014)'s dataset by further labeling the claim stances, and tackle the problem of stance classification of contextdependent claims. rinott et al. (2015) propose a task of detecting context-dependent evidence that supports a given claim (cded) and also introduce a new dataset for this task. unlike previous works with a specific focus on only one argument mining task, we introduce a comprehensive dataset that is able to support different tasks related to the debating system. such a dataset not only enlightens future research on the argument mining field but also shows strong potential for various practical applications. another difference is that existing tasks (e.g., cdcd, cded, csc, etc.) could be considered as subtasks in the emerging wider field of argumentation mining @xcite . while in this paper, we propose two integrated tasks (i.e., cesc and cepe) incorporating the existing subtasks in the debating system, which takes a step forward to automatic ai debate. a more detailed comparison to the most representative and relevant previous datasets will be shown in section 3.3. ## iam dataset we introduce a large and comprehensive dataset to facilitate the study of several essential am tasks in the debating system. we describe the collection process, annotation details and data analysis here. ## tasks in the debating system, our ultimate goal is to automate the whole debate preparation process as shown in figure 1 . with the introduced annotated dataset, we can tackle all core subtasks involved in the process at the same time. in this section, we first review the existing subtasks, and then propose two integrated argument mining tasks. ## approaches to tackle the two integrated tasks, we first adopt a pipeline approach to pipe the corresponding subtasks together by using sentence-pair classification on each subtask. we also propose two end-to-end models for the two integrated tasks. ## experiments 6.1 experimental settings we split our dataset randomly by a ratio of 8:1:1 for training, development, and testing. the dataset statistics are shown in with evidence, 10 to 15 sentences before and after the claims are chosen to be the evidence candidates. the negative sampling strategy is also applied for the evidence candidates in the training set, where the ratio of positive samples (i.e., 7,278 pieces of evidence) to negative samples (i.e., 50,120 pieces of non-evidence) is 1:6.9. it turns out that using 1 random negative sample for each piece of evidence is the best. we implement the sentence-pair classification model and the multi-label model for cesc with the aid of simpletransformers @xcite . the multi-task model for cepe is based on the implementation of the multi-task framework by @xcite . all models are run with v100 gpu. we train our models for 10 epochs. we experiment with two pre-trained models: bert @xcite and roberta @xcite . batch size is set as 128 for claim extraction and stance classification, and 16 for evidence extraction. we use 1 encoding layer for the multi-task model, and other parameters are the same as the previous work. 2 for the claim and evidence extraction subtasks, besides macro f 1 and micro f 1 , we also report the claim-class f 1 and the evidence-class f 1 , respectively. for cesc, we additionally report the claim-class f 1 of different stances (i.e., support and contest). for the claim stance classification subtask, we report overall accuracy and f 1 for each class, as this task can be simply considered as a binary classification problem with balanced labels. for cepe, we report precision, recall, and f 1 .