A New Dataset for Causality Identification in Argumentative Texts (2023)

Abstract: Existing datasets for causality identification in argumentative texts have several limitations, such as the type of input text (e.g., only claims), causality type (e.g., only positive), and the linguistic patterns investigated (e.g., only verb connectives). To resolve these limitations, we build the Webis-Causality-23 dataset, with sophisticated inputs (all units from arguments), a balanced distribution of causality types, and a larger number of linguistic patterns denoting causality. The dataset contains 1485 examples derived by combining the two paradigms of distant supervision and uncertainty sampling to identify diverse, high-quality samples of causality relations, and annotate them in a cost-effective manner.

## introduction causality identification is a vital task in natural language processing that can contribute to different downstream applications such as question answering, fact-checking, and commonsense reasoning. the task which concerns identifying texts with causality relations, the type of relations (positive or negative), and the concepts involved in the relations, is studied in diverse domains including biomedicine @xcite , education @xcite , and recently computational argumentation @xcite . in computational argumentation, causality identification impacts fundamental tasks such as topicindependent argument mining and building largescale argumentation graphs @xcite . despite its importance, only a few annotated datasets for identifying causality have been built so far. moreover, these datasets often focus only on one argument component (e.g., claim), encode bias towards the 'positive' type of causality, and/or consider a limited number of linguistic patterns that capture causality (e.g., verb connectives). as such, developing robust supervised learning approaches based on these datasets for causality identification becomes more laborious. this paper aims to expand and enrich the available data for causality identification in argumentative texts written in english with webis-causality-23, a new dataset comprised of 1485 examples of more sophisticated input text (i.e., the whole argument), covers more causality patterns, and maintains a balanced distribution of causality types. to this end, we develop an approach that comprises two main steps: distant supervision and uncertainty sampling. first, we identify 10,329 candidate sentences for causality via distant supervision. next, we employ uncertainty sampling on these candidates and manually annotate 1485 argumentative sentences (via crowdsourcing), 867 of which contain at least one causal relation. of these, 515 sentences are further annotated as containing a positive cause-effect relation, and 536 as containing a negative one. many sentences encode multiple relations, and involve diverse linguistic patterns (see section 3). we train transformer-based classifiers using our newly built dataset, and reach high effectiveness in identifying causal relations compared to several baselines. the developed resources in the paper (e.g., data and code) are made freely available. foot_0 ## the webis-causality-23 dataset in this section, we describe our method for constructing the dataset. in particular, we first outline the distant supervision step, then, we discuss the uncertainty sampling. ## dataset analysis in this section, we present both qualitative and quantitative analyses of the webis-causality-23 dataset. the qualitative analysis encompasses an examination of inter-annotator agreement, dataset statistics, and identified patterns within the dataset. on the other hand, the quantitative analysis involves leveraging the constructed dataset to develop a new causality classifier. ## related work in general, causality datasets are expensive to build, scarce, small, biased towards one class, focused on only a single aspect of causality (e.g., whether a sentence has a causal relation or not), and include limited linguistic patterns (due to their sampling method, e.g., via a seed list of causal verbs). recently, xu et al. ( 2020 ) reviewed six publiclyavailable datasets. the largest, altlex @xcite , comprises nearly 45,000 sentences, but they are annotated only for the presence of causal relations, and only 10% are causal; the other five datasets are an order of magnitude smaller, and exhibit similar bias. in addition, the eventstoryline corpus @xcite , which is frequently used in related work, comprises several thousand causal links but no annotated negative samples. additionally, al-khatib et al. (2020) introduce a corpus comprising 4740 claims extracted from argumentative texts, with 36% of these claims being annotated as containing a causal relation. given that this corpus is the only one specifically focused on argumentative texts, we utilize it in our distance supervision and uncertainty sampling techniques. our objective is to achieve broader coverage of new causality patterns through the incorporation of this corpus. @xcite present a human-annotated dataset consisting of over 21,000 causal reasoning questions, each accompanied by a natural language explanation providing insight into the underlying cause of the observed causation. due to the scarcity of multilingual datasets with reliable and consistent annotations for event causality relations, @xcite present a new multilingual dataset that utilizes consistent annotation guidelines for five typologically distinct languages. perhaps most closely related to our own work, @xcite propose a distant supervisionbased data augmentation framework to address the data scarcity problem in causality. whereas their approach involves a fully automated causal event pair extraction for distant supervision, we propose a framework based on uncertainty sampling, aiming to both improve the quality, and drive down the cost of hand-labeled corpora. ## conclusion in this paper, we present webis-causality-23, a new dataset for causality identification in argumentative texts that considers all argument units (claims, premises) as inputs. the 1485 argumentative sentences in the dataset comprise a balanced distribution of the positive and negative causality types and encode diverse linguistic patterns denoting causality. initial experiments on causality identification using transformer-based classifiers demonstrate the effectiveness of our smaller yet high-quality dataset in comparison to a larger existing corpus with some limitations. our future plans involve utilizing our dataset to extract causality relations from diverse argumentative resources on the web. our main objectives are to construct a large-scale argumentation graph, enhance argument scheme classification, and improve argument explanation and simplification methods. additionally, we intend to leverage this dataset to develop a question-answering system specifically designed to address causal questions.