On Selecting Training Corpora for Cross-Domain Claim Detection (2022)

Abstract: Identifying claims in text is a crucial first step in argument mining. In this paper, we investigate factors for the composition of training corpora to improve cross-domain claim detection. To this end, we use four recent argumentation corpora annotated with claims and submit them to several experimental scenarios. Our results indicate that the “ideal” composition of training corpora is characterized by a large corpus size, homogeneous claim proportions, and less formal text domains.

## introduction in the last decade, argument mining (am) has grown into a fruitful area of research @xcite . while early studies tended to focus on the annotation and detection of argument components in edited text domains @xcite , more recently the field progressed in different new directions. this includes intensified work on social media texts such as twitter, e.g. by @xcite , argument quality assessment @xcite and the identification of argumentation strategies @xcite . arguments consist of several components, and their identification is traditionally split into several subtasks, such as detecting argumentative text segments, specifying their function, and finding the relations among them. given that a claim is the central component of an argument, claim detection often constitutes a crucial part of an am pipeline. in this paper, we combine work in claim detection with recent advances in learning contextualized word representations in order to study crossdomain claim detection on the following set of recent english argumentation corpora: change my view (cmv) posts @xcite , persuasive essays @xcite , micro texts @xcite and political us debates @xcite . we selected them for achieving variation in genre or register, formality level, and topic. in principle, these dimensions should be distinguished, but for present purposes we do not study them separately, and thus we follow the common practice to use "domain" as an unspecific cover term. ultimately, we are interested in investigating the "ideal" composition of a training corpus for detecing claims in new domains or corpora. the paper is structured as follows. in section 2, we present relevant related work. in section 3, we describe the used corpora, and we outline our methods in section 4. we present our results in section 5 and provide a discussion in section 6. ## related work early work on claim detection was presented by @xcite , who introduced the concept of context-dependent claims for finding claims that are relevant for a particular predetermined topic and context. based on this idea, @xcite proposed an approach to more general topicindependent claim detection, where the context of the argumentation was not given to the detection model as input. @xcite focus on political debates and approach argument detection with the two subtasks of identifying argumentative sentences and subsequent classification of claims and premises. they report 0.84 f1 and 0.67 f1 scores for both tasks, respectively. our work differs from their study by only focusing on claims and classifying them directly, i.e., against "all other" material. @xcite propose models for argument role classification with mostly handcrafted features. later, in their work on a large heterogeneous corpus of argumentive sentences, @xcite develop an lstm cell that incorporates topic information in the process of sentence-level claim detection. f1 score compared to lstm cells without topic information. @xcite build on top of previously proposed recurrent architectures and successfully examine the positive influence of different contextualized word embeddings on the task of classifying argument components. @xcite investigate crossdomain claim identification in order to shed light on differences and similarities in claim conceptualizations across domains. they utilize linguistic feature-based and neural approaches (with and without pre-trained word embeddings). their study is a direct precursor of our work-we use some more recent data, and in addition, incorporate recent contextualized word embeddings that serve as input for our recurrent neural network classifier. for claim detection, their best feature-free models report 0.62 f1 and 0.67 f1 for persuasive essays and micro texts, respectively. ## data for determining factors influencing claim detection, we chose four english argumentation corpora of varying register (monologue and dialogue), formality level (written text and transcribed speeches), and topics. see to facilitate the task, we also ensured that our corpora have less variety in claim proportions than those used by @xcite . all our corpora contain further annotations, e.g., premises, but we only use claim annotations in this study. cmv. @xcite annotate claims, premises and semantic types of argument components on the change my view corpus from @xcite , reporting an iaa of 0.63 for claims. we segment this user-generated data into 3966 sentences. 34% sentences contain a claim. essay. the corpus of argumentative essays @xcite consists of 402 persuasive essays annotated for three argument components (major claim, claim, and premise) and their relations (support and attack). annotators achieved iaa scores of 0.88 and 0.64 for major claims and claims, respectively. all argument components are annotated on clause level. we combine major claim and claim into one single claim class. after sentence splitting, we obtain 6743 units, 31% of which contain a claim. micro. the argumentative microtext corpus @xcite was developed in a controlled setting where participants created short texts containing a single argument. annotators then built a complete argumentation graph per text, the agreement was 0.83. texts were originally written in german and then professionally translated to english. we work on this version; it consists of 451 sentences, 25% of which contain a claim. usdeb. the uselecdeb60to16 corpus of haddadan et al. ( 2019 ) is a collection of transcripts of political tv debates between 1960 and 2016. annotators labeled argumentativeness of sentences and sentences containing argument components, i.e., claim and premise. they achieved an iaa for component annotation of 0.40, which indicates the challenge for analyzing spoken language of this kind. this is the only corpus in our set where the number of claims exceeds those of premises. after sentence splitting, we obtain 38309 sentences. 38% contain a claim. to account for potential positional effects, we calculated percentages of claim positions by dividing a sentence into three equal parts on a token basis: beginning, middle and ending. a claim could potentially occur in individual parts or the combination of beginning and middle, middle and ending or all three parts. the percentages show that for the vast majority of sentences containing a claim, the claim occurs in all three parts. only in 1%-4% of sentences do the claims occur in two parts. see ## method for preprocessing, we perform tokenization and sentence segmentation with the trankit toolkit @xcite . following @xcite , we label a sentence as a claim if any token we conduct one in-domain and four crossdomain experiments in order to identify promising scenarios for claim detection: ## results all results presented in this section are produced with the roberta architecture trained on sentence units. we conducted additional experiments with bert models and with edus, which on the whole lead to worse results. for edus, this is especially the case for the claim class, which we are particularly interested in. we will discuss this briefly in section 6. in the following, we report macro f1 scores and f1, precision, and recall for the claim class. see ## discussion & conclusion as noted above, our bert and edu results cannot compete with the sentence-level roberta results. we surmise that roberta may have outperformed bert as it was pre-trained on an argument detection task; likewise, since it was trained on sentences, edu performance may be lower. while being a potentially interesting factor, we argue that the claim position in a sentence does not substantially affect our results. statistics on claim position show that claims in the vast majority of claim sentences occur in the beginning, middle, and ending, i.e. they cover more than 66% of tokens in a sentence. only in 1%-4% of sentences of a given corpus, claims merely occur in the beginning and middle or middle and ending, i.e. they cover a span of 34%-66% tokens in a given sentence. of course, this does not mean that position cannot have an effect in general, and justifies more research in the future. in contrast, our results suggest that different factors influence the choice of a suitable corpus for training claim detection models. first, corpus size seems to play a crucial role. this is especially the case when several corpora are combined for training. removing individual corpora from the training set while testing on all corpora (s3) shows that only the removal of the largest corpus (usdeb) has a profound effect on the results. our leave-one-out experiments (s4) confirm this finding, as models trained on all corpora except usdeb obtain worse results than models trained in other leave-one-out scenarios. also, training on usdeb in a corpus pair scenario (s5) consistently yields good results, indicating that a large training size has a beneficial effect, while training on the small micro corpus yields the worst results. second, although claim proportions vary less in our corpora than in those used by @xcite , differences in claim proportions may still have an effect. for instance, while usdeb is the largest corpus in our set, it also contains the highest proportion of claims, which may render it difficult for models trained on corpora with lower claim proportions to sufficiently capture the class distribution in usdeb. still, it appears that size effects outweigh claim proportion effects given that claim detection results improve when the micro corpus is left out for training, which is the corpus that is both the smallest and the one with the lowest claim proportion. third, our results suggest that domain plays a role. recall that the essay and micro corpora represent relatively "edited" text types, while cmv and usdeb contain web data and oral debates, which can be described as less formal. this may affect the way argumentation takes place. our corpus pair experiments show that models trained on the less formal cmv and usdeb yield better results than models trained on essay and micro. note that corpus size does not explain this pattern given that the cmv corpus is smaller than the essay corpus.