Full-Text Argumentation Mining on Scientific Publications (2022)

Abstract: Scholarly Argumentation Mining (SAM) has recently gained attention due to its potential to help scholars with the rapid growth of published scientific literature. It comprises two subtasks: argumentative discourse unit recognition (ADUR) and argumentative relation extraction (ARE), both of which are challenging since they require e.g. the integration of domain knowledge, the detection of implicit statements, and the disambiguation of argument structure. While previous work focused on dataset construction and baseline methods for specific document sections, such as abstract or results, full-text scholarly argumentation mining has seen little progress. In this work, we introduce a sequential pipeline model combining ADUR and ARE for full-text SAM, and provide a first analysis of the performance of pretrained language models (PLMs) on both subtasks. We establish a new SotA for ADUR on the Sci-Arg corpus, outperforming the previous best reported result by a large margin (+7% F1). We also present the first results for ARE, and thus for the full AM pipeline, on this benchmark dataset. Our detailed error analysis reveals that non-contiguous ADUs as well as the interpretation of discourse connectors pose major challenges and that data annotation needs to be more consistent.

## introduction argumentation mining (am) is concerned with the detection of the argumentative structure of text @xcite . it is commonly organized into two subtasks: 1) recognition of argumentative discourse units (adus), i.e. detecting argumentative spans of text and classifying them into types such as claim or premise, and 2) determining which adus have a relationship to each other and of what kind, e.g. support or attack. consider the following example, where the premise p supports the claim c: ::::::::::: dot-product ::::::::: attention ::: is :::::: much :::::: faster :::: than :::::::: additive ::::::::: attention c , since it can be implemented using highly optimized matrix multiplication code p . 1 since the amount of published scientific literature is growing exponentially @xcite , there is recently an increased interest in scholarly argumentation mining (sam). understanding the argumentative structure is key, not just to efficiently digest such work, but also to assess its quality @xcite . solving scholarly am is challenging, because it requires, among other things, the use of domain knowledge, the detection of implicit statements, and the disambiguation of argument structure @xcite . this is even harder when handling full-text that is often less concise and standardized, than, for example, abstracts. previous work in sam has focused on dataset construction @xcite , adu recognition @xcite , and the analysis of specific document sections, such as abstract or results @xcite @xcite . however, to get a thorough understanding of a scientific publication, all parts of the document matter. ideally, they back up the main argumentation and usually contain details that are relevant for the knowledgeable reader, thus, they should not be neglected. however, since the task is very complex, also for humans, there is not much training data for full-text sam available. pretrained language models (plms) such as scibert @xcite may help to address the above challenges because they contain a lot of linguistic and domain knowledge and have better long-range capabilities, allowing for improved contextualisation, especially when training data is rare. we hence propose a plm based model for full-text sam. to summarize, our contributions in this work are: ## preliminaries we first define the two tasks of adur and are, and discuss differences to the standard information extraction (ie) tasks of named entity recognition (ner) and relation extraction (re). an argumentative discourse unit (adu) can be defined as "span of text that plays a single role for an argument being analyzed and is demarcated by neighboring text spans that play a different role, or none at all" @xcite . it is the smallest unit of argumentation, and may span anything from an in-sentence clause up to multiple full sentences. adu recognition requires both detecting argumentative spans, as well as classifying them into predefined categories. typically, this is realised as sequence tagging task similar to ner, where a sequence of tokens @xmath0, t 2 , ..., t n } is assigned with a corresponding n -length sequence of labels @xmath1, l 2 , ..., l n } with l @xmath2. 2 in scholarly am, common adu classes are (own / background) claim, and evidence, data, or warrant @xcite ). ## bio2: begin, inside, outside of an entity in contrast to ner, adus typically vary much more in length than named entities. they are also highly context dependent and often discontinuous. adur is also related to discourse segmentation, but depends more on broader context and semantics instead of linguistic structure. elementary discourse units (edus), the building blocks in the context of rethorical structure theory @xcite , are more fine-grained, of shorter length and usually cover the complete text which is less the case for argumentative units. argumentative relation extraction is usually defined as classifying a pair of adus, head and tail, as either an instance of one of the target types or the artificial no-relation type. in other words, the task is to assign a label @xmath3, h, t}, where c is the set of relation types, t is the text and @xmath4, e h , l h ) and @xmath5, e t , l t ) describe the candidate head and tail entities where s and e are the start and end indices with respect to t and l is the entity type. typical relation types for sam are supports, mentions, attacks, contradicts, and contrasts @xcite @xcite . are is very similar to standard re, but sam relations are often marked by syntactic cues such as connectors, e.g. "because", "however", or "but", whereas in common re, content words like verbs and nouns are typical relation triggers. this makes are challenging because these connectors do not always realise argumentative structure, but also mark other aspects of discourse. consider, for example, the different meanings of "while" in the following example: here, the "while" in sentence 1) has a contrastive meaning, whereas sentence 2) denotes a temporal aspect. bi-lstm crf execution time increased quite substantially ( 4.54ms to 22.74ms ) furthermore b-own-claim i-own-claim b-data i-data l-data o i-own-claim l-own-claim o i-own-claim o scibert (a) adu recognition. tokens are embedded with a frozen plm, further contextualized with a trained lstm followed by a crf to calculate the tag sequence. supports bi-lstm classifier cnn+pooling argument embeddings adu embeddings scibert token embeddings execution substantially ( 4.54ms ) furthermore b-own-claim l-own-claim o b-data o o b-arg2 i-arg2 o b-arg1 o o (b) argumentative re. tokens are embedded with a frozen plm, adu tags and argument tags are embedded with simple embedding matrices. embeddings are concatenated, contextualized with a lstm and converted into a single vector that gets classified by a single fully connected layer. ## models we propose a pipeline of two distinct models, one for each subtask, that are described in the following. ## experimental setup dataset. we use the sci-arg dataset @xcite for model training and evaluation. it is the only available full text argumentation mining dataset for scientific publications. it contains 40 full text publications annotated with adus and argumentative relations. figure 1 shows an example excerpt, and preprocessing. we preprocess the documents by removing the initial xml headers. to decrease the sequence length of the input, we also split the documents into sections, e.g. introduction or conclusion. this is important to lower computational resource consumption since recent plms like scib-ert @xcite usually scale quadratic with the input length and are restricted to a certain max input size, e.g. 512 tokens. unfortunately, this leads to the removal of all relations labeled with semantically same, since these connect adus from different sections. however, this affects only 0.6% of the argumentative relations instances. ## results and discussion this section presents our experimental results. first, we compare against the adu recognition baseline as provided by @xcite . then, we present findings about prominent error cases and close with an ablation study. ## conclusion and future work in this paper, we presented a pipeline based approach to handle full-text argumentation mining on scientific publications and showed its effectiveness by establishing new state-of-the-art performance on the sci-arg corpus. however, there is still a significant gap to human performance. we used plm based models for both subtasks, argumentative discourse unit recognition (adur) and argumentative relation extraction (are), and found similar improvements gains (+7%) as reported elsewhere when using transformers over traditional approaches without attention mechanism, even without fine-tuning the plms. our detailed error analysis revealed several findings. first, recognizing instances is much harder than assigning the correct label, which is true for both tasks, but especially for are. the performance suffers from shallow processing, i.e. the models are tricked by linguistic surface features like author referencing pronouns in background claims or non-argumentative discourse connectors. furthermore, adur detection struggles a lot in the context of non-contiguous elements which is reasonable because it is trained with incomplete information. this calls for conceptional better modeling of the task, for instance with a joined model for adur and are. finally, we could confirm that sam is a complex problem that is even hard for hu-mans. however, the low inter-annotator-agreement reported by the sci-arg authors and our finding that a significant amount (16%) of the manually analysed are instances are questionable labeled raises the need for even more annotation rounds, maybe with multiple domain experts, or a simplified annotation scheme. figure 4 : distribution of input sequence lengths. this is after splitting the document text into sections and tokenization. note that this is primarily relevant for the adu model since we use a much smaller token window size k to restrict the input for the are model.