A Structure-Aware Argument Encoder for Literature Discourse Analysis (2022)

Abstract: Existing research for argument representation learning mainly treats tokens in the sentence equally and ignores the implied structure information of argumentative context. In this paper, we propose to separate tokens into two groups, namely framing tokens and topic ones, to capture structural information of arguments. In addition, we consider high-level structure by incorporating paragraph-level position information. A novel structure-aware argument encoder is proposed for literature discourse analysis. Experimental results on both a self-constructed corpus and a public corpus show the effectiveness of our model. Resources are available at https://github.com/lemuria-wchen/SAE .

## introduction with the growing amount of scientific literature, researchers pay increasing attention to developing computational methods for analyzing scientific literature @xcite @xcite @xcite , aiming to identify various components of arguments automatically @xcite @xcite @xcite . existing research focuses on constructing annotated corpus and learning representation of sentences for literature discourse analysis. they tend to treat tokens in a sentence equally and ignore the implied structure information of argumentative context @xcite @xcite . figure 1 shows two annotated abstracts of scientific literature, in which sentences are classified into four types, namely background, method, result and conclusion. we have some findings. first, tokens in sentences can be divided into two groups, i.e., topic words and framing words. topic words provide the fundamental knowledge of this argument while framing words organize the expression. second, the same argument components often use similar framing structure in discourses across topics. for example, structures like '... is employed / investigated to ...' usually appear in the method section. third, argument components are sensitive to their positions. for example, background almost always comes before method part and conclusion usually locates at the end. motivated by these findings, we propose a structureaware argument encoder (sae) based on the transformer to enhance the literature discourse analysis. experimental results show the effectiveness of our proposed model both on a self-constructed corpus and a public corpus. ## ccsa corpus there are several public annotated corpora for scientific literature analysis @xcite @xcite @xcite @xcite , most of which focus on medicine and computer science. however, as a highly controversial research area, climate science is less explored. to bridge the gap, we create the climate change scientific argumentation (ccsa) corpus. data source we search for climate change in the isi web of science † 2020 and collect all the retrieved papers published from 2000 to 2020 as the source. the domain of climate change covers a wide range of topics. in order to balance various sub-focus, we use latent dirichlet allocation (lda) @xcite to cluster all papers into different topics and choose similar number of publications from each group for annotation. we set 40 topics in lda, and selected 8 representative, data-rich ones with topic words such as air pollutants, carbon emission policy, global warming and so on. annotation scheme we treat each sentence in the abstract as an argument component and classify them into four types. c1) background explains the motivation and background. c2) method presents experimental procedures. c3) result includes data, facts, and descriptions of outcomes, without any subjective speculations or judgements. c4) conclusion gives opinions of the author. invalid sentences, such as copyright information, are labeled as other types. annotation process undergraduate students are hired for the annotation, about half of them are majored in environmental sciences. we develop a web-based annotation platform and each abstract is annotated by three annotators. the † http://isiknowledge.com/ inter-annotator agreement for argument type annotation is 0.68 in terms of fleiss's kappa coefficient @xcite , which shows a moderate consistency. the final result is determined by majority votes. if there is a disagreement, the label will be determined by the annotator with the greatest confidence † . there were 2,018 abstracts and 18,832 valid argument components in ccsa corpus. ## structure-aware argument encoder in order to incorporate the structure information of an argument, we propose a novel structure for argument representation learning, named structureaware argument encoder (sae). the main component of sae is a transformer structure with multiple attention mechanisms to capture interactions between different groups of tokens. the overall architecture is shown in figure 2 . argument structure in scientific discourse, some technical terms may introduce some noise to the identification of the argument structure. in sae, we divide the tokens in each sentence into framing tokens and topic tokens. framing token contains the structural information in the argument component. topic token contains the topic information in the argument component, such as technical terms in the research field. the sentence is tokenized and tagged with pos (part-of-speech) using nltk @xcite . we regard singular noun (nn), plural noun (nns), singular proper noun (nnp) and plural proper noun (nnps) as topic tokens and others as framing tokens. any method can be adopted for token division, not just pos tagging. ## experiment experimental setup we focus on the task of argument component identification which aims to predict the argument type of argument component (sentences). we conduct experiments on our ccsa corpus. to demonstrate that our sae is domain-independent, we also conduct experiments on another scientific publication abstract corpus biomedical-claims † @xcite . it annotates whether a sentence is a claim, whose setting is similar to ccsa. for ccsa corpus, we take the macro f1 as the evaluation metric of this multi-classification problem, and the f1 score of each sentence type on the test set is also reported. for biomedical-claims corpus, we report precision, recall and f1 score on the test set. the experiment configuration details are shown in a.1. to prove our argumentative attention mechanism has the advantage of modeling topic tokens and framing tokens, we also implement a variant of our sae model that utilizes token types in a simpler way, namely parameterized sae (p-sae). specifically, we initialize a learnable embedding layer for framing tokens and topic tokens instead of argumentative attention mechanism, and add them to token embeddings as input, similar to the segment embedding in bert. compared with sae, p-sae models topic tokens and framing tokens by initializing a trainable parameterized embedding layer at model input. overall performance for ccsa corpus, we compare our sae and p-sae with following baselines: bert @xcite , bidirectional lstm (bi-lstm) @xcite and sentence encoder (se), which contains a bert layer and on top of it, a bi-lstm layer. compared with sae and p-sae, se is a combination of bert and bi-lstm without the information of token types. for biomedical-claims corpus, we present the state-of-the-art model based on transfer learning (tl-crf) in the original paper as baseline @xcite . ## conclusion in this paper, we propose a structure-aware argument encoder (sae) that considers token types in the sentence and separate tokens into two groups, namely topic tokens and framing tokens. multiple argumentative attention mechanisms are utilized to capture internal and external interactions among different groups of tokens. experimental results on a self-constructed corpus and another publicly corpus of scientific literature show the effectiveness of our model.