Do Discourse Indicators Reflect the Main Arguments in Scientific Papers? (2022)

Abstract: In scientific papers, arguments are essential for explaining authors’ findings. As substrates of the reasoning process, arguments are often decorated with discourse indicators such as “which shows that” or “suggesting that”. However, it remains understudied whether discourse indicators by themselves can be used as an effective marker of the local argument components (LACs) in the body text that support the main claim in the abstract, i.e., the global argument. In this work, we investigate whether discourse indicators reflect the global premise and conclusion. We construct a set of regular expressions for over 100 word- and phrase-level discourse indicators and measure the alignment of LACs extracted by discourse indicators with the global arguments. We find a positive correlation between the alignment of local premises and local conclusions. However, compared to a simple textual intersection baseline, discourse indicators achieve lower ROUGE recall and have limited capability of extracting LACs relevant to the global argument; thus their role in scientific reasoning is less salient as expected.

## introduction arguments are made by presenting cascades of argument components (acs) called premises and conclusions, where the premises are intentional justifications that lend credibility to the conclusions @xcite . in scientific papers, arguments aim to make claims supported by evidences taken from experiments, observations, and references @xcite , and are usually presented as premise-conclusion pairs that are linked via an argumentative relation @xcite . in scientific papers, the main claim or global argument of a paper is drawn in the abstract and several local argument components (lacs) are formulated throughout the entire example lac in our dataset assuming that [gene duplications primarily evolve under purifying selection]premise, [the observed acceleration of evolution may be explained by epistatic interaction between gene copies]conclusion. regex rule: assuming that p, c it has been claimed that discourse indicators can be used to extract acs in unstructured text, such as news articles @xcite and student essays @xcite . however, the alignment between premises and conclusions in scientific papers is often implicit, especially when several premises correspond to one particular conclusion. moreover, the extraction rules for acs strongly depend on the pre-defined argumentation scheme and often do not generalize well @xcite . @xcite have annotated a small corpus of 24 scientific papers, but the argumentative relation scheme is only binary (attack or support) and thus cannot represent more complex argumentative relations. finally, the relation between arguments in the abstract and the body text remains understudied. therefore, although a lot of progress in mining arguments from unstructured texts @xcite @xcite @xcite has been made, it remains unclear whether discourse indicators can extract lacs that support the global argument in structured texts such as sci-entific papers. in this work, we create a sizeable scientific paper dataset consisting of biomedical papers with well-structured abstracts, which enables us to easily extract the global argument of papers. on this dataset, we propose an efficient discourse indicatorbased lac extraction pipeline. we first construct a set of regular expressions of argument-associated discourse indicators; then, for each regular expression, we define how the local premise and the local conclusion are organized either in the sentence or in two consecutive sentences that are linked by this discourse indicator. with this pre-defined set of rules, we extract and disentangle the local premise from the local conclusion, which serve as lacs (see our main contributions are: 1) we propose a set of regular expressions for over 100 wordand phrase-level discourse indicators for extracting lacs from the body text of scientific papers; 2) we show that counter-intuitively, lacs extracted by discourse indicators only poorly reflect the global argument, by the fact that lacs extracted with discourse indicators achieved lower rouge-n scores than a simple baseline approach; 3) human evaluation results suggest that lacs extracted by discourse indicators are precise in the exact wordings, but do not have a high information coverage of the global argument. ## related works the task of extracting lacs is most similar to argument mining @xcite @xcite , which typically classifies sentences into argumentative and non-argumentative text according to their rhetorical and syntactic role. argument mining usually depends on a carefully designed argumentation scheme, which is, in general, a predefined type of connection between premise and conclusion. @xcite proposed the first argumentative scheme which was later expanded to 14 categories of acs (e.g. aim, support, use, etc.) in scientific texts @xcite . in our work, we consolidate the argumentation scheme simply as premise-conclusion pairs. discourse indicators have been used as rhetorical features to determine the credibility of claimed premises in support of a conclusion @xcite . as a milestone, @xcite showed that premise-conclusion pairs could be located by discourse indicators. @xcite annotated a corpus including 88 german language documents of premise-conclusion pairs and found that particular discourse indicators are more closely linked to either premises or conclusions. @xcite used a small set of discourse indicators to extract premise-conclusion pairs and achieved high precision in recognizing the connections between propositional segments. in their later work @xcite , they further leveraged contextual knowledge such as topic information by constructing an inferential matrix that indicated the propositional relations, including premise-conclusion pairs. argument mining has also been studied in series of works of moens et al. @xcite @xcite , where sentences are classified into arguments and non-arguments in an unsupervised manner using syntactic and semantic features. in these studies, the extraction of acs is mainly done on the sentence level. nevertheless, in these works discussed above, the contribution of discourse indicators alone is not clear, and often the power of discourse indicators are only partially studied for news articles (e.g. @xcite ). unlike news articles, which are often written in plain language and are easy to understand, the readability of scientific papers decreases over time @xcite . in this work, we focus on understanding the role of discourse indicators in scientific papers particularly, mainly on how they contribute to extracting lacs in the body text supporting the global argument in the abstract. ## methodology this section outlines our approaches to extracting global arguments and lacs from scientific papers (see figure 1 for the proposed pipelines). in this work, we use the term global and local to refer to argument components located in the abstract and the body text of a paper separately. we make the following assumptions: 1) every scientific paper has one global argument and several paired lacs. the global argument expresses the textual intersection approach only makes use of method, result and conclusion sections, while the discourse indicator approach leverages the whole body text. we use well-structured abstracts to get the best human labeled global arguments we can find. the paper's central claim, whereas lacs are individual statements that support the global argument from diverse perspectives; 2) the global argument locates in the paper's abstract, whereas lacs are distributed across the entire body text of the paper. ## dataset our proposed argument mining pipelines are applied to the semantic scholar open research corpus, i.e., s2orc foot_3 @xcite , which is an extensive collection of 81.1m well-parsed peerreviewed english papers, among which around 12.7m are complete with full text. from the s2orc corpus, we create a subset of nearly 28k papers in the biomedical domain with full text and structured abstracts available. we use papers with well-structured abstracts of biomedical papers to extract the global arguments due to the following reasons: 1) well-structured abstracts are the best massive human annotated source of global arguments we can get since these papers are peer-reviewed and usually multi-round editor-revised, therefore the quality of the argumentative text is ensured; 2) many journals specialized for biomedicine research naturally require the authors to construct the abstract in a structured manner, where the argumentative text is purposely decomposed into different units; 3) a previous study @xcite demonstrates the success of generating global conclusions from global premises using the well-structured abstracts of pubmed papers, thus enlightening the usefulness of well-structured abstracts for mining argument components. for each paper in our dataset, we extract the lacs using both discourse indicators and textual intersection approaches. we also compute the upper bound of the rouge f-measure performance using the greedy strategy of @xcite that iteratively selects sentences to approximately maximize the sum of rouge-1 and rouge-2 f-measure scores. ## evaluation to evaluate the performance of our proposed approaches, we perform the local-to-global comparison between the lacs and the global argument us- we first concatenate the lacs within their original order of occurrence in the body text, then we average the rouge-1, rouge-2, and rouge-lsum scores foot_4 for precision, recall, and f-measure. all evaluations are performed separately for 1) the extracted local conclusions (against the global conclusions); 2) the extracted local premises (against global premises). discourse indicators themselves are excluded from lacs while computing the rouge scores. in addition, we are particularly interested in the n-gram precisions of the lacs compared to the global argument, since they provide information about whether n-grams in the global argument are favored in local conclusions or local premises. therefore, we use the rouge-n precision as the metric to evaluate the lexical preferences of lacs. ## human evaluation following the evaluation setups proposed by @xcite , we conducted a human evaluation on how well lacs extracted with the two proposed approaches reflect the global argument. the human evaluation is designed as a text comparison task where we asked the evaluators to choose between the lacs extracted by the two approaches in an interactive ui interface setting (see figure 5 in appendix e), by carefully reading the text displayed on the interface. we recruited 6 human evaluators with strong biology/neuroscience backgrounds. each evaluator was asked to evaluate 25 randomly picked samples from our proposed scinf-biomed dataset. lacs extracted by discourse indicators and textual intersection were randomly displayed in separate text wrappers @xcite . in order to prevent the evaluators from inferring the lacs extraction method, we presented the lacs extracted with discourse indicators as complete sentences. to discount for lac length (as in figure 2 ), we truncated lacs to the first 100, 200, and 300 unigrams, respectively. the evaluators were asked to choose the better extractor (value of #1) for each of the following criteria: ## conclusion in this work, we investigate the effectiveness of discourse indicators for retrieving lacs relevant to the global argument of scientific papers. we develop a set of regular expressions for over 100 word-and phrase-level discourse indicators and test the performance of extracting the lacs of scientific papers. our preliminary results show that discourse indicators have a limited capability of capturing lacs that are well-aligned with the global argument and thus cannot be solely used to extract arguments from scientific papers. in future works, we will explore the effectiveness of discourse indicators in different types of scientific paper, such as research article, case report, and technical notes, etc. at the moment a notable weakness of our work is the oversimplifying use of regular expressions to disentangle premises from conclusions, thus we believe that the extraction of lacs using discourse indicators may be improved using more sophisticated (hierarchical) parsing techniques. in addition, we will work on a gold standard dataset that consists human annotated premise-conclusion pairs for argument generation, at the same time investigate the power of other more advanced contextualized sentence encoders.