Argumentation Mining in Scientific Literature for Sustainable Development (2021)

Abstract: Science, technology and innovation (STI) policies have evolved in the past decade. We are now progressing towards policies that are more aligned with sustainable development through integrating social, economic and environmental dimensions. In this new policy environment, the need to keep track of innovation from its conception in Science and Research has emerged. Argumentation mining, an interdisciplinary NLP field, gives rise to the required technologies. In this study, we present the first STI-driven multidisciplinary corpus of scientific abstracts annotated for argumentative units (AUs) on the sustainable development goals (SDGs) set by the United Nations (UN). AUs are the sentences conveying the Claim(s) reported in the author’s original research and the Evidence provided for support. We also present a set of strong, BERT-based neural baselines achieving an f1-score of 70.0 for Claim and 62.4 for Evidence identification evaluated with 10-fold cross-validation. To demonstrate the effectiveness of our models, we experiment with different test sets showing comparable performance across various SDG policy domains. Our dataset and models are publicly available for research purposes.

## introduction arguments are the fundamental building blocks (i.e., groups of statements) in the reasoning path from assumptions to conclusions. argumentation mining (am 2 ) is becoming increasingly a popular topic that addresses the issue of converting unstructured text into structured argument data @xcite @xcite @xcite . am datasets have been developed in various domains such as legal collections (mochales and 1 https://github.com/afergadis/sciark 2 both argument and argumentation mining terms are used in the literature. ## related work a large number of annotated datasets has been released for argumentation mining in various domains @xcite . many researchers approach the claim annotation task under the prism of functional roles @xcite or ownership @xcite . evidence annotation has analogous categorisation, ranging from its strength @xcite to reproduction and generalisation @xcite . however, it is not always possible to apply these fine-grained differentiations to other domains. recently, researchers employed argumentative schemes that incorporate both argumentative units and their relationships. the most common argumentative relationships employed are the "support" and "attack" ones @xcite . researchers also adapt additional relation types from rhetorical structure theory @xcite such as "semantically same" @xcite , "detail", "sequence" @xcite , and "additional" @xcite . although those fine-grained relation types are of great value, their presence in the context of a scientific abstract is limited. the most recent methods for the au classification task leverage neural networks architectures. a common architecture is the one comprising a bil-stm layer followed by a crf layer for sequence tagging @xcite . @xcite evaluate many different neural network models and shows that those based on transformers have better performance on the au extraction and the relation classification tasks. the before mentioned models classify sequences of words within a au ignoring the context before and after the au. we experiment with transformers and bilstm layers taking advantage of the context of aus. most of the datasets based on scientific literature are domain specific with strong emphasis on the biomedical domain @xcite @xcite . other domains are educa-tion @xcite , computer graphics @xcite , and computational linguistics @xcite . sciark, as a multidisciplinary dataset, includes abstract from biomedical, social, environmental and other domains. ## discussion in this section, we discuss some of the erroneous predictions of our model. all possible misclassifications are: neither to claim/evidence, claim to evidence/neither, and evidence to claim/neither. in the model selects sentences for the evidence category mainly from those that report results. however, as we mentioned in section 4.3, the instruction to the annotators was to select the minimum sentences that support the claim. following the instruction, annotators find that sentences 1, 4 and 8 are surplus. however, the model predicted those as evidence because of their position on the abstract and the fact that they report results. finally, to highlight the difficulty of the task we discuss abstract #3. sentence 11 has a claim that "under-five mortality rate is a serious problem". although from the general knowledge of our world we recognise the statement as a claim, in the abstract there is no sentence to support it. on the other hand, the claim that "the hazard of underfive mortality has a decreasing pattern in years" is supported by sentence 10. ## conclusions argumentation mining is a young and gradually maturing research area within computational linguistics. to develop practical applications of importance, we need reliable datasets. in this paper, we describe sciark, a novel sti-driven multidisciplinary dataset with argumentation annotation on six of the 17 sdgs of the united nations. our annotation protocol resulted in a reliable dataset with a significant inter-annotation agreement. we evaluated the dataset on the claim/evidence extraction task using modern deep learning models getting promising results. we also demonstrate the need for multidisciplinary datasets since they could enable models to generalise better on unseen data annotators prediction sentence #1 climate-driven polar motion: 2003-2015 (adhikari and ivins, 2016) 1 neither evidence the changes in terrestrial water storage (tws) and global cryosphere together explain nearly the entire amplitude ... 2 evidence claim we also find that the tws variability fully explains the decadal-like changes in polar motion observed during the ... 3 claim neither this newly discovered link between polar motion and global-scale tws variability has broad implications for the study of ... #2 the impact of large scale biomass production on ozone air pollution in europe (beltman et al., 2013) 4 neither evidence although this scenario is rather conservative, our simulations project that isoprene emissions are substantially increased ... 5 evidence evidence as a consequence, ozone peak values are expected to increase by up to 6%, and ozone indicators for damage to human health ... 6 evidence claim finally, we show that after the change in land use nox emission reductions of 15-20 % in europe would be required to ... 7 claim claim because biomass production is expected to increase throughout europe in the coming decades, we conclude that... #3 comparison of under-five mortality for 2000, 2005 and 2011 surveys in ethiopia (ayele and zewotir, 2016) 8 neither evidence the effect of respondent's current age, age at first birth and educational level on the under-five mortality rate ... 9 evidence neither regarding total children ever born, child death is more for the year 2000 followed by 2005 and 2011. 10 neither claim conclusion: based on the study, our findings confirmed that under-five mortality is a serious problem in the country. 11 claim claim the analysis displayed that the hazard of under-five mortality has a decreasing pattern in years. 12 neither claim our study suggests that the impact of demographic characteristics and socio-economic factors on child mortality should ... #4 renewable energy consumption-economic growth nexus in emerging countries (ozcan and ozturk, 2019) 13 evidence evidence the results indicated that the neutrality hypothesis does hold for all of the markets studied except for poland, ... 14 claim evidence as such, because of the nonexistence of causality running from renewable energy demand to economic growth, energy saving ... 15 neither claim for poland; however, energy conservation policies may have detrimental effects on the country's economic performance level. and outperform systems trained on domain-specific data. sciark facilitates the development of "policy intelligence" by streamlining a big data, stidriven policy modelling approach, improving human judgement for evidence-informed policy making. our sciark framework will be further exploited in adapting policies in the continuously evolving sti landscape, addressing sustainable development.