M i ST : a Large-Scale Annotated Resource and Neural Models for Functions of Modal Verbs in E nglish Scientific Text (2022)

Abstract: Modal verbs (e.g., can, should or must) occur highly frequently in scientific articles. Decoding their function is not straightforward: they are often used for hedging, but they may also denote abilities and restrictions. Understanding their meaning is important for accurate information extraction from scientific text.To foster research on the usage of modals in this genre, we introduce the MIST (Modals In Scientific Text) dataset, which contains 3737 modal instances in five scientific domains annotated for their semantic, pragmatic, or rhetorical function. We systematically evaluate a set of competitive neural architectures on MIST. Transfer experiments reveal that leveraging non-scientific data is of limited benefit for modeling the distinctions in MIST. Our corpus analysis provides evidence that scientific communities differ in their usage of modal verbs, yet, classifiers trained on scientific data generalize to some extent to unseen scientific domains.

## introduction each year, an estimate of 1.5 million scientific articles are published @xcite ; hence, the construction of knowledge graphs (kgs) from scholarly texts for aggregating and navigating research findings is an active research area @xcite @xcite @xcite . professional academic writing makes ample use of hedges, linguistic devices indicating uncertainty, because scientific propositions are usually considered as valid only until they are overwritten by newer findings @xcite . distinguishing valid solutions to problems from unverified and/or potential solutions is a crucial step in information extraction (ie) from scientific text @xcite as kgs should at least mark untested hypotheses as such (see figure 1 ). yet, with the no- internal property capability this sensor can both respond to pressure and tension. sensor_x respond_to_pressure true hascapabilityto hasfactualityrating external action feasibility graphene polymer composites films can be used as protective elements in electronics. graphene_poly-mer_composites electronic_device protective_element possible composedof partof hasfactualityrating hypothesis speculation the inferior results may be related to catalyst particles. catalyst_particles results_paper_x speculation cause hasfactualityrating figure 1: modal verbs perform various functions in scientific text affecting kg representations. in this paper, we focus on modal verbs, a frequently used device for signaling hedging in academic discourse @xcite . other functions of modals include indicating abilities or restrictions. their meaning depends on the sociopragmatic context @xcite , i.e., here on the conventions of the community of a particular academic field. successful academic writing requires correct community-specific use. as shown in figure 1 , understanding the different notions has relevance to kg population (e.g., @xcite . computational modeling of the functions of modal verbs also has applications in language learning and writing assistance software @xcite . prior work in computational linguistics targeting modal verbs @xcite @xcite has primarily worked with data from the news domain. the annotation schemes of these datasets largely follow distinctions estab-lished in the linguistic literature (see, e.g., @xcite @xcite , differentiating between the following coarsegrained modal senses: (a) epistemic expresses judgments about the factual status of a proposition, (b) deontic relates to permission, obligation, and requirements, and (c) dynamic refers to internal abilities or conditions. our work differs from all of these approaches (a) in that we are the first to address the domain of scientific writing, and (b) in that we do not primarily study modal senses, but instead focus on the pragmatic function of modal verbs, i.e., our aim is to capture an author's reason for using a particular modal verb in a context. with this paper, we release mist (modals in scientific text), a manually annotated corpus for investigating the usage of modal verbs in scientific text. our multi-label annotation scheme for modal functions covers semantic, pragmatic, and rhetorical reasons for an author's use of a modal, with a focus on sub-distinctions that are crucial from an ie viewpoint. mist consists of 3737 annotated modal verb instances selected from texts of five scientific disciplines (henceforth domains), which is larger than all existing comparable datasets (see based on mist, as well as related corpora, we conduct an extensive computational study on automatically classifying functions of modals, comparing cnn-based @xcite and bert-based models (similar to @xcite . in contrast to prior modeling work, we circumvent modifying the transformer's input by selecting the modal's contextualized output embedding and/or the cls embedding as input to the classifier. we find that in most cases, a model using both embeddings works best. to sum up, our paper lays the groundwork for both corpus-linguistic and computational work on modeling functions of modal verbs in scientific text. our contributions are as follows. ## related work our work relates to several areas, which we survey in this section. ## mist corpus in this section, we describe our new dataset, including its annotation scheme and detailed corpus and inter-annotator agreement statistics. we annotate instances of can, could, may, might, must, and should in research papers from five scientific fields: computational linguistics (cl), materials science (ms), agriculture (agr), earth science (es), and computer science (cs). modal usage is influenced by sociopragmatic context @xcite and, as a form of hedging, needs to be understood in its social, cultural and institutional context @xcite , here the global scientific community. hence, we do not restrict document selection to native english authors. ## computational modeling we now describe our neural models for classifying functions of modal verbs. we assume that targets have been pre-defined, e.g., using a part-of-speech tagger. our models are based on a pre-trained transformer that provides embeddings for sentences and contextualized token embeddings. we fine-tune scibert (sb, @xcite , which has the same architecture as bert @xcite , but has been trained on large volumes of scientific text. on top, we use multiple classification heads, i.e., one per modal, as the label distributions vary substantially by modal. the largest version of our models is trained jointly on multiple datasets and therefore has the aforementioned output heads for each dataset (see figure 6 ). the output dimension of these heads varies according to the labelset size of the respective dataset. we test the following model variants: sbcls. we feed the cls embedding of an input sentence into a linear layer with softmax (for singlelabel classification) or sigmoid (for multi-label classification) activation. this model uses the same decision basis for all modal verbs within a single sentence. sbmodal. we select the embedding of the word-piece token corresponding to the modal to be classified (modal embedding), 9 and feed this embedding into the linear layer as above. we expect this model to be able to distinguish different modal verbs in the same sentence. the model primarily reflects local context, but to some extent also dependency context @xcite . sbcls,modal. we concatenate the cls embedding with the modal embedding before feeding it into the linear layer. this model should distinguish modal verbs in the same sentence, at the same time leveraging the cls embedding that intends to cover the entire sentence. ## experiments in this section, we report our experimental results. ## conclusion and outlook in this paper, we have introduced a new large-scale dataset of scientific text annotated for functions of modal verbs. our corpus and computational studies reveal differences and similarities in modal usage across genres and domains. we have shown that neural classification is robust across scientific domains, but also that annotated scientific text is essential for good performance. to sum up, our paper lays the groundwork for informed ie from sentences containing modals in scientific texts, e.g., distinguishing speculations from capabilities attributed to a method or device. future work. our experiments on mist point to various next steps, e.g., identifying domainadaptation methods that more effectively leverage annotations across domains, genres, or even languages, or developing data-augmentation techniques targeted to scientific text. another next step is to integrate our methods for generating metadata for facts into ie systems. existing open ie systems do not handle the meaning of modal verbs adequately. as an outlook, in appendix e, we outline how this could be improved using a classifier trained on mist.