S ci NLI : A Corpus for Natural Language Inference on Scientific Text (2022)

Abstract: Existing Natural Language Inference (NLI) datasets, while being instrumental in the advancement of Natural Language Understanding (NLU) research, are not related to scientific text. In this paper, we introduce SciNLI, a large dataset for NLI that captures the formality in scientific text and contains 107,412 sentence pairs extracted from scholarly papers on NLP and computational linguistics. Given that the text used in scientific literature differs vastly from the text used in everyday language both in terms of vocabulary and sentence structure, our dataset is well suited to serve as a benchmark for the evaluation of scientific NLU models. Our experiments show that SciNLI is harder to classify than the existing NLI datasets. Our best performing model with XLNet achieves a Macro F1 score of only 78.18% and an accuracy of 78.23% showing that there is substantial room for improvement.

## introduction natural language inference (nli) or textual entailment @xcite aims at recognizing the semantic relationship between a pair of sentences-whether the second sentence entails the first sentence, contradicts it, or they are semantically independent. nli was introduced @xcite to facilitate the evaluation of natural language understanding (nlu) that significantly impacts the performance of many nlp tasks such as text summarization, question answering, and commonsense reasoning. to date, several nli datasets are made available @xcite @xcite . these datasets have not only been instrumental for developing and evaluating nli models but also have been useful in advancing many other nlp areas such as: representation learning @xcite , transfer learning @xcite and multi-task learning @xcite . however, despite their usefulness, none of the existing nli datasets is related to scientific text that is found in research articles. the vocabulary as well as the structure and formality used in sentences in scientific articles are very different from the sentences used in the everyday language. moreover, the scientific text captured in research papers brings additional challenges and complexities not only in terms of the language and its structure but also the inferences that exist in it which are not available in the existing nli datasets. for example, a sentence can present the reasoning behind the conclusion made in the previous sentence, while other sentences indicate a contrast or entailment with the preceding sentence. these inferences are crucial for understanding, analyzing, and reasoning over scientific work @xcite @xcite . therefore, ideally, the scientific language inference models should be evaluated on datasets which capture these inferences and the particularities seen only in scientific text. to this end, we seek to enable deep learning for natural language inference over scientific text by introducing scinli, foot_0 a large dataset of 107, 412 sentence pairs extracted from scientific papers related to nlp and computational linguistics (cl) and present a comprehensive investigation into the inference types that occur frequently in scientific text. to capture the inference relations which are prevalent in scientific text but are unavailable in the existing nli datasets, we introduce two new classes-contrasting and reasoning. we create scinli by harnessing cues in our data in the form of linking phrases between contiguous sentences, which are indicative of their semantic relations and provide a way to build a labeled dataset using distant supervision @xcite . dur- ## related work to date, several datasets exist for nli of varying size, number of labels, and degree of difficulty. @xcite introduced the rte (recognizing textual entailment) dataset of text-hypothesis pairs from the general news domain and considered two labels: entailment or noentailment (i.e., a hypothesis is true or false given a text). the rte dataset is paramount in develop-ing and advancing the entailment task. the sick (sentences involving compositional knowledge) dataset introduced by @xcite was created from two existing datasets of image captions and video descriptions. sick consists of sentence pairs (premise-hypothesis) labeled as: entailment, contradiction, or neutral. despite being instrumental in the progress of nli, both rte and sick datasets are less suitable for deep learning models due to their small size. in recent years, snli @xcite and mnli @xcite are the most popular datasets for training and evaluating nli models, in part due to their large size. similar to sick, snli is derived from an image caption dataset where the captions are used as premises and hypotheses are created by crowdworkers, with each sample being labeled as: entailment, contradiction, or neutral. mnli is created in a similar fashion to snli except that the premises are extracted from sources such as faceto-face conversations, travel guides, and the 9/11 event, to make the task more challenging and suitable for domain adaptation. more recently, @xcite released anli which was created in an iterative adversarial manner where human annotators were used as adversaries to provide sentence pairs for which the state-of-the-art models make incorrect predictions. unlike the datasets specific to classifying the relationships between two sentences, @xcite combined nli with commonsense reasoning to introduce a new task of predicting the most likely next sentence from a number of options along with their new dataset called swag which was also created with an adversarial approach. however, different from anli, the swag approach was automatic. all these datasets have been widely used for evaluating nlu models and many of them appear in different nlu benchmarks such as glue @xcite and superglue @xcite . heretofore, khot, sabharwal, and clark (2018) created the only nli dataset related to science. their dataset, scitail was derived from a school level science question-answer corpus. as a result, the text used in scitail is very different from the type of text used in scientific papers. furthermore, the sentence pairs in scitail are classified into one of two classes: entailment or no-entailment. thus, scitail does not cover all the inference relationships necessary to understand scientific text. in other lines of research, discourse cues, e.g., linking phrases have been previously used to extract inter-sentence and/or inter-clause semantic relations in discourse parsing @xcite @xcite @xcite , causal inference @xcite @xcite and why-qa @xcite . however, none of the aforementioned bodies of research investigates these relations in scientific text, nor do they exploit the discourse cues to create nli datasets. furthermore, discourse parsing studies a broader range of semantic relations, many of which are unrelated to the task of nli while causal inference and why-qa are limited to only cause-effect relations. in contrast to these tasks, we focus on the semantic relations which are either relevant to the task of nli or highly frequent in scientific text and leverage linking phrases to create the first ever scientific nli dataset, which we call scinli. ## scinli: a new corpus for nli in order to better understand the inter-sentence relationships that exist in scientific text, we started the process of creating our dataset by perusing through scientific literature with the intent of finding clues that are revealing of those relationships. we found that to have a coherent structure, authors often use different linking phrases in the beginning of sentences, which is indicative of the relationship with the preceding sentence. for example, to elaborate or make something specific, authors use linking phrases such as "in other words" or "in particular," which indicate that the sentence supports or entails the previous sentence. we also found that some linking phrases are used to indicate additional relationships that are prevalent in scientific text but are not captured in the existing nli datasets. for instance, when a sentence starts with "therefore" or "thus," it indicates that the sentence is presenting a conclusion to the reasoning in the previous sentence. similarly, the phrase "in contrast" is used to indicate that the sentence is contrasting what was said in the previous sentence. therefore, inspired by the framework of discourse coherence theory @xcite @xcite that characterizes the inferences between discourse units, we extend the nli relations commonly used in prior nli work-entailment, contradiction, and semantic independence-to a set of inference relations that manifest in scientific text-contrasting, reasoning, entailment, and semantic independence ( §3.1). in order to create a large training set with minimal manual effort, we employ a distant supervision method based on linking phrases that are commonly used in scientific writing and are indicative of the semantic relationship between adjacent sentences ( §3.2). we avoid the noise incurred by the distant supervision method in our development and test sets by manually annotating these sets ( §3.3). ## scinli evaluation we evaluate our dataset by performing three sets of experiments. first, we aim to understand the difficulty level of scinli compared to related datasets ( §4.1). second, we investigate a lexicalized classifier to test whether simple similarity based features can capture the particularities of our relations and potentially perform well on our dataset ( §4.2). third, we experiment with traditional machine learning models, neural network models and transformer based pre-trained language models to establish strong baselines ( §4.3). ## analysis research has shown that some stylistic and annotation artifacts are present (only in the hypotheses) in nli datasets created using crowdsource annotators @xcite . to verify that the models do not learn similar spurious patterns in our dataset and predict the labels without understanding the semantic relation between the sentences, we start our analysis by experimenting with only the second sentence as the input to bert and scibert models. next, to intuitively understand the errors made by the models, we perform a qualitative analysis of the predictions made by the scibert model on 100 randomly selected examples from our test set. finally, we show that the neutral examples extracted with firstrand and secondrand approaches are harder to classify than the examples extracted with bothrand. ## conclusion & future directions in this paper, we introduced scinli, the first natural language inference dataset on scientific text created with our novel data annotation method. we manually annotated a large number of examples to create our benchmark test and development sets. our experiments suggest that scinli is harder to classify than existing nli datasets and deep semantic understanding is necessary for a model to perform well. we establish strong baselines and show that our dataset can be used as a challenging benchmark to evaluate the progress of nlu models. in the future, we will leverage knowledge bases to improve the models' ability to understand scientific text. we make our code and the scinli dataset available to further research in scientific nli.