MS ci NLI : A Diverse Benchmark for Scientific Natural Language Inference (2024)

Abstract: The task of scientific Natural Language Inference (NLI) involves predicting the semantic relation between two sentences extracted from research articles. This task was recently proposed along with a new dataset called SciNLI derived from papers published in the computational linguistics domain. In this paper, we aim to introduce diversity in the scientific NLI task and present MSciNLI, a dataset containing 132,320 sentence pairs extracted from five new scientific domains. The availability of multiple domains makes it possible to study domain shift for scientific NLI. We establish strong baselines on MSciNLI by fine-tuning Pre-trained Language Models (PLMs) and prompting Large Language Models (LLMs). The highest Macro F1 scores of PLM and LLM baselines are 77.21% and 51.77%, respectively, illustrating that MSciNLI is challenging for both types of models. Furthermore, we show that domain shift degrades the performance of scientific NLI models which demonstrates the diverse characteristics of different domains in our dataset. Finally, we use both scientific NLI datasets in an intermediate task transfer learning setting and show that they can improve the performance of downstream tasks in the scientific domain. We make our dataset and code available on Github.

## introduction natural language inference (nli) @xcite or textual entailment is the task of recognizing the semantic relation between a pair of sentences where the first sentence is called premise and the second sentence is called hypothesis. traditional nli datasets such as snli @xcite , mnli @xcite , sick @xcite , and anli @xcite classify the premise-hypothesis pairs into one of three classes indicating whether the hypothesis entails, contradicts or is neutral to the premise. these datasets have been used both as a benchmark for natural language understanding (nlu) and to improve downstream tasks such as fact verification @xcite and fake news detection @xcite . in addition, they have aided in the advancement of representation learning @xcite , transfer learning @xcite , and multi-task learning @xcite . however, since the examples in these datasets are derived from non-specialized domains, e.g., image captions, they do not capture the unique linguistic characteristics of different specialized domains such as the scientific domain. more recently, @xcite introduced the task of scientific nli along with the first dataset for this task named scinli, which contains 107, 412 sentence pairs extracted exclusively from scientific papers related to computational linguistics published in the acl anthology @xcite . to capture the inferences that frequently occur in scientific text, @xcite extended the three classes in traditional nli to four classes for scientific nli-entailment, rea-soning, contrasting, and neutral. since its introduction, scinli has gained great interest in the research community @xcite @xcite . despite introducing a challenging task and enabling the exploration of nli with scientific text, scinli lacks the diversity to serve as a general purpose scientific nli benchmark because it is limited to a single domain (acl). moreover, due to the unavailability of multiple domains, scinli is not suitable for studying domain adaptation and transfer learning on scientific nli. to this end, in this paper, we propose mscinli, a scientific nli dataset containing 132, 320 sentence pairs extracted from papers published in five different domains: "hardware", "networks", "soft-ware & its engineering", "security & privacy", and "neurips." similar to @xcite , we use a distant supervision method that exploits the linking phrases between sentences in scientific papers to construct a large training set and directly use these potentially noisy sentence pairs during training. for the test and development sets, we manually annotate 4, 000 and 1, 000 examples, respectively, to create high quality evaluation data for scientific nli. we evaluate the difficulty of mscinli by experimenting with a bilstm based model. we then establish strong baselines on mscinli by a) fine-tuning four transformer based pre-trained language models (plms): bert @xcite , scibert @xcite , roberta @xcite and xlnet @xcite ; and b) prompting two large language models (llms) in both zero-shot and few-shot settings: llama-2 @xcite , and mistral @xcite . furthermore, we provide a comprehensive investigation into the robustness of scientific nli models by evaluating their performance under domain-shift at test time. finally, we explore both scinli and mscinli in an intermediate task transfer learning setting @xcite to evaluate their usefulness in improving the performance of other downstream tasks. our key findings are: a) mscinli is more challenging than scinli; b) the best performing plm baseline, which is based on roberta, shows a macro f1 of 77.21% on mscinli indicating the challenging nature of the task and a substantial headroom for improvement; c) the best performing llm baseline with llama-2 shows a macro f1 of only 51.77% indicating that our dataset can be used to benchmark the nlu and complex reasoning capabilities of powerful llms; d) domain-shift at test time reduces the performance; and e) diversity in the scientific nli datasets helps toimprove the performance of downstream tasks. ## related work since the introduction of the nli task, many datasets derived from different data sources have been made available. datasets such as rte @xcite and sick @xcite were instrumental in the progress of nli research in its earlier days. however, the training set sizes of these datasets are too small for large scale deep learning modeling. @xcite was introduced as a large dataset for nli. snli contains 570k sentence pairs where premises are extracted from image captions and human crowdworkers were employed to write the hypotheses and assign the labels. while snli is significantly larger than all other prior datasets, due to the premises being extracted from a single source, it lacks the diversity to serve as a challenging and general purpose nlu benchmark. consequently, @xcite introduced mnli containing 433k sentence pairs where the premises are extracted from a diverse number of sources such as face-to-face conversations, travel guides, and the 9/11 event. apart from the premise sources, both snli and mnli are constructed in a similar fashion and are the most popular nli datasets in the recent years. other nli datasets include qnli @xcite -derived from the squad @xcite question-answering dataset; xnli @xcite -a cross lingual evaluation corpus derived by translating examples from mnli; anli @xcite )-constructed in an iterative adversarial fashion to reduce spurious patterns where human annotators develop examples that can cause the model to make errors in each iteration; scitail @xcite -derived from a school level science question-answer corpus in which the sentence pairs are classified into two classes: entailment or not-entailment. these datasets have also seen wide applications both as nlu benchmarks and to improve other downstream nlp tasks. however, none of these datasets contains sentences from scientific text that is found in research articles. moreover, the classes in these datasets are not sufficient to study the intersentence inferences and complexities that occur frequently in scientific text. thus, to capture both the particularities in scientific text and provide coverage to the frequently occurring inter-sentence semantic relations, @xcite introduced scinli. the sentence pairs in scinli were extracted from papers published in the acl anthology @xcite using distant supervision based on different linking phrases. given that scinli was derived from a single data source (acl), it also lacks the necessary diversity in the data. therefore, with a similar motivation behind constructing mnli-to extend snli to multiple domains, we propose mscinli, the first diverse benchmark for scientific nli, to extend scinli to multiple domains. the availability of multiple domains in mscinli enables ## mscinli: a multi-domain scientific nli benchmark in this section, we describe the data sources for mscinli, its construction process and statistics. ## mscinli evaluation our main experiments for evaluating mscinli consists of three stages. first, we evaluate its difficulty by experimenting with a bilstm model ( §4.1). next, we establish strong baselines on mscinli with four pre-trained language models (plms) and two large language models (llms), and compare them with human performance ( §4.2). finally, we analyze our best performing baseline by investigating its performance when it is finetuned on various subsets of the training set and its performance under domain shift ( §4.3). our implementation details are given in appendix b. additional experiments on the impact of dataset size and diversity in model training; performance of another llm; spurious correlations @xcite ; and class-wise performances of the baselines are shown in appendix c. ## conclusion & future directions we introduce a diverse scientific nli benchmark, mscinli derived from five scientific domains. we show that mscinli is more difficult to classify than the only other related dataset, scinli. we establish strong baselines on mscinli and find that our dataset is challenging for both plms and powerful llms. furthermore, we provide a comprehensive investigation into the performance of scientific nli models under domain-shift at test time and their usage in downstream nlp tasks. in the future, we will develop methods to improve the construction of prompts that enable better reasoning and inference capabilities of llms.