A MISMATCHED Benchmark for Scientific Natural Language Inference (2025)

Abstract: Scientific Natural Language Inference (NLI) is the task of predicting the semantic relation between a pair of sentences extracted from research articles. Existing datasets for this task are derived from various computer science (CS) domains, whereas non-CS domains are completely ignored. In this paper, we introduce a novel evaluation benchmark for scientific NLI, called MisMatched. The new MisMatched benchmark covers three non-CS domains–Psychology, Engineering, and Public Health, and contains 2,700 human annotated sentence pairs. We establish strong baselines on MisMatched using both Pre-trained Small Language Models (SLMs) and Large Language Models (LLMs). Our best performing baseline shows a Macro F1 of only 78.17% illustrating the substantial headroom for future improvements. In addition to introducing the MisMatched benchmark, we show that incorporating sentence pairs having an implicit scientific NLI relation between them in model training improves their performance on scientific NLI. We make our dataset and code publicly available on GitHub.

## introduction the task of natural language inference (nli) has received significant attention, initially through several pascal recognising textual entailment (rte) challenges @xcite @xcite which focused on recognizing if two given sentences exhibit an entailment relationship. subsequently, several nli datasets @xcite @xcite have been introduced to facilitate progress on the nli task. more recently, there has been an increasing interest in domain specific nli tasks, including scientific nli @xcite . the scientific nli task classifies the semantic relation between a pair of sentences extracted from research articles into one of four classes-entailment, reasoning, contrasting, and neutral. this task is challenging for both pre-trained small language models (slms) and large language models (llms) @xcite , making it suitable to serve as a challenging benchmark for evaluating the natural language understanding of state-of-theart models. in addition, @xcite have shown that scientific nli can aid in improving the performance of other downstream tasks such as topic classification and citation intent classification. to date, two datasets have been made available to facilitate research on scientific nli-scinli @xcite , and mscinli @xcite . scinli is derived from papers published in the acl anthology, related to natural language processing (nlp) and computational linguistics. to introduce diversity in scientific nli, mscinli is constructed using sentence pairs extracted from five different scientific domains-hardware, networks, software & its engineering, security & privacy, and the neurips conference which is related to machine learning. the training sets of these datasets are constructed using a distant supervision method that harnesses explicit signals conveyed by various linking phrases. for example, if the second sentence in an adjacent sentence pair starts with "however" or "in contrast," the sentence pair is labeled as contrasting. the test and development sets of both scinli and mscinli are human annotated to ensure a realistic evaluation. despite the diversity introduced in mscinli, the domains covered by the existing scientific nli datasets are still related to only computer science (cs), while non-cs domains are completely ignored. thus, in this paper, we propose a new evaluation benchmark for scientific nli called mis-matched, which contains sentence pairs collected from 3 non-cs domains: psychology, engi-neering, and public health. we constructed mismatched as an out-of-domain (ood) testbed for scientific nli models. that is, mis-matched contains only development and test sets that are human annotated (of sizes 300 and 2400, respectively), without any training data. mis-matched is designed as an out-of-domain (ood) benchmark for evaluating the robustness of scientific nli models, similar to the mismatched (mm) portion of mnli @xcite . like mnli's mismatched set, which uses unseen genres to test model generalization, mismatched is aimed at evaluating ood robustness when models are trained on existing scientific nli training sets. we establish strong baselines on mismatched by fine-tuning four slms-bert @xcite , scibert @xcite , roberta @xcite and xlnet @xcite ; and by prompting four llms-llama-2 @xcite , llama-3 @xcite , mistral @xcite and @xcite using the training sets from existing scientific nli datasets. we find that our best performing slm baseline with scibert and best performing llm baseline with phi-3 show macro f1 of only 78.17% and 57.16%, respectively, illustrating the highly challenging nature of the mismatched set, and a significant amount of headroom for future improvements. in addition, given that all sentence pairs in the training sets of existing scientific nli datasets are constructed using distant supervision that harnesses explicit relations conveyed by various linking phrases, we analyze the impact on models' performance of sentence pairs which have an implicit relation between them (i.e., sentence pairs which are adjacent in text and have a scientific nli relation but the second sentence in the pair does not start with a linking phrase). we find that incorporating implicit relations can indeed improve the performance of scientific nli models. our key contributions can be summarized as follows: ## related work since the introduction of the nli task @xcite , numerous datasets have been introduced that include sentence pairs from the general domain. rte @xcite ) is an early dataset, which went through several iterations @xcite @xcite . the rte dataset contains premise-hypothesis pairs, which are labeled as entailment or non-entailment. more recent datasets such as sick @xcite , snli @xcite and mnli @xcite contain sentence pairs that are classified as entailment, contradiction, or neutral. the sick dataset @xcite ) contains sentence pairs automatically extracted from paired image captions and video descriptions. snli @xcite contains premise-hypothesis pairs, where the premises are extracted from image captions, and the hypotheses are manually written by human annotators. mnli @xcite contains premise-hypothesis pairs where premises are extracted from a variety of sources such as travel guides and face-to-face conversation, while the hypotheses are manually written by human annotators, as in snli. anli @xcite , another nli dataset, was constructed in an adversarial fashion with human annotators in the loop who were instructed to write sentence pairs for which the models make mistakes in their predictions. several domain-specific nli datasets have also been introduced. for example, mednli (romanov and shivade, 2018) was derived from medical records of patients with the premise-hypothesis pairs being annotated by experts (in the medical domain) as entailment, contradiction, or neutral. the nli4ct dataset @xcite ) contains premise-hypothesis pairs, in the form of clinical trial reports (ctr) and statements, labeled as entailment or contradiction by human annotators. nli4ct-p (perturbed) @xcite is an extension of the original nli4ct dataset @xcite and was obtained by adding a contrast set derived from perturbations to the original statements, to facilitate causal analyses. ## the mismatched benchmark in this section, we describe our proposed mis-matched benchmark for scientific nli. specifically, we outline the data sources for deriving mis-matched, the construction process and the key statistics. ## baselines since mismatched only consists of dev and test, we use the training sets of scinli and mscinli, containing 101k and 127k sentence pairs, respectively, to establish the slm and llm baselines. our implementation details are in appendix c. ## harnessing implicit relations existing training sets for scientific nli datasets (i.e., scinli and mscinli) only include sentence pairs where the relation between them is made explicit with linking phrases. we posit that, if two sentences are adjacent, there can potentially be a scientific nli relation between them despite the second sentence not starting with a linking phrase. we define the relation between these sentence pairs as an implicit relation. here, we propose to incorporate adjacent sentences with implicit relations in model training and analyze their impact on models' performance. we detail below the data sources from which we extract implicit sentence pairs, how we annotate them, and how we use them in model training. ## conclusion & future directions in this paper, we introduce a mismatched testbed for scientific nli, derived from non-cs domains unlike the existing datasets. we establish strong baselines on the mismatched set with both slms and llms using the training sets from scinli and mscinli. our results show that the best performing baseline achieves a macro f1 of only 78.17%, illustrating the substantial room for future improvements. furthermore, we show that sentence pairs containing an implicit scientific nli relation can aid in improving the performance of two scientific nli benchmarks. in our future work, we will develop domain adaptation methods for scientific nli to improve the performance on the mismatched set.