B io NLI : Generating a Biomedical NLI Dataset Using Lexico-semantic Constraints for Adversarial Examples (2022)

Abstract: Natural language inference (NLI) is critical in many domains requiring complex decision-making, such as the biomedical domain. We introduce a novel semi-supervised procedure that bootstraps biomedical NLI datasets from positive entailment examples present in abstracts of biomedical publications. We focus on challenging texts where the hypothesis includes mechanistic information such as biochemical interactions between two entities. A key contribution of this work is automating the creation of negative examples that are informative without being simplistic. We generate a range of negative examples using nine strategies that manipulate the structure of the underlying mechanisms both with rules, e.g., flip the roles of the entities in the interaction, and, more importantly, by imposing the perturbed conditions as logical constraints in a neuro-logical decoding system (CITATION).We use this procedure to create a novel dataset for NLI in the biomedical domain, called . The accuracy of neural classifiers on this dataset is in the mid 70s F1, which indicates that this NLI task remains to be solved. Critically, we observe that the performance on the different classes of negative examples varies widely, from 97% F1 on the simple negative examples that change the role of the entities in the hypothesis, to barely better than chance on the negative examples generated using neuro-logic decoding.

## introduction biomedical research has progressed at a tremendous pace, to the point where pubmed 2 has indexed well over 1m publications per year in the past eight years. many of these publications include high-level mechanistic knowledge, e.g., proteinsignaling pathways, which is critical for the understanding of many diseases (valenzuela-escarcega 1 code and data is available at https://github.com/ stonybrooknlp/bionli 2 https://pubmed.ncbi.nlm.nih.gov premise:the outflow of uracil from the yeast saccharomyces cerevisiae is known to be relatively fast in certain circumstances, to be retarded by proton conductors and to occur in strains lacking a uracil proton symport. in the present work, it was shown that uracil exit from washed yeast cells is an active process, creating a uracil gradient of the order of -80 mv relative to the surrounding medium. glucose accelerated uracil exit, while retarding its entry. dnp or sodium azide each lowered the gradient to about -30 mv, simultaneously increasing the rate of uracil entry. they also lowered cellular atp content. manipulation of the external ionic conditions governing delta mu h+ at the plasma membrane had no detectable effect on uracil transport in yeast preparations thoroughly depleted of atp. consistent hypothesis:it was concluded @xmath0. adversarial hypothesis:it is concluded @xmath1. cerevisiae is an active process facilitated by @xmath2. in this work, we introduce a novel semisupervised procedure for the creation of biomedical nli datasets that include mechanistic information. our key contribution is automating the creation of negative examples that are informative without being simplistic. intuitively, we achieve this by defining lexico-semantic constraints based on the mechanism structures in the biomedical literature abstracts. our dataset creation is as follows: we employ this procedure to create a new dataset for natural language inference (nli) in the biomedical domain, called bionli. ## related work previous work on nli in scientific domains include: medical question answering (abacha and demner-fushman, 2016), entailment based text exploration in health care @xcite , entailment recognition in medical texts @xcite , textual inference in clinical trials @xcite , nli on medical history @xcite , and scitail @xcite which is created from multiple-choice science exams and web sentences. these datasets either have modest sizes @xcite , target specific nlp problems such as coreference resolution or named entity extraction @xcite , and make use of experts in the domain to generate inconsistent data which is costly and labor-intensive. additionally, they also focus on sentence-to-sentence entailment tasks, where both the premise and the hypothesis are no longer than one sentence. most importantly, none of these are directly aimed at inference on mechanisms in biomedical literature. our work is also related to nli tasks that go beyond sentence-level entailments. for example, @xcite include premises longer than a sentence, but only use three simple rule-based methods to create negative samples. @xcite use larger contexts as premises for the nli task but only on general purpose domains like news, fiction, and wiki. on the other hand, the bionli dataset is an inference problem with large contexts as premises but in the biomedical domain which often requires handling more complex texts and domain knowledge. there is also a growing body of research into exploring factual inconsistency in text generation models @xcite @xcite . we take advantage of the known weakness of generation models for hallucination and also employ a constraint based neurological decoding from recently introduced decoding methods @xcite to generate adversarial examples for bionli dataset. ## bionli creation we model the task of understanding if a high-level mechanistic statement is supported by lower-level experimental evidence as natural language inference (nli). the goal of nli is to understand whether the given hypothesis can be entailed from the premise or not @xcite . this is typically modeled with three labels (entailed or not, plus a neutral class if the two texts are unrelated). in our case, the premise contains the experimental evidence, while the hypothesis summarizes the higher-level mechanistic information. both of these texts are extracted from abstracts of biomedical publications, where the beginning sentences (the supporting set) describe experimental evidence, and a conclusion sentence summarizes the mechanistic information that is entailed by these experiments. in this work, we introduce the bionli dataset, an nli dataset automatically created from a set of abstracts of pubmed open-access publications. we collected all the abstracts which contain a conclusion sentence with mechanistic information at the end of the abstract, and filter out the rest. following previous work in mechanism generation @xcite , we focus on conclusion sentences that discuss binary biochemical interactions between a regulator and a regulated entity (both of which are proteins or chemicals). we then generate negative examples by manipulating the structure of the conclusion sentences. in the following subsections we describe in detail the generation of both positive and negative examples in bionli. ## evaluation in this section we benchmark the performance of state-of-the-art (sota) biomedical language models on the bionli dataset. our evaluation is aimed at assessing the following aspects of the bionli: 1. how difficult is the inference task captured by the dataset? 2. what kinds of perturbations are difficult for the models? 3. how consistent are the models on adversarial instances? ## conclusion in this paper, we introduced a novel semisupervised procedure for the creation of biomedical nli datasets that include mechanistic information. our key contribution is automating the creation of negative examples that are informative without being simplistic. we achieve this by manipulating the lexico-semantic constraints in the mechanism structures captured in the hypotheses, which we implement both with rules and with neuro-logic decoding. to our knowledge, this is the first paper that employs neuro-logic decoding for the generation of adversarial examples. all in all, we implemented nine different strategies for the creation of adversarial examples. we used this procedure to create the bionli dataset, which addresses nli for mechanistic texts in the biomedical domain. an evaluation of a sample of these negative examples by human biomedical experts indicated that 86% of these examples are indeed true negatives. we trained two state-ofthe-art neural nli classifiers on this dataset, and showed that the overall performance remains relatively low, which indicates that this nli task is not solved. critically, we observe that the performance on the different classes of negative examples varies widely, from 97% accuracy on the simple negative examples that change the role of the entities in the hypothesis, to 55% (i.e., barely better than chance) on the negative examples generated using neuro-logic decoding. we hope that this openaccess dataset foot_1 will enable further research both on unlike many scientific nli datasets @xcite no instance in the bionli dataset was directly annotated by human domain experts. instead, following the trend of machine-generated datasets @xcite , we build upon recent developments in text generation and generate bionli automatically. the only human annotation in this effort was performed by one expert on a sample of 50 sentences, to check the quality of automatically created negative examples. this minimal effort was justified by previous work in the biomedical space (citation hidden for review), in which we observed that experts had high inter-annotator agreement on the interpretations of scientific information in abstracts. the premise-free experiments show the presence of artifacts in some categories of the bionli dataset, similar to several other nli datasets @xcite @xcite . addressing these artifacts remains an open research issue.