Entangled Relations: Leveraging NLI and Meta-analysis to Enhance Biomedical Relation Extraction (2025)

Abstract: Recent research efforts have explored the potential of leveraging natural language inference (NLI) techniques to enhance relation extraction (RE). In this vein, we introduce MetaEntail-RE, a novel adaptation method that harnesses NLI principles to enhance RE performance. Our approach follows past works by verbalizing relation classes into class-indicative hypotheses, aligning a traditionally multi-class classification task to one of textual entailment. We introduce three key enhancements: (1) Meta-class analysis which, instead of labeling non-entailed premise-hypothesis pairs with the less informative “neutral” entailment label, provides additional context by analyzing overarching meta-relationships between classes; (2) Feasible hypothesis filtering, which removes unlikely hypotheses from consideration based on domain knowledge derived from data; and (3) Group-based prediction selection, which further improves performance by selecting highly confident predictions. MetaEntail-RE is conceptually simple and empirically powerful, yielding significant improvements over conventional relation extraction techniques and other NLI formulations. We observe surprisingly large F1 gains of 17.6 points on BioRED and 13.4 points on ReTACRED compared to conventional methods, underscoring the versatility of MetaEntail-RE across both biomedical and general domains.

## introduction relation extraction (re) is an nlp task that distills factual information from text by identifying relationships between entities in the form of fact triplets (e.g.,〈head, relation, tail〉) (califf and @xcite @xcite . re facilitates various downstream applications such as knowledge graph construction, question answering, and information * corresponding author retrieval @xcite @xcite ; however, creating datasets for training re models is costly and challenging, requiring annotators to identify entities and relations across large sections of text @xcite . recent efforts have explored adapting the re task into a natural language inference (nli) task, where the goal is to determine whether a given hypothesis logically follows from, contradicts, or is neutral with respect to a premise. this adaptation enables the use of relatively large nli datasets to improve performance on an re-adapted task @xcite @xcite . re-to-nli works transform relation instances into premises paired with m class-indicative hypotheses where m is the number of relation classes in a dataset. a language model is trained to label premise-hypothesis pairs as entailed, contradicted, or neutral. we build on this work by introducing metaentail-re, a novel nli adaptation method that improves re performance by leveraging three key enhancements: automatic feasible hypothesis filtering, meta-class analysis, and group-based prediction selection. feasible hypothesis filter: we first introduce a feasible hypothesis filter that automatically removes infeasible hypotheses based on domain knowledge derived from data. to develop this filter automatically, we approximate valid sets of entitytype pairs corresponding to each relation class by aggregating all relations in the training data. these approximated sets of valid type-pairs are then used to remove hypotheses that verbalize infeasible relationships. for instance, in the biored dataset @xcite , it is impossible for a gene to "bind" to a disease (i.e., the "bind" label is not applicable to gene-disease entity-type pairs). we therefore remove the "bind" hypothesis from all instances with gene-disease entity types. this filter improves training efficiency by reducing the number of nli instances. meta-class analysis: in past re-to-nli works, if a premise does not entail a hypothesis, the corresponding nli label assigned is "neutral" @xcite . however, this misses an implicit training signal we can gain by analyzing the semantics of a dataset's relation classes. when assigning nli labels to adapted re instances, we distinguish between task-based mutual exclusivity and definition-based mutual exclusivity. taskbased mutual exclusivity is an artifact of the singleclass classification task inherent to a dataset. each input instance is annotated with a single relation class, thereby arbitrarily making all classes mutually exclusive. in contrast, definition-based mutual exclusivity is derived from definitions of relation classes. for example, within the biored dataset @xcite , the "positive correlation" class is definitionally mutually exclusive and contradictory to the "negative correlation" class @xcite . if two classes are definitionally mutually exclusive, we apply the "contradict" label to the appropriate premise-hypothesis pair, thereby injecting additional information about the meta-relationship between relation classes which the model can exploit while learning relationship representations. leveraging this insight, we can glean multiple informative training signals from a single relation instance when adapting the relation extraction task into the natural language inference task. we call this method meta-class analysis (mca) and use it to determine the appropriate nli labels for each premise-hypothesis pair. we show through ablation experiments that mca leads to significant gains on an re-adapted task. group-based prediction selection: groupbased prediction selection exploits the feature of re-to-nli adaptation in that each relation instance is converted into a group of premise-hypothesis pairs where each hypothesis verbalizes a relation class in the dataset. when evaluating cases where the model predicts multiple "entail" labels within a single group, we can select the most confident "entail" prediction and ignore other predictions. our results demonstrate that this group-based prediction selection method leads to additional gains. metaentail-re as an re-to-nli adaptation method is technically domain agnostic; however, it is particularly well-suited for biomedical re where associations often have opposing classes such as "positively correlated" and "negatively correlated" @xcite or "agonist" and "antagonist" @xcite enabling a rich mca. we also find that associations in biomedical re are often type-dependent compared to general domain re, making the feasible hypothesis filter more effective at trimming infeasible hypotheses. still, we extend our evaluations beyond the biomedical domain to determine how metaentail-re fares on general domain re datasets. notably, we observe improvements in both domains, reinforcing the effectiveness and versatility of metaentail-re. we summarize the main contributions of this work as the following: traditionally, re has been approached as a classification task, where input instances are classified as belonging to a relational class (califf and @xcite @xcite . these methods have several drawbacks: they tend to generalize poorly @xcite , and they heavily rely on relatively small and disjoint re datasets. to account for these drawbacks, recent works have proposed clever adaptation methods to recast re into adjacent nlp tasks, such as a question-answering @xcite and @xcite @xcite . task adaptation presents an opportunity to leverage the relatively large datasets available for other tasks (e.g., squad @xcite , multinli @xcite , snli @xcite , etc.), which can be particularly advantageous in the context of biomedical re where datasets are often limited. @xcite recast re into a questionanswering task by associating relation instances with one or more natural-language questions, resulting in predicted spans denoting class indicative text. obamuyide and vlachos (2018) adapts general domain re into an nli task by using relation hypothesis verbalizer (a) (b) (c) (d) (e) (f) (g) a novel @gene@xmath0. a novel scn5a mutation manifests as a malignant form of long qt syndrome. input fact triplet: 〈scn5a, positively correlated, long qt syndrome〉 surface form abstraction nli labels are generated via meta-class analysis (e), which are the labels used to fine-tune an llm via cross-entropy (f). finally, we use softmax probabilities as a proxy for the model's confidence and select the most confident "entail" prediction among the group of predictions (g). note that the model makes three predictions in this example-one for each feasible hypothesis. the second "entail" prediction is incorrect but the group-based prediction module selects the first and correct "entail" prediction by assessing the model's confidence. ## problem statement our problem is a hybridization of re and nli; as such, we describe both tasks, as well as the adapted re-to-nli task. relation extraction (re): re takes inputs {x 1 , x 2 , . . . , x n } ∈ x re where x re is a corpus of sentences, paragraphs, or documents of size n and x i is a singular instance containing an entity pair e i 1 and e i 2 . each input x i has a corresponding label y i . labels {y 1 , y 2 , . . . , y n } = y re belong to a set of m relation classes @xmath1, r 2 , . . . , r m }. re seeks to identify which class links the co-mentioned entities to form a fact triplet ⟨e i 1 , y i , e i 2 ⟩, or, semantically, 〈head, relation, tail〉. natural language inference (nli): nli takes a premise p @xmath2, where p and h are the set of premises and hypotheses in a corpus, respectively, and seeks to determine whether the premise entails, contradicts, or is neutral to the respective hypothesis @xcite . using ŷi to represent an nli label applied to the i th instance, ŷi ∈ {entail, contradict, neutral}, and a single nli example can be expressed as ⟨p i , ŷi , h i ⟩. re-to-nli adaptation: re-to-nli adaptation converts re inputs and labels into premisehypothesis pairs such that each input instance maps to |r| premise-hypotheses pairs: the set of nli labels ŷ is generated via a function which takes the original instance label y i and the premise-hypothesis pair f target (y i , p i , h j ) → ŷj where nli label ŷj = entail iff verbalized class-indicative hypothesis h j aligns with the ground truth label y i , and, depending on the adaptation method, ŷj is assigned neutral or contradict for nonaligned hypotheses. the re-to-nli task is to correctly predict entailed premise-hypothesis pairs where each entailed pair has a 1-to-1 mapping to the original re label. ## methods this section sequentially discusses the modules used in metaentail-re (see figure 1 ). premise construction: following xu et al. (2023), a relation instance x i is transformed into a premise by replacing surface forms of the subject and object entities, e 1 and e 2 , respectively, with their corresponding entity types, e 1type and e 2type . abstracting entity surface forms into entity types helps alleviate the long-tail nature of biomedical entities and encourages language models to learn from context instead of shallow heuristics @xcite . the start and end spans of entity types are denoted with "@" and "$," respectively. hypothesis verbalizer: past works have manually generated hypothesis templates for each relation class in a dataset which are then used, in turn, to generate hypotheses to pair with a given premise. a secondary contribution of metaentail-re is that we reduce this human effort by leveraging llms to automatically generate the set of hypothesis templates {t 1 , t 2 , . . . , t m } ∈ t , where m corresponds to the number of relation classes in a dataset. we prompt an llm foot_1 to verbalize each relation class using natural language and placeholders for subject and object entities (see appendix a.1 for more details). the placeholders within the hypothesis templates are replaced by the entity types, e 1type and e 2type , found in the premise. feasible hypothesis filter: there is an implicit multiplicative effect of adapting re into an nli task where each relationship instance produces m class-indicative hypotheses resulting in |x re | × m premise-hypothesis pairs. to mitigate this effect, we develop a feasible hypothesis filter which automatically filters out improbable hypotheses by aggregating valid sets of entity-type pairs by relationship classes across all training data: e @xmath3, r 2 → s 2 , . . . , r @xmath4, 2, . . . , m and each s j is the set of tuples of entity-type pairs associated with all instances of relationship class r j . using this filter, we assess the feasibility of hypotheses given a pair of entity types: since sets of feasible hypotheses are approximated using the training data's relationships and entity-type pairs, the filter may remove valid hypotheses based on an entity-type pair and corresponding relation that exists only in the test set. for these instances, the entailed premise-hypothesis pair will not be presented to the model, leading to false negatives. however, in practice, we observe that this does not occur with the datasets we use for evaluation and should not occur as long as training data is sufficiently representative of the test data (i.e., the training data contains at least one relationship with a specific entity-type pair for every relation and entity-type pair found in the test set). ## results we observe an interesting comparison between the biolinkbert large model and metaentail-re. both experiments share the same backbone language model, yet the performance of our metaentail-re method is significantly higher providing evidence of the effectiveness of adapting the re task into one of textual entailment. we hypothesize that the boost in performance primarily comes from the additional data abstraction reto-nli introduces by training the model to recognize entailed premise-hypothesis pairs instead of di-model bc5cdr biored biored (novel) chemprot ddi13 gad traditional multi-class classification biom-albert xxlarge @xcite 0.679 0.668 0.863 0.940 0.911 0.815 biom-bert large @xcite 0.681 0.709 0.904 0.934 0.917 0.795 biomed roberta base @xcite 0.664 0.714 0.897 0.919 0.911 0.803 pubmedbert base @xcite 0.651 0.715 0.891 0.923 0.916 0.803 biolinkbert large @xcite 0.682 0.699 0.899 0.931 0.917 0.806 nli adapted models nbr @xcite 0 rectly predicting suppositional classes. by combining re-to-nli adaptation with surface-form entity abstraction, the model is less prone to memorizing entities and shallow heuristics of relation classes; instead, it must understand the context and the natural language interplay between a premise and hypothesis. furthermore, the boost in performance between the nbr model and metaentail-re highlights the effectiveness of leveraging mca, feasible hypothesis filtering, and group-based prediction selection. within the biomedical domain experiments, the nli-adapted auto-regressive models generally underperform compared to the discriminative models. predictably, the larger phi-3 outperforms phi-2 and fine-tuning smaller auto-regressive models outperforms larger models, gpt 3.5 and gpt 4, leveraging few-shot in-context learning. this aligns with findings from @xcite that llms using in-context learning underperform relative to smaller, fine-tuned language models on information extraction tasks. we observe better performance from autoregressive architectures in the general domain. the performance from phi-3 approaches that of metaentail-re on both retacred and semeval-2010 task 8 datasets which is promising for auto-regressive models, in general. we leave fine-tuning larger auto-regressive models to future work but expect additional gains to be made, potentially overtaking the discriminative models. ## conclusion the exploration of nli techniques to enhance relation extraction has opened new avenues in natural language processing, and our study introduces metaentail-re as an advancement in this area. by adapting the re task into an nli framework and incorporating innovative strategies such as meta-class analysis, feasible hypothesis filtering, and group-based prediction selection, metaentail-re demonstrates remarkable improvements in re performance. our experiments, conducted across biomedical and general domain datasets, highlight the robustness and versatility of metaentail-re. by openly sharing our code, experimental settings, and datasets, we aim to facilitate further research and development in this promising intersection of nli and re, paving the way for more sophisticated and accurate information extraction systems in diverse domains.