Can NLI Provide Proper Indirect Supervision for Low-resource Biomedical Relation Extraction? (2023)

Abstract: Two key obstacles in biomedical relation extraction (RE) are the scarcity of annotations and the prevalence of instances without explicitly pre-defined labels due to low annotation coverage. Existing approaches, which treat biomedical RE as a multi-class classification task, often result in poor generalization in low-resource settings and do not have the ability to make selective prediction on unknown cases but give a guess from seen relations, hindering the applicability of those approaches. We present NBR, which converts biomedical RE as natural language inference formulation through indirect supervision. By converting relations to natural language hypotheses, NBR is capable of exploiting semantic cues to alleviate annotation scarcity. By incorporating a ranking-based loss that implicitly calibrates abstinent instances, NBR learns a clearer decision boundary and is instructed to abstain on uncertain instances. Extensive experiments on three widely-used biomedical RE benchmarks, namely ChemProt, DDI and GAD, verify the effectiveness of NBR in both full-set and low-resource regimes. Our analysis demonstrates that indirect supervision benefits biomedical RE even when a domain gap exists, and combining NLI knowledge with biomedical knowledge leads to the best performance gains.

## introduction in silico studies of biology and medicine have primarily relied on machines' understanding of relations between various molecules and biomolecules. for instance, disease-target prediction requires accurate identification of the association between the drug target and the disease @xcite , and drug-drug interaction recognition is essential for polypharmacy side effect studies @xcite . due to the complexity and high cost 1 code is released at https://github.com/luka-group/ nli_as_indirect_supervision of human curation of such biomedical knowledge @xcite , there has been a growing interest in the field of biomedical relation extraction (re), a task of automatically inferring the relations between biomedical entities described in domain-specific corpora. however, two obstacles remain in training a reliable biomedical re model. first, biomedical re often suffers from insufficient and imperfect annotations, due to that the annotation process is very challenging and requires expert annotators to identify complex structures from lengthy and sophisticated biomedical literature. the existing biomedical learning resources either require very costly expert annotations @xcite or resort to weak supervision @xcite . the insufficiency and imperfection of annotations inevitably cause existing state-of-the-art (sota) biomedical re systems @xcite @xcite , though showing satisfactory results in a fully supervised setting, to result in poor generalization regarding the more common low-resource regime in this domain. for example, @xcite showed that model performance deteriorated quickly as the number of instances for each relation drops, hindering the applicability of those approaches in real-world scenarios. second, given that biomedical re annotations tend to be incomplete or have low coverage, it is difficult for models to learn a clear decision boundary @xcite . specifically, in many scenarios where the described biomedical entities are not related in the context, the model may fail to abstain but give a guess from seen relations @xcite ). an overconfident model can be particularly harmful in high-stakes fields such as medicine, where incorrect predictions can have severe direct consequences for patients. recently, indirect supervision @xcite @xcite @xcite is proposed that leverages supervision androgen antagonistic effect of estramustine phosphate @xcite metabolites on wild-type and mutated androgen receptor. upregulator @chemical@xmath0. signals from resource-rich source tasks to enhance resource-limited target tasks. in this approach, the training and inference pipeline of the target task is transformed into the formulation of the source task, thus introducing additional supervision signals not accessible in the target task. recent works @xcite @xcite transfer cross-task learning signals from the natural language inference (nli) task. the nli task aims at determining whether the hypothesis can be entailed given the premise, and inductive bias of nli models learns adaptive generalized logical reasoning which aligns well with the goal of biomedical re. on the other hand, traditional direct supervision on the biomedical re fails to capture semantic information of relations since they are merely transformed to logits of a classifier. by converting relations to meaningful hypotheses in nli, the indirectly supervised method bypasses this shortage and can adapt the the preexisting inductive bias of nli-finetuned models to make meaningful predictions based on relation semantics @xcite . this critically benefits the generalizability of the model in low-resource regimes where limited direct supervision signals are provided @xcite to remedy insufficient annotations. however, previous studies focus on general domain tasks and explore little in specific domains such as biomedical. moreover, to maximize the utility of indirect supervision, it is found that incorporating task knowledge into the model, i.e. nli model that is trained on nli data, yields the best performance @xcite . yet, biomedical nli is rarely available and whether general domain nli can provide strong indirect supervising signals to specific target domains remains unexplored. this study presents a general learning framework, dubbed nli improved biomedical relation extraction (nbr), to enhance biomedical re with indirect supervision from general domain nli task. fig. 1 illustrates the structure of nbr. specifically, given an input sentence, nbr reformulates re to nli by treating the input as the premise while verbalizing each relation label into template-based natural language hypotheses. nbr learns to rank the relations based on the entailment scores such that the hypothesis of a correct relation should be scored higher than those of any incorrect ones. furthermore, to learn a fine-grained, instance-aware decision boundary, nbr deploys ranking-based loss for implicit abstention calibration that handles abstinent relations in the dataset. during inference, the relation whose verbalized hypothesis achieved the highest score becomes the prediction. nbr fully exploits indirect supervision from nli and performs exceptionally well even in low-resource scenarios. our contributions are three-fold: first, to the best of our knowledge, this is the first work to leverage indirect supervision from nli on biomedical re. instead of solely relying on provided re annotations, nbr leverages additional supervision signals from nli indirect supervision and can generalize well in low resource regimes. second, we show that nbr provides a proper indirect supervision signal even if there is a domain gap between general nli knowledge nbr trained on and biomedical downstream task. third, we propose a new ranking-based loss that implicitly handles abstinent relations ubiquitous in biomedical re by contrastively calibrating the score of abstinent instances. by extensive experiments on three commonly-used biomedical re benchmarks, namely, chemprot @xcite , ddi @xcite and gad @xcite , we verify our contributions and show that general domain nli can provide a proper supervision signal, especially in low resource settings where annotations are scarce. nbr provides consistent improvements on three datasets (1.10, 1.79, and 0.96 points of f1 improvement respectively), and up to 34.25 points of f1 improvement in low-resource settings. further analysis demonstrates that combing nli knowledge with biomedical knowledge leads to the best performance gains. indirect supervision. indirect supervision @xcite transfers supervision sig-nals from a more resource-rich task to enhance a specific more resource-limited task. often this line of work reformulates the training and inference pipeline of the target task into the form of the source task to facilitate the cross-task signal transfer. @xcite demonstrate that relation extraction can be solved using machine reading comprehension formulation. similarly, @xcite and @xcite further show that relation extraction performance can be improved by multi-turn question answering and summarization, respectively. recently @xcite and @xcite propose to leverage indirect supervision from the nli task. lite @xcite al. (2022) ) enhances entity typing by incorporating nli and a learning-to-rank training objective while @xcite observes the benefits of indirect supervision in low-resource relation extraction. as discussed, nli aligns well with relation extraction, but to the best of our knowledge, there is no prior work that investigates the effectiveness of indirect supervision when there is a domain gap between the target task and the source task, e.g. biomedical domain and general domain in this study. ## method we hereby present nbr. we discuss how to frame relation extraction as a nli task in §3.2, illustrate how to leverage cross-domain nli knowledge in §3.3, and lastly provide an optional explicit abstention detector to handle abstinent instances in §3.4. ## experiments in this section, we discuss our experiment setup ( §4.1) and evaluation results ( §4.2), followed by detailed ablation studies ( §4.3) and analyses ( §4.4). ## conclusion we present a novel method nbr that leverages indirect supervision by cross-task transfer learning from nli tasks to improve the biomedical re task. nbr verbalizes relations to natural language hypotheses so that model is able to exploit semantic information to make informed predictions. furthermore, nbr adopts a ranking-based abstinent calibration loss that penalizes overconfident abstinent instances while encouraging non-abstinent instances, thus being capable of abstaining on uncertain instances. extensive experiments on three widely-used biomedical re benchmarks demonstrate that nbr is effective in both full-set and low-resource settings. we further investigate two key ingredients for effective nli indirect supervision on biomedical re. future work could involve further investigation of other indirect supervision approaches and automatic relation template generation based on prompt learning.