Slot Filling for Biomedical Information Extraction (2022)

Abstract: Information Extraction (IE) from text refers to the task of extracting structured knowledge from unstructured text. The task typically consists of a series of sub-tasks such as Named Entity Recognition and Relation Extraction. Sourcing entity and relation type specific training data is a major bottleneck in domains with limited resources such as biomedicine. In this work we present a slot filling approach to the task of biomedical IE, effectively replacing the need for entity and relation-specific training data, allowing us to deal with zero-shot settings. We follow the recently proposed paradigm of coupling a Tranformer-based bi-encoder, Dense Passage Retrieval, with a Transformer-based reading comprehension model to extract relations from biomedical text. We assemble a biomedical slot filling dataset for both retrieval and reading comprehension and conduct a series of experiments demonstrating that our approach outperforms a number of simpler baselines. We also evaluate our approach end-to-end for standard as well as zero-shot settings. Our work provides a fresh perspective on how to solve biomedical IE tasks, in the absence of relevant training data. Our code, models and datasets are available at https://github.com/tba .

## introduction in information extraction (ie) we are interested in extracting structured knowledge from unstructured text. this structured knowledge takes most usually the form of directed binary relations between entities, in other words triples of the form headrelation -tail, which can then be used to populate a knowledge base or a knowledge graph with factual information. the standard approach to perform ie relies on a cascade of natural language processing (nlp) models. first, named entity recognition (ner) is employed to find and extract entities of interest, subsequently entity linking (el) to link the extracted entities to knowledge base identifiers and finally relation extraction (re) to identify existing relations between entities. these individual sub-tasks tasks have attracted a great deal of attention in recent years with methods and datasets fuelling further research @xcite @xcite . ie is largely regarded as a main facilitator of structured data reasoning, such as knowledge base completion. ## related work recent years have witnessed a series of significant advances in the field of qa, primarily owing to the transformer architecture @xcite and the bert self-supervised pre-training paradigm @xcite . these advances, both in terms of methods @xcite @xcite and datasets @xcite , motivated researchers to formulate a series of different nlp tasks as open domain qa, including entity linking or relation extraction @xcite . in this work we follow this paradigm by formulating biomedical ie as a slot-filling task. in open domain qa, given a query, a retrieval module first retrieves relevant documents from the knowledge source (such as wikipedia). a reading comprehension module is then used to extract a span from the relevant documents, the answer. the retrieval step was, up to very recently, dominated by statistical-based approaches, namely bm25 or tf-idf @xcite . orqa @xcite and realm @xcite have been the first neural based methods to clearly outperform statistical based retrieval, although they required expensive language model pre-training. dense passage retrieval (dpr) @xcite has improved upon these methods by employing bertbased encoders, one for the queries and one for passages. these are jointly optimized during training to classify passages as relevant versus irrelevant. this approach has proved superior to other neural based approaches and has quickly become the preferred method for open domain qa in subsequent work @xcite @xcite . among the subsequent works, retrieval augmented generation @xcite ) employs an architecture based on dpr and bart @xcite ) that is optimized end to end during finetuning, to retrieve relevant documents and generate answers to queries. fusion-in-decoder (izacard and grave, 2021) employs dpr or bm25 as retrievers coupled with a t5 language model, to generate answers by attending at multiple passages simultaneously. for simplicity, we are not considering these approaches in this work, leaving their implementation for the biomedical domain for future work. in an effort to fuel further research on this field, @xcite introduced kilt, a new benchmark of knowledge intensive tasks, which contains among others two slot filling datasets, zero-shot re which was first presented in @xcite and t-rex introduced by @xcite . in building our biomedical slot filling dataset we largely follow the conventions and format of kilt, with the intention to ease experimentation. finally, @xcite have presented a rag model specifically finetuned for slot filling on the above datasets, showing significant improvement over the generic alternatives, which were finetuned on natural questions (nq). ## biomedical slot filling formally, let us first define the task of ie. we assume a knowledge source k, consisting of passages p i . furthermore, we assume there exists a knowledge base that contains a number of entities e i . our goal is to extract from k all possible triples of the form e a -r i -e b where r @xmath0. for each e i we assume that it has a specific entity type e t and that each e t can be involved in a specific subset of r. slot filling further formulates the above task as follows: we first employ a retrieval model m r that encodes all passages p i from k. the encoded passages are indexed to allow fast retrieval. at inference, for each e i of type e t , we consider all possible relations from r and construct the relevant queries q i : e i -r i . each query is then encoded and the resulting vector is used to query the index, returning the n most similar p i in terms of the maximum inner product: here, we adopt as m r a neural, dense bi-encoder, namely dpr, which uses a different encoder for passages and queries, but any type of retriever can be used such as bm25, where e @xmath1. we initialize dpr's encoders with the ones presented in @xcite which were finetuned on the nq benchmark. we subsequently train dpr on the dataset presented in section 4, with the following loss function: regarding the reader comprehension model m qa , we employ a pretrained biobert @xcite ) model and finetune it on the dataset of section 4. to finetune we follow the standard approach for question answering with bert where the input is the concatenated query and passage separated by special token [sep] and the outputs are the start and end token positions within the passage. the training objective is the sum of the log-likelihoods of the correct start and end positions. for more details we refer the interested reader to @xcite . ## biomedical slot filling dataset in order to build a slot filling dataset for biomedicine, we resort to a number of publicly available biomedical ner and re datasets, summarized in specifically for the retriever training, we use negative, i.e., null relation instances, as negatives. @xcite compound-disease 1 15,796 biocreative vi chemprot @xcite compound-protein 9 15,568 ddiextraction 2013 @xcite drug-drug 1 32,018 ## experiments in this section we present the experiments that we conducted, followed by a discussion on their implications. we are interested in evaluating our biomedical dpr retriever, our biomedical slot filling reader and finally the end to end slot filling approach. ## conclusions and future work in this work we formulated the task of biomedical information extraction as a slot filling problem. this approach aims to forgo the need for entity and relation type specific training data, which is scarce and costly to annotate in the biomedical domain. additionally, this formulation allows to deal with the addition of new relation types, without needing to re-train the relevant models. additionally, we have introduced a new biomedical slot filling benchmark and used it to train a biomedical dpr model, a dual bert-based encoder for retrieval, as well as a biomedical slot filling reader based on biobert. in a series of experiments our approach outperforms significantly a number of general domain baselines as well as the simpler bm25 retriever. furthermore, our results illustrate the importance of in-domain, taskspecific training data, in line with findings from recent works @xcite . in future work, we aim to focus on sequence to sequence variants of this work such as the work in @xcite , as well as to conduct a thorough comparison of a standard biomedical ie system against our slot filling approach.