A Comprehensive Evaluation of Biomedical Entity-centric Search (2022)

Abstract: Biomedical information retrieval has often been studied as a task of detecting whether a system correctly detects entity spans and links these entities to concepts from a given terminology. Most academic research has focused on evaluation of named entity recognition (NER) and entity linking (EL) models which are key components to recognizing diseases and genes in PubMed abstracts. In this work, we perform a fine-grained evaluation intended to understand the efficiency of state-of-the-art BERT-based information extraction (IE) architecture as a biomedical search engine. We present a novel manually annotated dataset of abstracts for disease and gene search. The dataset contains 23K query-abstract pairs, where 152 queries are selected from logs of our target discovery platform and PubMed abstracts annotated with relevance judgments. Specifically, the query list also includes a subset of concepts with at least one ambiguous concept name. As a baseline, we use off-she-shelf Elasticsearch with BM25. Our experiments on NER, EL, and retrieval in a zero-shot setup show the neural IE architecture shows superior performance for both disease and gene concept queries.

## introduction the amount of text data being produced is overwhelming, especially in biomedicine; pubmed 1 covers over 33 million articles from biomedical and life sciences journals and other texts, with about 1.5 million added each year. meanwhile, many of these articles are about specific entities (e.g. proteins, diseases, chemicals), i.e., entity-centric. in general, entities are central to many search queries; e.g., @xcite demonstrated that 71% of search queries contained named entities, while @xcite found that more than half of the traffic in the allen institute's scholar search engine is about research concepts. in this paper, we describe the design and evaluation of a bert-based ie system as an entitycentric search engine for a target discovery platform pandaomics 2 . in particular, we seek to answer the following research question: considering near excellent performance on ner and el @xcite , are there models capable of finding relevant publications for disease and gene queries from diverse biomedical subdomains as real-world applications? to help answer this question, we develop a novel search collection of pubmed abstracts for disease and gene queries with corresponding relevance judgments. we evaluate the ie pipeline with two trained bertbased models for ner and el and standard document retrieval model bm25 with off-the-shelf elasticsearch software. we perform error analysis on the models' predictions to shed light on future work directions. ## dataset this section describes our dataset, including queries, and the process of collecting relevance assessments. ## models the goal of our work is to evaluate retrieval models in a zero-shot setup, with no available training data to train the ir system. ## evaluation for evaluation, we use precision, recall, and fmeasure. we calculate the precision as a fraction of relevant documents among all retrieved documents. as well the recall is calculated as a fraction of relevant documents from all possibly relevant documents in the dataset. for experiments, we use query-document pairs with relevant and nonrelevant labels excluding the doubtful category. tables 3 and 4 present the performance of the bert-based pipeline compared to bm25 on the full set of queries and the subset of concept with ambiguous names, respectively. several observations can be made based on tables 3 and 4 . first, the bert-based system outperformed bm25 on both sets of the dataset and both types of entities. as expected, the performance difference between the two models is larger on the subset with ambiguous concept names. third, for the bert-based pipeline, precision is higher than recall. in addition, we investigate search precision further by developing a dataset for out-of-domain abstract detection. approximately 30,000 records are included in the pubmed journal list. these journals publish papers not only about biological entities, but also on cultural topics, economics and econometrics, artificial intelligence, law, linguistics and language, and so on (out-of-domain categories for us). our expert annotator manually selected out-ofdomain journals on which we expect the ie system to return zero results. we randomly select 58,790 abstracts from these journals, where each abstract includes at least one gene of disease concept retrieved by elasticsearch. in 90% of these abstracts, the bert-based system did not find any entities. error analysis for error analysis of the bertbased ie system, we reviewed a sample of 152 false positive (fp) documents and 168 false negative (fn) results. ## conclusion and future work in this work, we present a comprehensive evaluation of a biomedical entity-centric search engine based on bert models for disease and gene extraction and linking. this engine is a part of a target discovery platform, where users can return a list of relevant publications given a disease or gene concept query. we evaluate bert models on two information extraction tasks, entity-centric information retrieval, and out-of-domain abstract detection. moreover, we present an error analysis for both retrieval and extraction tasks. this work suggests several interesting directions for future research. we plan to conduct similar studies on other text sources such as full publication texts and patents. moreover, we plan to expand the list of entity types with pathways and biological processes. to extract explicit associations between drug targets and diseases, we plan to add relation extraction/event detection models and study knowledge graph completion with novel disease-gene edges.