Mitigating Data Shift of Biomedical Research Articles for Information Retrieval and Semantic Indexing (2022)

Abstract: Researchers have explored novel methods for both semantic indexing and information retrieval of biomedical research articles. Moreover, most solutions treat each task independently. However, both tasks are related. For instance, semantic indexes are generally used to filter results from an information retrieval system. Hence, one task can potentially improve the performance of models trained for the other task. Thus, this study proposes a unified retriever-ranker-based model to tackle the tasks of information retrieval (IR) and semantic indexing (SI). Particularly, our proposed model can adapt to rapid shifts in scientific research. Our results show that the model effectively leverages task similarity to improve the robustness to dataset shift. For SI, the Micro f1 score increases by 8% and the LCA-F score improves by 5%. For IR, the MAP increases by 5% on average.

## introduction the pandemic caused the rapidly evolving curation of scientific publications about covid-19, resulting in an information crisis @xcite . as a result, healthcare practitioners, policymakers, and other individuals fighting against covid-19 require specialized information retrieval (ir) and semantic indexing (si) systems to keep track of the ever-evolving literature landscape @xcite . researcher's methods to address these tasks have generally focused on one task, ir or si @xcite . ir and si are related. for example, ir addresses the question, "what are the most relevant research papers, and why are they deemed relevant?" si is essential to facilitate easy browsing and filtering of ir-retrieved manuscripts. for instance, if a user finds all relevant papers related to the question, "what vaccines are the most effective for covid-19?", they can use si to filter papers associated with a specific covid-19 variant, e.g., mesh terms on pubmed @xcite . hence, this paper proposes a novel architecture that jointly addresses both tasks. there has been an array of research in ir and si of biomedical documents. for instance, since 2012, there has been an annual competition where researchers compete to develop more accurate biomedical ir and si methods (bioasq foot_0 ). the competition is essential to improve the national library of medicine's (nlm) infrastructure, which provides ir and si systems for biomedical scientists and healthcare professionals to search for biomedical research articles. nlm manually indexes biomedical research articles with medical subject headings (mesh). mesh terms are used for biomedical si purposes (e.g., filtering search results), to facilitate hypothesis generation by biomedical scientists, and to help general knowledge discovery. unfortunately, there are over 29 thousand mesh terms. thus manually identifying the subset of terms applicable to each article is difficult and expensive to complete in a timely manner. hence, the competition has helped researchers introduce various methods for automated mesh coding. for instance, many researchers have trained linear models, which still result in strong baselines @xcite . for example, @xcite combined linear models with a learning-to-rank framework, which is still used today in combination with neural networks @xcite . similarly, bioasq had a part in advancing biomedical ir systems. for instance, @xcite used convolutional neural networks for biomedical snippet retrieval. similar to bioasq, recent ir efforts have focused on covid-related ir as part of the annual trec competition (trec-covid) @xcite . for example, @xcite evaluated two commercial deep learning ir systems on the trec-covid dataset, showing that both systems underperformed the expected results. researchers have proposed other models beyond the commercial systems, including pre-trained transformer models for text ranking @xcite , along with zero-shot retrieval systems for covid @xcite . some researchers have recently explored combining ir and si. as an example, researchers have used an ir system as part of a knn-based component of an ensemble model to improve mesh identification @xcite . nevertheless, to the best of our knowledge, no prior work has used si to improve ir systems, especially in ir systems for covid-related retrieval. there are four major technical challenges with developing covid-related ir and si systems: sparse datasets, shifts in the data distribution, scale, and interpretability. the limited amount of labeled data and dynamic changes in the covid-19 landscape has made it challenging to generalize ir and si methods beyond the datasets used to train them @xcite ). because information is quickly becoming outdated in research articles, understanding what is relevant is difficult for current ir methods. for example, expert human judgments did not identify 70% of the retrieved results as relevant @xcite . however, the manual assessment process is time-consuming. therefore, it is important to improve current models and provide textual evidence for "why" it detected a document as relevant to facilitate easier manual assessments by human experts @xcite -providing answers to "why" is useful, especially if we develop systems that work to help experts. for instance, @xcite shows that human indexers at nlm become significantly more efficient and accurate if they are provided semantically sensible associations between the input text and system outputs. to address the technical challenges, we propose a specialized ir and si approach that combines interpretability, multi-task learning, and a mechanism of using unlabeled data via self-supervised learning to improve model robustness. overall, our model will allow for quick adaptation and robustness to the dataset shift problem, becoming suitable for the context of the pandemic. we summarize the major contributions of this paper below: (1.) we propose a novel interpretable, self-supervised, multi-task learning method to tackle the tasks of ir and si covid-19-related research articles. we devise a mechanism to train a unified retriever-ranker on a self-supervised masked language modeling (mlm), si, and an ir task. this joint training framework enables inter-document representation learning, quick adaptation to new changes in the data distribution, and interpretability, which we demonstrate to be important for the context of the pandemic. to the best of our knowledge, this is the first study to show the utility of joint training of si and ir tasks-showing both tasks complement each other in a single model, not just one task helping the other one. (2.) we introduce a novel output layer transformation method that allows us to predict new concepts as they appear over time without retraining the model. (3.) our study provides detailed quantitative and qualitative analysis of our model's interpretability and transfer learning components that highlight the dataset shift challenges of ir and si tasks during a health crisis. researchers have been trying to address biomedical natural language processing problems effectively for more than a decade, e.g. bioasq @xcite , which has led to introduction of many models for ir and si @xcite @xcite @xcite . a successful group of submissions involves deep learning models with substantial hand-coded features and supervision. deepmesh @xcite , the best performing model in the bioasq challenge, combines docu-ment to vector models with crafted features from the document and mesh indexes, along with ensemble models fed by those features. other deep learning approaches include uima concept extractor links @xcite , and auth, which also uses a document-to-vector approach with an ensemble of machine learning classifier (svm) fed with document-mesh features @xcite . @xcite and @xcite combined retrieval systems with deep recurrent neural networks and attention mechanism and also provide explainability for mesh indexing decisions. the amount of hand-crafted features and supervision required for these models make it difficult to scale up as the biomedical databases change during pandemic crises @xcite . most si models are developed to perform well in normal situations across a broad range of biomedical concepts. researchers evaluate si models based on their overall performance on all major mesh indices @xcite . in the pandemic situation, however, the focus of the literature has drastically shifted toward the specific concepts and subconcepts related to the current coronavirus disease. the number of published documents related to coronavirus has risen from a few articles per month to more than 10k articles in june 2020-roughly 1 out of every 11.5 citations are about coronavirus these days. @xcite the rapidly growing and evolving literature on covid-19 causes challenges for automatic si models @xcite . previously introduced si models are based on supervised learning approaches and heavily hand-coded features. therefore, they require large amounts of labeled data for a specific concept to perform well. they also have challenges scaling up to newly introduced terminologies and sub-concepts. hence, they are unsuitable for emergencies, like the ongoing health crisis. in this paper, we focus on measuring and improving shifts in this setting. ## datasets this paper uses three datasets: bioasq tasks 8a and 8b dataset, cord-19 @xcite , and trec-covid. we describe each dataset below: bioasq 8a and 8b. first, we use the si and ir datasets that were part of the bioasq 8a and 8b competitions. specifically, we use the pubmed articles from bioasq's @xcite task 8a dataset, which includes almost 15 million article abstracts and titles. we select 8m recent articles published from 2007 to 2019. for ir, we use bioasq's task 8b dataset, which includes 3,243 questions paired with related article abstracts. we use validation sets for each task for hyperparameter tuning. we also use the validation dataset as a pretraining procedure for the covid-19-related corpora. but, we ensure there is no overlap between these general sets and their corresponding covid-19 datasets. ## methods intuitively, our method reformulates the semantic indexing task as ir such that we can train a single model-with a single output layer-that can perform both indexing and retrieval. furthermore, our method does not require learning class-level parameters, thus allowing it to adapt easily to changes in the data distribution. specifically, our method has three main phases: 1. given an input document, we query all similar pubmed articles using a robust ir approach (combining bm25 and document embeddings). 2. we generate document embeddings that combine information from the input document (query) with each candidate (similar) document returned in step 1 (the initial retrieval phase) 3. finally, given the query-candidate joint embeddings, we introduce a novel output layer that can apply to both the semantic indexing (classification) and ir tasks. given the novel output layer, we take advantage of multi-task learning, jointly training the model for si, ir, and additional tasks (self-supervised learning) to improve performance further. we describe each step in the following subsections. initial retrieval. for the first stage, we use an initial retrieval system to identify a subset of related articles along with their task-specific annotations (for example, for extraction of semantic indexes, the task annotations include each candidate article's mesh terms). our initial retrieval system combines a document-level embedding model of specter @xcite with a bagof-words representation fused with bm25 following the schema of @xcite and @xcite . we initialize specter with scib-ert @xcite and trained on a bipartite graph of citations to capture document-level relatedness and minimize a triplet loss between related articles and maximize over unrelated ones. we further pre-train specter on pubmed articles and fine-tune it on the covid-19 dataset only. in addition, we use a bm25 weighted sum of article tokens to compute a keyword-based representation as well. formally, the input query and each candidate document are described as sequences of word tokens, denoted as @xmath0, respectively. for every candidate article, we also track associated metadata such as manually assigned mesh terms defined as l @xmath1, where u d is the total number of mesh terms assigned to candidate document c. we represent every article as an embedding @xmath2 next, we use the cosine similarity scores between each input query representation d q and every candidate article representation d d in our database to find the top k most relevant articles @xmath3, . . . , d k }. ## results evaluation metrics. to evaluate the performance of si we use two sets of evaluation measures; i) flat measures such as micro-and macro-f1 scores, and ii) hierarchical measures such as lowest common ancestor f-measure (lca-f) @xcite for which we leverage bioasq suggested algorithm 7 . for evaluation of ir tasks, we leverage trec_eval, the evaluation metrics and algorithms provided by trec-covid foot_7 . the evaluation metrics include normalized discounted cumulative gain (ndcg@n), p@n, mean average precision (map), and binary preference (bpref) @xcite . baselines. we compare to three baseline models: the default mti, mti first line index, and the top models from the bioasq competition. we explore mti with base and "first line" parameters. mti is a pre-trained model that is for si of biomedical articles by the us national library of medicine. the first line version is the current version used by nlm that partially automates the standard indexing process at the us national library of medicine before human annotators further fine tune the indexes. we also report the scores for the best bioasq team in each batch as "average top score". finally, to compare state-of-the-art methods on the covid data, we retrain attention mesh @xcite . hyperparameters and model variations. we optimize hyperparameters using a held-out validation dataset. for the si experiments, k (i.e., the number of relevant articles retrieved) is set to 512. for the ir experiments, we set k to 1024. we use two versions of our re-ranker, a longformer base version (4 layers, 256 hidden size, 8 heads) and a large version (6 layers, 512 hidden size, 8 heads). furthermore, we evaluate different attention mechanism on the base model. we also experiment with a naïve full attention mechanism (r+tr (full attention)) to compare the effect of the specific attention mechanism suggested by (beltagy et al., 2020) 9 . all hyperparameters were chosen using the valida-9 r+tr (full attention) requires truncation of the input documents, resulting in poor performance. however, longformer uses dilated sliding window attention to avoid truncation. dilation and window sizes are the target hyperparameters here. see the appendix for results with various dilation parameters. all training data micro f1 model lca-f mif maf accu. 0% 5% 10% 20% mti (default) .563 .730 .506 .491 .222 .332 .459 .564 mti (first line indes) .553 .722 .501 .507 .218 .309 .462 .578 attention mesh .579 .764 .529 .558 .271 .396 .504 .619 r+tr (base) (w/o ssl & mt) .540 .700 .492 .485 .307 .433 .504 .591 r+tr (base) (w/ ssl) .552 .728 .506 .510 .380 .486 .616 .663 r+tr (base) (w/ ssl & mt) .563 .755 .511 .523 .485 .592 .656 .724 r+tr (large) (w/o ssl & mt) .562 .742 .502 .523 .363 .474 .559 .595 r+tr (large) (w/ ssl) .597 .777 .532 .569 .490 .619 .698 .733 r+tr (large) (w/ ssl & mt) .612 .810 .558 .586 .564 .676 .741 .789 ## conclusion in this study, we have unified the tasks of ir for question answering with the extraction of semantic indexes and with a self-supervised pre-text task. our approach allows us to simultaneously train on downstream tasks and unlabeled data to maximize the advantages of transfer learning in addressing the data efficiency, generalization, and dataset shift issues. compared to benchmarks, our model learns with less labeled data (it does not even need to learn class-specific parameters) and shows a substantially higher zero-shot (out-of-domain) performance. overall, our study brings focus towards state-of-the-art remedies to the current challenges of the pandemic, which opens up new doors to a more systematic analysis of each of these challenges and more sophisticated algorithms. as future research, we will look to combine more ir and si-related tasks as more data is being annotated and prepared for the domain-specific environment of the pandemic. to better evaluate the performance of the global-local interpretability, we plan to perform qualitative analysis by providing this tool to human experts. the goal is for the tool to improve their time efficiency and perfor-mance when they are performing manual indexing of biomedical research articles.