B io R eader: a Retrieval-Enhanced Text-to-Text Transformer for Biomedical Literature (2022)

Abstract: The latest batch of research has equipped language models with the ability to attend over relevant and factual information from non-parametric external sources, drawing a complementary path to architectural scaling. Besides mastering language, exploiting and contextualizing the latent world knowledge is crucial in complex domains like biomedicine. However, most works in the field rely on general-purpose models supported by databases like Wikipedia and Books. We introduce BioReader, the first retrieval-enhanced text-to-text model for biomedical natural language processing. Our domain-specific T5-based solution augments the input prompt by fetching and assembling relevant scientific literature chunks from a neural database with ≈60 million tokens centered on PubMed. We fine-tune and evaluate BioReader on a broad array of downstream tasks, significantly outperforming several state-of-the-art methods despite using up to 3x fewer parameters. In tandem with extensive ablation studies, we show that domain knowledge can be easily altered or supplemented to make the model generate correct predictions bypassing the retraining step and thus addressing the literature overload issue.

## introduction in the last decade, deep learning advancements have boosted the development of many solutions for effectively extracting knowledge from biological data @xcite and biomedical literature @xcite -widely accessible through repositories such as pubmed, pmc, and sciencedirect. large pre-trained language models (plms) have become the dominant nlp paradigm, achieving unprecedented results in a panoply of tasks, from named entity recognition @xcite and semantic parsing @xcite to information retrieval (moro 1 https://github.com/disi-unibo-nlp biomedical scientific evidence indexing heterogeneity in dna damage within the cell population was observed as a function of radiation dose related context genomic instability and mutation cholestyramine: concomitant * drug * administration decreased the mean auc of total * drug * approximately 55%. ddi-mechanism in which location is dimeric iga most likely to be found in highest concentrations? colostrum query document classification relation extraction ... non-knowledge intensive tasks open-domain question answering ... knowledge intensive tasks bioreader pmid: 12947391 radiation-induced genomic instability is... and valgimigli, 2021) and document summarization @xcite . to justify this success, plms have been shown to implicitly hold a substantial amount of indepth knowledge in their parameters @xcite , resulting from selfsupervised learning on extreme-scale text corpora. efforts to this point have mainly focused on predictably improving nlp performance by increasing datasets, training compute, or model sizes. notably, the most recent transformer-based solutions reach up to 10 11 parameters @xcite , with benefits due to extended memorization of training data @xcite . however, encoding all factual and domain-specific competencies into opaque weight matrices is inefficient, especially for specialized, dynamic, and trust-demanding fields like biomedicine @xcite -where the volume of scientific publications evolves and grows continuously @xcite . indeed, capturing more world facts requires training ever-larger networks, which can be prohibitively slow or expensive. similarly, changing what a plm knows entails retraining the entire model with new documents. moreover, high-dimensional noninterpretable parametric spaces make it difficult to determine what knowledge is stored where, to update the theoretical background, or to provide provenance for decisions. recent developments in the field @xcite @xcite have reversed the architectural scaling trend by showing that smaller plms can perform on par with massive models if we augment them with a way to search for external information. the key intuition is following a retrieve-then-predict approach by asking the plm to directly fetch potentially relevant unlabeled world knowledgeeven structured @xcite -from highly-comprehensive datastores, and use it as additional context during inference. remarkably, semi-parametric (or hybrid) contributions combine "closed-book" (parametric-only) and "openbook" (retrieval-based) methods to complement each source. they allow for revising or even supplementing knowledge dynamically, treating the latter in a more modular and interpretable way. on the other hand, semi-parametric models have so far been only investigated for general-domain knowledge bases and nlp tasks, e.g., context-free question answering conditioned on wikipedia and books evidence. based on previous publications @xcite @xcite , the word distribution shift from general corpora to health informatics corpora prevents or seriously limits the direct application of such models to biomedical nlp. in this paper, we introduce bioreader, the first retrieval-enhanced transformer for biomedical literature, empowered by a differentiable access towards a large-scale text memory grounded on pubmed (≈60m tokens). we continue and build on a broad spectrum of retrieval work in the research community @xcite , exploring efficient means of augmenting biomedical plms with a domain-specific memory, avoiding expanding computations significantly. with bioreader, the biomedical knowledge is not necessary to be implicitly stored in model parameters but is explicitly acquired in a plug-and-play manner, leading to great scalability. mechanically, bioreader is a novel encoder-decoder model based on t5 @xcite and retro @xcite , with a frozen neural retriever. it splits the input sequence into chunks and autoregressively retrieves scientific text semantically similar to the previous fragment. in this way, it expands the input prompt context for better-predicting tokens in the current chunk thanks to a cross-attention mechanism. our main contributions are the following: 1. we devise bioreader, a novel text-to-text biomedical language model fusing memory retrieval and generative components ( §3). 2. we advance biomedical nlp research, pushing the state-of-the-art in several knowledgeintensive and non-knowledge-intensive tasks via fine-tuning ( §4 and §5), outperforming previous methods by a significant margin with up to 3x fewer parameters. we extensively prove the contribution of each module with ablation studies. 3. we show that bioreader can be improved at evaluation time by updating the knowledge base and the number of retrieved neighbors without retraining ( §5), also offering qualitative benefits in terms of interpretability. ## related work we first give a bird's-eye view of existing work on biomedical language modeling and retrievalenhanced neural networks (see ## method motivated by a recent stream of architectural contributions and training modalities @xcite , our proposed text-to-text model bioreader extends t5 with the nimble ability to generate a sequence conditioned by a collection of passages retrieved from a specialized datastore with several millions of biomedical evidence tokens, other than the input. coarse-grained retrieval of contiguous token chunks allows us to retain storage and computation requirements. the unique characteristics of the biomedical text and the related design choices are summarized in §a. figure 2 sketches the overall framework. ## experimental setup implementation details, computing infrastructure, and experiment hyperparameters are described in §b.1 and §b.2. ## results we push the state-of-the-art on 2/7 ner, 1/2 re, 1/1 dc, and 3/3 qa, staying highly competitive in all the other cases. we beat scifive-large (3x our size) on 5 different tasks, while, in the majority of cases, we considerably outperform models which have a comparable number of parameters to ours. overall performances testify to consistent retrieval effectiveness. predicting tokens with the aid of relevant human-written references alleviates the difficulty of text generation. as expected, we notice that ner, re, and nli are the tasks where bioreader contributes less. naturally, all these translations are strictly related to the provided inputs and hardly take advantage of additional external context, which often acts as noise. the strength of bioreader is accentuated when limited training data is available; we point up that our biomedical benchmarks only have few thousand annotated instances ( §c). unexpectedly, the adoption of in-domain vocabularies appears to be non-correlated with higher downstream task scores. from although not retrained, bioreader adapts correctly to unseen questions on the covid-19 literature in "zero-shot datastore" settings ( replicating our solution (see §b) only asks for cca calibration and task-specific model finetuning, ultimately saving a vast amount of computation and memory. our model can be handled on a single gpu machine, while a fully end-to-end retriever generally demands industry-scale computational resources for training @xcite . ## conclusions in this paper, we introduce bioreader, a new state-of-the-art semi-parametric biomedical language model steered by literature passages retrieved from explicit memory. our experimental results show that augmenting the generation process by accessing scientific repositories during training and evaluation induces performance gains greater than raw parameter scaling, both on knowledgeintensive and non-knowledge-intensive tasks. not only do we provide a way of handling the opaqueness of large language models, but we also prove that updating the datastore helps the model with domain adaption without retraining, a property of paramount importance for rapidly evolving domains like biomedicine. future work should aim to integrate a differentiable write-to-memory operator @xcite , structured retrieval databases (e.g., multi-relational graphs from semantic parsing, symbolic knowledge graphs), long text-totext tasks @xcite , knowledge-augmented self-alignment pretraining to rewire the space before retrieval @xcite , and the evaluation of distributed and parallel learning approaches to scale to big data repositories @xcite .