SLEDGE-Z : A Zero-Shot Baseline for COVID -19 Literature Search (2020)

Abstract: With worldwide concerns surrounding the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), there is a rapidly growing body of scientific literature on the virus. Clinicians, researchers, and policy-makers need to be able to search these articles effectively. In this work, we present a zero-shot ranking algorithm that adapts to COVID-related scientific literature. Our approach filters training data from another collection down to medical-related queries, uses a neural re-ranking model pre-trained on scientific text (SciBERT), and filters the target document collection. This approach ranks top among zero-shot methods on the TREC COVID Round 1 leaderboard, and exhibits a P@5 of 0.80 and an nDCG@10 of 0.68 when evaluated on both Round 1 and 2 judgments. Despite not relying on TREC-COVID data, our method outperforms models that do. As one of the first search methods to thoroughly evaluate COVID-19 search, we hope that this serves as a strong baseline and helps in the global crisis.

## introduction the emergence of the severe acute respiratory syndrome coronavirus 2 (sars-cov-2) prompted a worldwide research response. in the first 120 days of 2020, researchers published over 10,000 articles related to sars-cov-2 or covid-19. together with articles about similar viruses researched before 2020, the body of research approaches 60,000 articles. such a large body of research results in a considerable burden for those seeking information about various facets of the virus, including researchers, clinicians, and policy-makers. to help improve covid-19 search, we introduce sledge-z: a simple yet effective zero-shot baseline for coronavirus @xcite what are the initial symptoms of covid-19? ## sledge-z: zero-shot covid-19 search to build a ranking model for covid search, we modify the standard zero-shot vanilla bert document re-ranking pipeline @xcite . we find that while these modifications are simple, they are effective for maximizing ranking performance. we note that this process neither requires covid relevance training data nor involves a priori inspection of the queries and their characteristics. thus, we consider our method zero-shot. to train in a zero-shot setting, we employ a large dataset of general-domain natural language question and answer paragraphs: ms-marco @xcite . however, naïve domain transfer is not optimal since most questions in the dataset are not medical-related, causing a domain mismatch between the training and evaluation data. to overcome this challenge, we apply a heuristic to filter the collection to only medical-related questions. the filter removes questions that do not contain terms appearing in the medsyn @xcite , a lexicon of layperson and expert terminology for various medical conditions. we manually remove several common terms from the lexicon that commonly introduce queries that are not medical-related. for example, medsyn includes the term gas (referring to the medical concept of flatulence in north american english), commonly also refers to gasoline or natural gas. see appendix a.1 for a complete list of excluded medsyn terms. note that we made these decisions without considering covid-19 specificallyonly a broad relation to the medical domain. ms-marco originally consists of 809k questions. after filtering, 79k of the original questions remain (9.7%). we refer to this subset of ms-marco as med-marco. from a random sample of 100 queries from med-marco, 78 were judged by the authors as medical-related, suggesting the filter has reasonable precision. examples questions from this process include causes of peritoneal cancer prognosis and what is squalene anthrax sleep apnea. we make a list of the query ids corresponding to med-marco available, 2 as well as additional examples of filtered queries (see appendix a.2). second, we replace the general-language bert model with a variant tuned on scientific literature (including medical literature). specifically, we use scibert @xcite , which has an identical structure as bert, but was trained on a multidomain corpus of scientific publications. it also uses a wordpiece lexicon based on the training data, allowing the model to better account for subwords commonly found in scientific text. during model training, we employ the pairwise cross-entropy loss function from @xcite . relevant and non-relevant documents are sampled in sequence from the official ms-marco training pair list (filtered down to med-marco queries). third, we apply a filter to the document collection that removes any articles published before january 1, 2020. this filter aims to improve the retrieval system's precision by eliminating articles that may discuss other topics. the date was chosen because little was known about covid-19 prior to 2020, and some documents do not include a full publication date (only a year), making this filter simple to apply. in real-life search engines, date filtering can often be applied at the discretion of the user. ## experimental setup we now explore the ranking effectiveness of our approach. we evaluate the performance of sledge-z using round 1 and 2. at the time of writing, the only training data available for the task was the round 1 data. of the trec-covid information retrieval benchmark @xcite ). foot_3 trec-covid uses the cord-19 document collection (wang et al., 2020) (2020-05-01 version, 59,943 articles), with a set of 35 topics related to covid-19. these topics include natural questions such as: what is the origin of covid-19 and how does the coronavirus respond to changes in the weather. the top articles of participating systems in each round were judged by expert assessors, who rated each article as non-relevant (0), partiallyrelevant (1), or fully-relevant (2) to the topic. in total, 20,728 relevance judgments were collected (avg. 592 per topic), with 74% non-relevant, 12% partially relevant, and 14% fully-relevant. these rates remained nearly constant between rounds 1 and 2. we use normalized discounted cumulative gain with a cutoff of 10 (ndcg@10), precision at 5 of partially and fully-relevant documents (p@5), and precision at 5 of only fully relevant documents (p@5 (f)). both ndcg@10 and p@5 are official task metrics; we include the p@5 filtered to only fully-relevance documents because it exposed some interesting trends in our analysis. we also report the percentage of the top 10 documents for each query that have relevance judgments (j@10). in an additional evaluation, we measure the performance using only judged documents to ensure that unjudged documents do not impact our findings. we used trec eval foot_4 for all metrics. these measures represent a precision-focused evaluation; since reranking methods like ours focus on improving precision, we leave recall-oriented evaluations to future work. our initial ranking is conducted using bm25 with default settings over the full document text to adhere to the zero-shot setting. re-ranking is conducted over the abstracts only, avoiding the need to perform score aggregation (since bert models are limited in the document length). we utilize only the natural-language question (ignoring the keyword query and extended narrative). we conduct an ablation that compares sledge-z to versions using bert (instead of scibert), and the full ms-marco dataset (msm) (rather than the med-marco subset (medm)). we compare with several baselines under the same evaluation settings. including unjudged judged only model training ndcg@10 p@5 p@5 (f) j@10 ndcg@10 p@5 p@5 (f) -fusion: a reciprocal rank fusion method @xcite of bm25 over the abstract, full text, and individual paragraphs. fusion1 uses a concatenation of the keywords and question, and fusion2 uses the entity extraction technique from the round 1 udel submission. 5 our work utilizes a variety of existing opensource tools: opennir (macavaney, 2020), anserini @xcite , and the huggingface transformers library @xcite . we utilize a held-out subset of 200 queries from the ms-marco training set as a validation set for the sole purpose of picking the optimal training epoch. model hyper-parameters were chosen from values in prior work and can be found in appendix a.4, along with information about the hardware used. the vanilla bert and scibert models take approximately 3 hours to train/validate, and inference on trec-covid takes approximately 15 minutes on modern gpus. the bert model has 157m parameters, and the scibert model has 158m parameters. ## results ranking effectiveness is presented in compared to baseline systems (bottom section), we observe that sledge-z offers superior effectiveness. specifically, we see that convknrm, cedr-knrm, and seq2seq t5 all improve upon the initial bm25 ranking. training on med-marco (rather than the full ms-marco) also improves each of the baselines, except, curiously, seq2seq t5. this model may benefit from the larger amount of training data the full ms-marco dataset offers. finally, both fusion methods outperform the base bm25 model. however, we note that these models utilize two fields available for each query: the keyword-based query and the full natural-language question text-a luxury not available in practical search environments. (recall that sledge-z and the other baselines in we now compare our approach with the topperforming submissions to the trec covid shared task (many of which are not zero-shot methods). full participating system descriptions are provided in appendix a.3. we note that these experimental settings for these runs differ from our main experiments. for instance, mpiid5 run3 (li we apply the date filter for a more complete comparison. note that experimental differences exist between our system and these submissions, including the use of multiple topic fields and the utilization of round 1 training data for training or tuning. the top result is marked in bold. et al., 2020) and sparsedensescibert use relevant information from round 1 as training data, and covidex.t5 uses combined keyword query and natural-language questions. therefore, these performance metrics are not directly comparable to our zero-shot runs. despite this, sledge-z still achieves competitive performance compared to these models. for instance, it consistently scores comparably or higher than covidex.t5 (includes a more powerful language model, a more effective initial ranking model, and multiple topic fields) and sparsedensescibert (which uses neural approaches for the initial ranking stage). our method even performs comparably to the mpiid5.run3 model, which was trained directly on round 1 judgments. interestingly, we observe that our simple baseline approach of re-ranking using t5 strictly with the natural-language question against the paper title and abstract (seq2seq t5 in to gain a better understanding of the impact of filtering the document collection to only articles published on or after january 1, 2020, we first com-pare the performance of sledge-z with and without the filter. disregarding unjudged documents, it has an ndcg@10 of 0.668 (-0.051), p@5 of 0.777 (-0.069) and p@5 (f) of 0.589 (-0.108). all these differences are statistically significant. by far the largest reduction is on fully-relevant p@5, meaning that it can be more difficult to find highly relevant documents when considering the full document collection. we observed similar trends for bm25, with and without the 2020 filter. these trends also align with observations we made from the judgments themselves; we find that only 16% of judged documents from prior to 2020 were considered relevant (with only 5% fully relevant). meanwhile, 32% of judged documents after 2020 were considered relevant (19% fully relevant). ## conclusion in this work, we present sledge-z, an adaptation of a neural ranking pipeline for covid-19 scientific literature search. the approach is zeroshot and adapts to medical literature by filtering the training data, using a contextualized language model based trained on scientific text, and by filtering the document collection. the zero-shot setting is important because it suggests that the approach can be generally applied to similar problems-even when no training data are available (which can be expensive to collect). through experiments and analysis on trec-covid, we find that each component of our approach is beneficial, and it outperforms or is comparable to approaches that are trained or tuned on trec-covid judgments. these observations underscore the importance of properly considering the domain when building medical search engines. we hope that techniques like sledge-z can help overcome the global covid-19 crisis.