Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID -19 Open Research Dataset (2020)

Abstract: We present Covidex, a search engine that exploits the latest neural ranking models to provide information access to the COVID-19 Open Research Dataset curated by the Allen Institute for AI. Our system has been online and serving users since late March 2020. The Covidex is the user application component of our three-pronged strategy to develop technologies for helping domain experts tackle the ongoing global pandemic. In addition, we provide robust and easy-to-use keyword search infrastructure that exploits mature fusion-based methods as well as standalone neural ranking models that can be incorporated into other applications. These techniques have been evaluated in the multi-round TREC-COVID challenge: Our infrastructure and baselines have been adopted by many participants, including some of the best systems. In round 3, we submitted the highest-scoring run that took advantage of previous training data and the second-highest fully automatic run. In rounds 4 and 5, we submitted the highest-scoring fully automatic runs.

## introduction as a response to the worldwide covid-19 pandemic, on march 13, 2020, the allen institute for ai (ai2) released the covid-19 open research dataset (cord-19) @xcite . with regular updates since the initial release (first weekly, then daily), the corpus contains around 300,000 scientific articles (as of october, 2020), including most with full text, about covid-19 and coronavirus-related research more broadly (for example, sars and mers). these articles are gathered from a variety of sources, including pubmed, a curated list of articles from the who, as well as preprints from arxiv, biorxiv, and medrxiv. the goal of the effort is "to mobilize researchers to apply recent advances in natural language process-ing to generate new insights in support of the fight against this infectious disease." we responded to this call to arms. as motivation, we believe that information access capabilities (search, question answering, etc.) can be applied to provide users with high-quality information from the scientific literature, to inform evidence-based decision making and to support insight generation. examples include public health officials assessing the efficacy of wearing face masks, clinicians conducting meta-analyses to update care guidelines based on emerging studies, and virologist probing the genetic structure of covid-19 in search of vaccines. we hope to contribute to these efforts via a three-pronged strategy: all three efforts have been successful. in the trec-covid challenge, our infrastructure and baselines have been adopted by many teams, which in some cases have submitted runs that scored higher than our own submissions. this illustrates the success of our infrastructure-building efforts (1). in round 3, we submitted the highest-scoring run that took advantage of previous training data and the secondhighest fully automatic run. in rounds 4 and 5, we submitted the highest-scoring fully automatic runs. these results affirm the quality of our own ranking models (2). finally, usage statistics offer some evidence for the success of our deployed covidex search engine (3). ## ranking components multi-stage search architectures represent the most common design for modern search engines, with work in academia dating back over a decade @xcite @xcite . known production deployments of this design include the bing web search engine @xcite as well as alibaba's ecommerce search engine @xcite . the idea behind multi-stage ranking is straightforward: instead of a monolithic ranker, ranking is decomposed into a series of stages. typically, the pipeline begins with an initial retrieval stage, most often using bag-of-words queries against an inverted index. one or more subsequent stages reranks and refines the candidate set successively until the final results are presented to the user. the multi-stage design provides a clean interface between keyword search, neural reranking models, and the user application. this section details individual components in our architecture. we describe later how these building blocks are assembled in the deployed system (section 3) and for trec-covid (section 4.2). ## the covidex beyond sharing our keyword search infrastructure and reranking models, we've built the covidex as an operational search engine to demonstrate our capabilities to domain experts who are not interested in individual components. as deployed, we use the paragraph index and monot5-base as the reranker. an additional highlighting module based on biobert was described in @xcite . to decrease end-to-end latency, we rerank only the top 96 documents per query and truncate reranker input to a maximum of 256 tokens. the covidex was built using the fastapi python web framework, where all incoming api requests are handled by a service that performs searching, reranking, and text highlighting. search is performed with pyserini (section 2.1), and the results are then reranked with pygaggle (section 2.2). the frontend (which is also open source) was built with the react javascript library to support the use of modular, declarative components, taking advantage of its vast ecosystem. a screenshot of our system is shown in figure 1 . covidex provides standard search capabilities, either based on keyword queries or natural-language input. users can click "show more" to reveal the abstract as well as excerpts from the full text, where potentially relevant passages are highlighted. clicking on the title brings the user to the article's source on the publisher's site. in addition, we have implemented a faceted browsing feature. from cord-19, we were able to easily expose facets corresponding to dates, authors, journals, and sources. navigating by year, for example, allows a user to focus on older coronavirus research (e.g., on sars) or the latest research on covid-19, and a combination of the journal and source facets allows a user to differentiate between preprints and the peer-reviewed literature, and between venues with different reputations. the system is currently deployed across a small cluster of servers, each with two nvidia v100 gpus, as our pipeline requires neural network inference at query time. each server runs the complete software stack in a simple replicated setup (no partitioning). on top of this, we leverage cloudflare as a simple load balancer, which uses a round robin scheme to dispatch requests across the different servers. the end-to-end latency for a typical query is around two seconds. the first implementation of our system was deployed in late march, and we have been incrementally adding features since. based on cloudflare statistics, our site receives around two hundred unique visitors per day and the site serves more than one thousand requests each day. of course, usage statistics were (up to several times) higher when we first launched due to publicity on social media. however, the figures cited above represent a "steady state" that has held up over the past few months, in the absence of any deliberate promotion. ## trec-covid reliable, large-scale evaluations of text retrieval methods are a costly endeavor, typically beyond the resources of individual research groups. fortunately, the community-wide trec-covid challenge sponsored by the u.s. national institute for standards and technology (nist) provided a forum for evaluating our techniques. ## conclusions our project has three goals: build community infrastructure, advance the state of the art in neural ranking, and provide a useful application. we believe that we have succeeded in all three goals. beyond covid-19, the capabilities we've developed can be applied to analyzing the scientific literature more broadly.