Pandemic Literature Search: Finding Information on COVID -19 (2020)

Abstract: Finding information related to a pandemic of a novel disease raises new challenges for information seeking and retrieval, as the new information becomes available gradually. We investigate how to better rank information for pandemic information retrieval. We experiment with different ranking algorithms and propose a novel end-to-end method for neural retrieval, and demonstrate its effectiveness on the TREC COVID search. This work could lead to a search system that aids scientists, clinicians, policymakers and others in finding reliable answers from the scientific literature.

## introduction as covid-19-an infectious disease caused by a coronavirus-led the world to a pandemic, a large number of scientific articles appeared in journals and other venues. in a span of five months, pubmed alone indexed over 60,000 articles matching coronavirus related search terms such as sars-cov-2 or covid-19. this volume of published material can be overwhelming. there is a need for effective search algorithms and question answering systems to find relevant information and answers. in response to this need, an international challenge-trec covid search @xcite )-was organised by several institutions, such as nist and allen institute for ai, where research groups and tech companies developed systems that searched over scientific literature on coronavirus. through an iterative setup organised in different rounds, participants are presented with several topics. the evaluations measure the effectiveness of these systems in finding the relevant articles containing answers to the questions in the topics. we propose a method that improves the systems developed for the trec-covid challenge by adopting a novel hybrid neural end-to-end approach for ranking of search results. our method combines a traditional inverted index and word-matching retrieval with a neural indexing component based on bert architecture @xcite . our neural indexer leverages the siamese network training framework @xcite finetuned on an auxiliary task (unrelated to literature retrieval) to produce universal sentence embeddings. this means that neural indexing can be performed offline for the entire document collection and does not need to be retrained on additional queries. this allows for incorporating the neural component for the entire retrieval process, contrasting with the typical multi-stage neural re-ranking approaches @xcite @xcite . our method is competitive with the top systems presented in trec covid foot_1 . it improves as corpus size increases despite not being trained on additional data which is a useful property in pandemic information retrieval. ## related work the use of neural networks in search has mostly been limited to reranking top results retrieved by a 'traditional' ranking mechanism, such as okapi bm25 @xcite . only a portion of top results is rescored with a neural architecture @xcite . since the most successful neural reranking models depend on joint modelling of both documents and the query, rescoring the entire collection becomes costly. moreover, the effectiveness gains achieved with neural reranking are debated @xcite until recently @xcite . since late 2018, large neural models pre-trained on language modeling-specifically bert @xcite which uses bi-directional transformer architecture-achieve state-of-the-art for several nlp tasks. the architecture is successfully applied to ad-hoc reranking @xcite @xcite . the existing applications of bert in search share the limitation of being restricted to reranking because they rely on its next sentence prediction mechanism for a regression score. however, our approach builds on @xcite , where a bert architecture is trained to produce sentence embeddings. leveraging these embeddings allows for a cost-efficient application of bert to neural indexing. neural indexing is a less explored field. whereas @xcite leverages sparse neural representations for retrieval, @xcite uses sparse and dense representations for learning to rank. these methods rely on networks trained to produce representations directly for ranking documents. for our proposed method, we use universal embeddings foot_2 generated from transformer encoders trained on an auxiliary task of semantic similarity scoring or natural language inference foot_3 . ## dataset documents cord-19 (the covid-19 open research dataset) @xcite is a dataset of research articles on coronaviruses (covid-19, sars and mers). it is compiled from three sources: pubmed central (pmc), the who articles, and biorxiv and medrxiv. evaluations in subsequent stages (referred to as rounds) of trec covid search task are performed on growing snapshots of cord-19 dataset ( topics as part of the trec covid search challenge, nist provides a set of important covidrelated topics. over five rounds, the topic set is augmented. round 1 has 30 topics, with five new topics added per subsequent round. each topic consists of three parts: query, question, and narrative (see figure 1 ). relevance judgements and evaluation trec organises manual judgements per each round of the shared task, using a pooling method over a sample of the submitted runs @xcite . given a topic, a document is judged as: irrelevant (0), partially relevant (1), and relevant (2). as judgements are manually annotated by biomedical experts, only a subset of runs submitted to the track are judged. the evaluation procedure in each of the subsequent rounds discards (topic, document) pairs included judged in previous rounds. we use this procedure (referred to as residual scoring) when comparing against the top-performing runs in the competitive. in additional experiments, we use cumulative scoring, which means evaluating topics for round 2 using human judgments for rounds 1 and 2. topics of round 3 are evaluated using judgments of rounds 1-3, and so on. using cumulative scoring allows us to use a larger proportion of judged documents for the topic sets corresponding to subsequent rounds. metrics four precision focused metrics are used to evaluate the rankings: ndcg @xcite at rank 10 (ndcg@10), precision at rank 10 (p@10), mean average precision (map) and recall-precision (r-prec). bpref takes into account the noisy and incomplete judgements. neural index retrieval (nir) we built a hybrid neural index by appending neural representation vectors to document representations of a traditional inverted index. the neural representations are created using an average over individual representations of sentences (bag-of-sentences) from a bert-based universal sentence encoder for the title, abstract and full-text facets. sentence representations are created by averaging token-level representations produced by the encoder (average pooling strategy outlined in @xcite ). we investigate a selection of models derived from applying the training of the sentence transformer @xcite , a siamese network built to enable cosine comparability between transformer sentence embeddings, and the biomedically-themed bert-based pre-trained models, such as biobert @xcite . to obtain individual sentences, we use a neural sentence segmentation model, scispacy @xcite . for retrieval, we propose a hybrid approach. we score (topic, document) pairs by combining: (1) okapi bm25 scores for all pairs of topic fields and document facets; and, (2) cosine similarities calculated for neural representations of all pairs of topic fields (calculated ad hoc) and document facets stored in the index 5 . the final score adds a log-normalised sum of bm25 scores to the sum of neural scores. formally, the relevance score ψ for i th topic t i and document the z hyper-parameter normalizes the bm25 score such that its range will be the same as the range of the summed cosine similarity score, r cos . this is to ensure both components, neural and bm25, have equal contribution to the final score. we also filter by date. the documents created before december 31st 2019 (the first reported covid-19 case) are removed. ## experimental results we present: (1) a comparison of retrieval effectiveness of our method with different embedding models using cumulative scoring on rounds 1-3 ( choice of the embedding model ablations comparisons with best runs aside from rounds 3 and 4, our models remain competitive with the top run. our model scored higher ndcg@10 for rounds 1 and 2 over the baseline automatic runs. most of the top runs used neural re-rankers which have been specifically trained on related tasks such as med-marco @xcite . where does the model succeed or fail? our model consistently outperforms the bm25 baseline ( we expect that the top ranked documents are scored highly by both components, however, we found that our model placed an irrelevant document at the rank one for topic 3. this document was scored highly by bm25 but much lower in the neural/cosine component. it saturated the scoring function as it repeated many of the keywords in the query, however, the semantic content of the text was irrelevant to the query itself as it discussed "coronavirus crossing continents" rather than "coronavirus cross protection". on the other hand, for topic 1, "coronavirus origins", we found that the neural index overcame semantic mismatches of the bm25 scoring. in the dataset, most documents are related to coron-avirus, the word "origin" contributes more to the final score and bm25 retrieved an irrelevant document at rank three which is a document that discusses origins of a different virus. however, when using the neural scorer, this document is placed at rank 42. from ## conclusions we propose a novel neural ranking approach (nir) for pandemic information retrieval. experimenting with the trec covid search challenge, we show that our method is competitive compared to other automatic systems. we show that a neural scoring is beneficial in alleviating some of the shortcomings of the keyword-based retrieval. empirically, our model shows improvements with time in a pandemic scenario without additional training data. a balanced scoring function combines the strengths of the inverted and neural indices. a neural index explicitly trained for ranking would be a suitable avenue for future research.