Frugal neural reranking: evaluation on the Covid-19 literature (2020)

Abstract: The Covid-19 pandemic urged the scientific community to join efforts at an unprecedented scale, leading to faster than ever dissemination of data and results, which in turn motivated more research works. This paper presents and discusses information retrieval models aimed at addressing the challenge of searching the large number of publications that stem from these studies. The model presented, based on classical baselines followed by an interaction based neural ranking model, was evaluated and evolved within the TREC Covid challenge setting. Results on this dataset show that, when starting with a strong baseline, our light neural ranking model can achieve results that are comparable to other model architectures that use very large number of parameters.

## introduction the emerging pandemic of covid-19 caused a surge of worldwide scientific studies to be published as a form of open or peer-reviewed articles, as observable in figure 1 . unfortunately, this growing collection of scientific studies tends to be extremely broad, comprehending diverse presentations of the subject, with different perspectives coming from different scientific areas. as a consequence, it is continually more challenging to successfully navigate through the amount of information already published about covid-19, which deprecates precious researching time. so, it is imperative to study and provide ways of successfully navigating (searching) this unstructured type of information (articles), helping the researchers to rapidly find consistent information about their research topic. this work presents our approach to rapidly address the enunciated search problem over the covid-19 literature. we made an adaptation of a system developed for document and snippet retrieval that had been evaluated through the bioasq 8b competition. as a quick reference, bioasq @xcite provides annual competitions on biomedical semantic indexing and questionanswering. so, our hypothesis was to adapt our working bioasq system @xcite to the covid-19 literature, supported by the assumption that the underlying searching data may be from a close information source, and thus, sharing some similarity. our system follows a traditional retrieval pipeline, where an initial search is performed with an efficient retrieval solution to reduce the search space, and then a neural model is leveraged to rerank the articles in this reduced space. however, contrary to what is currently the state-of-the-art in nlp related tasks, we did not follow the trend of using a transform-based architecture for our reranking model. instead, we built upon an interactionbased model, which yields a final model with only 620 trainable parameters, easing the deployment of a system powered by this pipeline to the public use. furthermore, although the gains in performance obtained from transform-based models are undeniable when correctly finetuned, it may not be trivial how these can be adapted and applied on unseen data, which corresponds to this special case. finally, we built a search engine that was made available foot_0 on 31st of march foot_1 , and was, to the best of our knowledge, one of the first tools available for public use. subsequently, other tools have appeared, namely neural covidex foot_2 , cord-19 search foot_3 , covidask foot_4 , and notably a contribution from google, covid-19 research explorer 6 . following this section, we show a more detailed description of the retrieval pipeline and present an evaluation of the system on the ongoing competition trec covid followed by a thorough analysis/discussion. ## retrieval pipeline to keep this paper self-contained, we will now briefly address our bioasq 8b system. as previously mentioned, and as shown in figure 2 , we split our pipeline into two phases. first, a traditional ir approach is adopted to search a given collection of documents and retrieve the top n documents that share some relevance signal with a specific question. then, these top documents are reranked using a neural model that explores in more detail the context where the original relevance matches occurred. the following subsections addresses, in more detail, the document collection used and both retrieval phases. 6 https://covid19-research-explorer.appspot.com/ ## evaluation trec-covid is a challenge launched by ai2 and the national institute of standards and technology (nist) to the information retrieval and text processing communities, to create systems capable of searching the growing literature related to the novel coronavirus. the challenge follows the trec conventions and is divided into five individual rounds, where each team should submit a ranking order of relevant documents, retrieved from the cord-19 collection, for each topic. a topic is a description of the information need, in this case, it is composed of three fields: a query, i.e., most relevant terms; a question, i.e., natural language question; and a narrative, i.e., a more detailed description of the information need with respects to the documents. at least one submission per team is then manually evaluated by domain experts and the union of every evaluation contributes to the trec-covid relevant feedback dataset. given this opportunity, we decided to put our system to the test in this challenge, both as a way to produce empirical measures of performance and also to continually improve the current solution. at the time of writing, this challenge is currently on the fourth round, which means that only the results of the first, second and third rounds are available. an important note is that, since cord-19 is a continually growing collection, each round uses a different snapshot. the task for the first round was to retrieve documents for a total of 30 topics with no feedback data available to use. each subsequent round then re-used the topics from previous rounds, for which feedback data was also made available, plus five new topics. additionally, evaluation was performed in residual manner, which means that the documents evaluated in previous rounds were discarded from the evaluation of subsequent rounds. systems were measured in terms of ndcg@10, p@5, bpred and map, with ndcg@10 being the adopted ranking metric. each team participating in the challenge could submit a maximum of three runs, which could be either a manual run, if some manual action was performed after looking at the topics, a feedback run, if the available feedback data was used, or automatic. ## results and discussion in this section we show and discuss our results for each round separately, since we utilize the feedback from the previous round to continuously improve our solution. for each round, we present the top 3 performing systems and our best submission comparatively. additionally we also compare it with the ranking order before reranking, i.e. the output from phase-i, represented by the name baseline. ## performance comparison in this section, we empirically analyse the performance of our neural ranking solution by comparing it against several transform based-models. following the literature, we selected bert @xcite model and bert variants, since they seem to be the most widely adopted. more precisely in our tests we use the following pretrained models: regarding the methodology, we performed an inference time evaluation over 100 queries, each associated with 250 documents, totalling 25000 query-documents pairs as our samples. furthermore, we performed all the tests in the machine presented on ## conclusion this paper presents and analyses several adaptations of an information retrieval pipeline to address the challenge of covid-19 literature search. our system is based on previous work on the bioasq 8b challenge that we adapt to the novel coronavirus literature. the system follows a two-stage retrieval pipeline, with bm25 as the first stage and a light interaction-based model, with only 620 trainable parameters, as the second stage. we evaluate our system performance on the three available rounds of the ongoing trec covid competition, where in general we obtained competitive results against a wide variety of solution. we also explored the trends of the top submissions to evolve our initial approach. in terms of results, our best ranks were twelfth in one hundred, in round 1, and second in seventy-nine in round 3. as more direct future work we propose to continually explore a better reranking strategy to apply on a strong baseline, especially a baseline based on relevance feedback that seems to be suitable to this residual evaluation setting. another line of work could encompass combining a transformbased model, like bert, in our architecture, since these are well suited to our objective of better eval-uating the passage context. this combination may be achieved by simply replacing the word2vec embeddings by context embeddings or by completely replacing the interaction network.