A Neural Model for Joint Document and Snippet Ranking in Question Answering for Large Document Collections (2021)

Abstract: Question answering (QA) systems for large document collections typically use pipelines that (i) retrieve possibly relevant documents, (ii) re-rank them, (iii) rank paragraphs or other snippets of the top-ranked documents, and (iv) select spans of the top-ranked snippets as exact answers. Pipelines are conceptually simple, but errors propagate from one component to the next, without later components being able to revise earlier decisions. We present an architecture for joint document and snippet ranking, the two middle stages, which leverages the intuition that relevant documents have good snippets and good snippets come from relevant documents. The architecture is general and can be used with any neural text relevance ranker. We experiment with two main instantiations of the architecture, based on POSIT-DRMM (PDRMM) and a BERT-based ranker. Experiments on biomedical data from BIOASQ show that our joint models vastly outperform the pipelines in snippet retrieval, the main goal for QA, with fewer trainable parameters, also remaining competitive in document retrieval. Furthermore, our joint PDRMM-based model is competitive with BERT-based models, despite using orders of magnitude fewer parameters. These claims are also supported by human evaluation on two test batches of BIOASQ. To test our key findings on another dataset, we modified the Natural Questions dataset so that it can also be used for document and snippet retrieval. Our joint PDRMM-based model again outperforms the corresponding pipeline in snippet retrieval on the modified Natural Questions dataset, even though it performs worse than the pipeline in document retrieval. We make our code and the modified Natural Questions dataset publicly available.

## introduction question answering (qa) systems that search large document collections @xcite @xcite typically use pipelines operating at gradually finer text granularities. a fully-fledged pipeline includes components that (i) retrieve possibly relevant documents typically using conventional information retrieval (ir); (ii) re-rank the retrieved documents employing a computationally more expensive document ranker; (iii) rank the passages, sentences, or other 'snippets' of the top-ranked documents; and (iv) select spans of the top-ranked snippets as 'exact' answers. recently, stages (ii)-(iv) are often pipelined neural models, trained individually @xcite @xcite @xcite @xcite . although pipelines are conceptually simple, errors propagate from one component to the next @xcite , without later components being able to revise earlier decisions. for example, once a document has been assigned a low relevance score, finding a particularly relevant snippet cannot change the document's score. we propose an architecture for joint document and snippet ranking, i.e., stages (ii) and (iii), which leverages the intuition that relevant documents have good snippets and good snippets come from relevant documents. we note that modern web search engines display the most relevant snippets of the top-ranked documents to help users quickly identify truly relevant documents and answers @xcite @xcite . the top-ranked snippets can also be used as a starting point for multi-document query-focused summarization, as in the bioasq challenge @xcite . hence, methods that identify good snippets are useful in several other applications, apart from qa. we also note that many neural models for stage (iv) have been proposed, often called qa or machine reading comprehension (mrc) models @xcite @xcite ), but they typically search for answers only in a particular, usually paragraph-sized snippet, which is given per question. for qa systems that search large document collections, stages (ii) and (iii) are also important, if not more important, but have been studied much less in recent years, and not in a single joint neural model. the proposed joint architecture is general and can be used in conjunction with any neural text relevance ranker @xcite . given a query and n possibly relevant documents from stage (i), the neural text relevance ranker scores all the snippets of the n documents. additional neural layers re-compute the score (ranking) of each document from the scores of its snippets. other layers then revise the scores of the snippets taking into account the new scores of the documents. the entire model is trained to jointly predict document and snippet relevance scores. we experiment with two main instantiations of the proposed architecture, using posit-drmm @xcite , hereafter called pdrmm, as the neural text ranker, or a bert-based ranker @xcite . we show how both pdrmm and bert can be used to score documents and snippets in pipelines, then how our architecture can turn them into models that jointly score documents and snippets. experimental results on biomedical data from bioasq @xcite show the joint models vastly outperform the corresponding pipelines in snippet extraction, with fewer trainable parameters. although our joint architecture is engineered to favor retrieving good snippets (as a near-final stage of qa), results show that the joint models are also competitive in document retrieval. we also show that our joint version of pdrmm, which has the fewest parameters of all models and does not use bert, is competitive to bert-based models, while also outperforming the best system of bioasq 6 @xcite in both document and snippet retrieval. these claims are also supported by human evaluation on two test batches of bioasq @xcite . to test our key findings on another dataset, we modified natural questions @xcite , which only includes questions and answer spans from a single document, so that it can be used for document and snippet retrieval. again, our joint pdrmmbased model largely outperforms the corresponding pipeline in snippet retrieval on the modified natural questions, though it does not perform better than the pipeline in document retrieval, since the joint model is geared towards snippet retrieval, i.e., even though it is forced to extract snippets from fewer relevant documents. finally, we show that all the neural pipelines and joint models we considered improve the bm25 ranking of traditional ir on both datasets. we make our code and the modified natural questions publicly available. foot_0 2 methods ## related work neural document ranking @xcite @xcite @xcite only recently managed to improve the rankings of conventional ir; see @xcite for caveats. document or passage ranking models based on bert have also been proposed, with promising results, but most use only simplistic task-specific layers on top of bert @xcite , similar to our use of bert for document scoring (fig. 3 ). an exception is the work of @xcite , who explored combining elmo @xcite and bert @xcite models that directly retrieve documents by indexing neural document representations, rather than re-ranking documents retrieved by conventional ir, have also been proposed @xcite @xcite , but none addresses both document and snippet retrieval. @xcite use bert to encode, index, and directly retrieve snippets, but do not consider documents; indexing snippets is also computationally costly. @xcite propose a joint model for direct snippet retrieval (and indexing) and answer span selection, again without retrieving documents. no previous work combined document and snippet retrieval in a joint neural model. this may be due to existing datasets, which do not provide both gold documents and gold snippets, with the exception of bioasq, which is however small by today's standards (2.7k training questions, section 3.1). for example, @xcite used much larger clickthrough datasets from a chinese search engine, as well as datasets from the 2007 and 2008 trec million query tracks @xcite , but these datasets do not contain gold snippets. squad @xcite and squad v.2 @xcite provide 100k and 150k questions, respectively, but for each question they require extracting an exact answer span from a single given wikipedia paragraph; no snippet retrieval is performed, because the relevant (paragraph-sized) snippet is given. @xcite provide modified versions of squad and natural questions, suitable for direct snippet retrieval, but do not consider document retrieval. searchqa @xcite provides 140k questions, along with 50 snippets per question. the web pages the snippets were extracted from, however, are not included in the dataset, only their urls, and crawling them may produce different document collections, since the contents of web pages often change, pages are removed etc. ms-marco @xcite was constructed using 1m queries extracted from bing's logs. for each question, the dataset includes the snippets returned by the search engine for the top-10 ranked web pages. however the gold answers to the questions are not spans of particular retrieved snippets, but were freely written by humans after reading the returned snippets. hence, gold relevant snippets (or sentences) cannot be identified, making this dataset unsuitable for our purposes. ## conclusions and future work our contributions can be summarized as follows: we hope to extend our models and datasets for stage (iv), i.e., to also identify exact answer spans within snippets (paragraphs), similar to the answer spans of squad @xcite . this would lead to a multi-granular retrieval task, where systems would have to retrieve relevant documents, relevant snippets, and exact answer spans from the relevant snippets. bioasq already includes this multi-granular task, but exact answers are provided only for factoid questions and they are freely written by humans, as in ms-marco, with similar limitations. hence, appropriately modified versions of the bioasq datasets are needed.