Question answering ( QA ) systems for large document collections typically use pipelines that ( i ) retrieve possibly relevant documents , ( ii ) re-rank them , ( iii ) rank paragraphs or other snippets of the top-ranked documents , and ( iv ) select spans of the top-ranked snippets as exact answers . Pipelines are conceptually simple , but errors propagate from one component to the next , without later components being able to revise earlier decisions . We present an architecture for joint document and snippet ranking , the two middle stages , which leverages the intuition that relevant documents have good snippets and good snippets come from relevant documents . The architecture is general and can be used with any neural text relevance ranker . We experiment with two main instantiations of the architecture , based on POSIT-DRMM ( PDRMM ) and a BERT-based ranker . Experiments on biomedical data from BIOASQ show that our joint models vastly outperform the pipelines in snippet retrieval , the main goal for QA , with fewer trainable parameters , also remaining competitive in document retrieval . Furthermore , our joint PDRMM-based model is competitive with BERT-based models , despite using orders of magnitude fewer parameters . These claims are also supported by human evaluation on two test batches of BIOASQ . To test our key findings on another dataset , we modified the Natural Questions dataset so that it can also be used for document and snippet retrieval . Our joint PDRMM-based model again outperforms the corresponding pipeline in snippet retrieval on the modified Natural Questions dataset , even though it performs worse than the pipeline in document retrieval . We make our code and the modified Natural Questions dataset publicly available . Question answering ( QA ) systems that search large document collections ( Voorhees , 2001 ; Tsatsaro-nis et al . , 2015 ; Chen et al . , 2017 ) typically use pipelines operating at gradually finer text granularities . A fully-fledged pipeline includes components that ( i ) retrieve possibly relevant documents typically using conventional information retrieval ( IR ) ; ( ii ) re-rank the retrieved documents employing a computationally more expensive document ranker ; ( iii ) rank the passages , sentences , or other ' snippets ' of the top-ranked documents ; and ( iv ) select spans of the top-ranked snippets as ' exact ' answers . Recently , stages ( ii)-(iv ) are often pipelined neural models , trained individually ( Hui et al . , 2017 ; Pang et al . , 2017 ; Lee et al . , 2018 ; McDonald et al . , 2018 ; Pandey et al . , 2019 ; Mackenzie et al . , 2020 ; Sekulić et al . , 2020 ) . Although pipelines are conceptually simple , errors propagate from one component to the next ( Hosein et al . , 2019 ) , without later components being able to revise earlier decisions . For example , once a document has been assigned a low relevance score , finding a particularly relevant snippet can not change the document 's score . We propose an architecture for joint document and snippet ranking , i.e. , stages ( ii ) and ( iii ) , which leverages the intuition that relevant documents have good snippets and good snippets come from relevant documents . We note that modern web search engines display the most relevant snippets of the top-ranked documents to help users quickly identify truly relevant documents and answers ( Sultan et al . , 2016 ; Xu et al . , 2019 ; Yang et al . , 2019a ) . The top-ranked snippets can also be used as a starting point for multi-document query-focused summarization , as in the BIOASQ challenge ( Tsatsaronis et al . , 2015 ) . Hence , methods that identify good snippets are useful in several other applications , apart from QA . We also note that many neural models for stage ( iv ) have been proposed , often called QA or Machine Reading Comprehension ( MRC ) models ( Kadlec et al . , 2016 ; Cui et al . , 2017 ; Zhang et al . , 2020 ) , but they typically search for answers only in a particular , usually paragraph-sized snippet , which is given per question . For QA systems that search large document collections , stages ( ii ) and ( iii ) are also important , if not more important , but have been studied much less in recent years , and not in a single joint neural model . The proposed joint architecture is general and can be used in conjunction with any neural text relevance ranker ( Mitra and Craswell , 2018 ) . Given a query and N possibly relevant documents from stage ( i ) , the neural text relevance ranker scores all the snippets of the N documents . Additional neural layers re-compute the score ( ranking ) of each document from the scores of its snippets . Other layers then revise the scores of the snippets taking into account the new scores of the documents . The entire model is trained to jointly predict document and snippet relevance scores . We experiment with two main instantiations of the proposed architecture , using POSIT-DRMM ( McDonald et al . , 2018 ) , hereafter called PDRMM , as the neural text ranker , or a BERT-based ranker ( Devlin et al . , 2019 ) . We show how both PDRMM and BERT can be used to score documents and snippets in pipelines , then how our architecture can turn them into models that jointly score documents and snippets . Experimental results on biomedical data from BIOASQ ( Tsatsaronis et al . , 2015 ) show the joint models vastly outperform the corresponding pipelines in snippet extraction , with fewer trainable parameters . Although our joint architecture is engineered to favor retrieving good snippets ( as a near-final stage of QA ) , results show that the joint models are also competitive in document retrieval . We also show that our joint version of PDRMM , which has the fewest parameters of all models and does not use BERT , is competitive to BERT-based models , while also outperforming the best system of BIOASQ 6 ( Brokos et al . , 2018 ) in both document and snippet retrieval . These claims are also supported by human evaluation on two test batches of BIOASQ 7 ( 2019 ) . To test our key findings on another dataset , we modified Natural Questions ( Kwiatkowski et al . , 2019 ) , which only includes questions and answer spans from a single document , so that it can be used for document and snippet retrieval . Again , our joint PDRMMbased model largely outperforms the corresponding pipeline in snippet retrieval on the modified Natural Questions , though it does not perform better than the pipeline in document retrieval , since the joint model is geared towards snippet retrieval , i.e. , even though it is forced to extract snippets from fewer relevant documents . Finally , we show that all the neural pipelines and joint models we considered improve the BM25 ranking of traditional IR on both datasets . We make our code and the modified Natural Questions publicly available.1 2 Methods Our contributions can be summarized as follows : ( 1 ) We proposed an architecture to jointly rank documents and snippets with respect to a question , two particularly important stages in QA for large document collections ; our architecture can be used with any neural text relevance model . ( 2 ) We instantiated the proposed architecture using a recent neural relevance model ( PDRMM ) and a BERTbased ranker . ( 3 ) Using biomedical data ( from BIOASQ ) , we showed that the two resulting joint models ( PDRMM-based and BERT-based ) vastly outperform the corresponding pipelines in snippet re-trieval , the main goal in QA for document collections , using fewer parameters , and also remaining competitive in document retrieval . ( 4 ) We showed that the joint model ( PDRMM-based ) that does not use BERT is competitive with BERT-based models , outperforming the best BIOASQ 6 system ; our joint models ( PDRMM-and BERT-based ) also outperformed all BIOASQ 7 competitors . ( 5 ) We provide a modified version of the Natural Questions dataset , suitable for document and snippet retrieval . ( 6 ) We showed that our joint PDRMM-based model also largely outperforms the corresponding pipeline on open-domain data ( Natural Questions ) in snippet retrieval , even though it performs worse than the pipeline in document retrieval . ( 7 ) We showed that all the neural pipelines and joint models we considered improve the traditional BM25 ranking on both datasets . ( 8) We make our code publicly available . We hope to extend our models and datasets for stage ( iv ) , i.e. , to also identify exact answer spans within snippets ( paragraphs ) , similar to the answer spans of SQUAD ( Rajpurkar et al . , 2016 ( Rajpurkar et al . , , 2018 ) ) . This would lead to a multi-granular retrieval task , where systems would have to retrieve relevant documents , relevant snippets , and exact answer spans from the relevant snippets . BIOASQ already includes this multi-granular task , but exact answers are provided only for factoid questions and they are freely written by humans , as in MS-MARCO , with similar limitations . Hence , appropriately modified versions of the BIOASQ datasets are needed . Table 4 shows that further performance gains ( 6.80 to 7.85 document MAP , 15.42 to 17.34 snippet MAP ) are possible by tuning the weights of the two losses . The best scores are obtained when using both the extra sentence and document features . However , the model performs reasonably well even when one of the two types of extra features is removed , with the exception of λ snip = 10 . The standard deviations of the MAP scores over the folds of the cross-validation indicate that the performance of the model is reasonably stable .

Challenge: Pipeline systems for question answering systems for large document collections suffer from error propagation from one stage to the following.
Approach: They propose to jointly model document and snippet rankings in a generic way to make work with any neural text relevance rankers.
Outcome: The proposed systems outperform existing BERT-based models by using much fewer parameters on biomedical data and also on the Natural Questions dataset.