Joint Inference of Retrieval and Generation for Passage Re-ranking (2024)

Abstract: Passage retrieval is a crucial component of modern open-domain question answering (QA) systems, providing information for downstream QA components to generate accurate and transparent answers. In this study we focus on passage re-ranking, proposing a simple yet effective method, Joint Passage Re-ranking (JPR), that optimizes the mutual information between query and passage distributions, integrating both cross-encoders and generative models in the re-ranking process. Experimental results demonstrate that JPR outperforms conventional re-rankers and language model scorers in both open-domain QA retrieval settings and diverse retrieval benchmarks under zero-shot settings.

## introduction passage retrieval is a crucial component in opendomain question answering (qa) @xcite , a task that requires answering questions from a wide range of domains and could be applied in systems that fulfill user's information needs @xcite . retrieval offers downstream qa systems grounding information, which not only improves accuracy in a lot of cases but also provides transparency to how systems generate answers, similar to how articles provide references and citations, such that model hallucinations can be checked with ease. furthermore, the set of documents to be retrieved from, or knowledge base, can be quickly updated with new documents and knowledge such that models can adapt to temporal changes, and do not need to be continuously re-trained nor require online training paradigms for continual learning. early retrieval methods are typically based on term-matching, such as bm25 @xcite or tf-idf @xcite . such methods, called sparse retrievers, perform keyword matching efficiently with an inverted index to find relevant contexts. sparse retrievers often achieve reasonable performance while being computationally efficient and does not require training, but are shown to have limited abilities beyond lexical matching. recently, dense retrievers that encode text with continuous embeddings have been heavily studied and utilized in contemporary qa systems, often outperforming their sparse counterparts on high resource evaluation settings @xcite . there are a few drawbacks however, such as higher computational demands during both training and inference, inability to handle large contexts @xcite , and difficulty in generalizing to new domains especially those with limited data @xcite . hybrid methods have been explored to get the best of both worlds, generally utilizing an efficient sparse method to retrieve a larger number of possibly relevant contexts, and then perform passage re-ranking with a more computationally-intensive dense model for refined scoring @xcite . in this work, we focus on passage re-ranking and explore the use of generative models alongside conventional re-rankers. previous work have explored pre-trained language models (lm) as the re-ranking scorer @xcite , however we find that it underperforms conventional re-rankers for both supervised and zero-shot settings. starting from maximizing mutual information (mi) for inference, which measures how much more queries and passages co-occur compared to appearing independently, we show how a small generative model can be effectively used with conventional crossencoding re-rankers for improved performance. experiments on a supervised setting for open-domain qa retrieval and a zero-shot setting across a suite of diverse retrieval benchmarks validate our approach. our contributions can be summarized as follows: consider the two distributions p(x) and p(z) over all queries @xmath0. the conditional distributions p(z|x) and p(x|z) can be used to infer one domain based on the other. the joint distribution p(x, z) characterizes the combined structure of both domains, where p(x, z) = p(x)p(z|x) = p(z)p(x|z). here p ϕ (z|x) defines a passage retrieval model, which we parametrize by ϕ, generally trained with maximum likelihood estimation (mle): l retrieval (ϕ) ≜ -e x,z∼p(x,z) [log p ϕ (z|x)]. during inference, finding the most probable relevant passage can be written as: since we focus on passage re-ranking, we treat p ϕ (z|x) in eq. 1 as re-ranking scores. ## related work passage re-ranking seeks to combine the advantages of sparse retrieval methods, such as efficiency, precise matching, and low-resource generalizability @xcite , with the superior performance of dense methods in the presence of extensive annotated data @xcite . early work by @xcite examined bert-based supervised re-rankers, while later research proposed reader prediction based reranking @xcite and attempted to use lms as re-rankers @xcite , although with limitations. sequence-to-sequence models have also been investigated to directly generate ranking labels @xcite , and further training with explanations can yield improvements under lower-resource scenarios @xcite . more recently, @xcite explored using the proprietary and exceptionally larger chat-gpt models for re-ranking foot_3 . departing from existing ensembling techniques for re-ranking such as fusing bi-encoder embeddings @xcite , our method establishes the combination of discriminative and generative re-rankers through pmi maximization. mi-based objectives, originally introduced in speech recognition to measure input-output dependence @xcite , have been applied to different tasks such as dialogue @xcite , machine translation @xcite , and qa @xcite . mi-based joint inference and learning have been explored in question answering and generation @xcite , language understanding and generation @xcite , and various vision and language tasks @xcite . ## conclusion in this study, we introduce a simple and effective approach to enhance re-ranking for passage retrieval. by jointly utilizing a conventional crossencoding re-ranker and a conditional query generator for inference, we optimize the pointwise mutual information between the query and passage distributions, achieving improvements in open-domain