GENRA : Enhancing Zero-shot Retrieval with Rank Aggregation (2024)

Abstract: Large Language Models (LLMs) have been shown to effectively perform zero-shot document retrieval, a process that typically consists of two steps: i) retrieving relevant documents, and ii) re-ranking them based on their relevance to the query. This paper presents GENRA, a new approach to zero-shot document retrieval that incorporates rank aggregation to improve retrieval effectiveness. Given a query, GENRA first utilizes LLMs to generate informative passages that capture the query’s intent. These passages are then employed to guide the retrieval process, selecting similar documents from the corpus. Next, we use LLMs again for a second refinement step. This step can be configured for either direct relevance assessment of each retrieved document or for re-ranking the retrieved documents. Ultimately, both approaches ensure that only the most relevant documents are kept. Upon this filtered set of documents, we perform multi-document retrieval, generating individual rankings for each document. As a final step, GENRA leverages rank aggregation, combining the individual rankings to produce a single refined ranking. Extensive experiments on benchmark datasets demonstrate that GENRA improves existing approaches, highlighting the effectiveness of the proposed methodology in zero-shot retrieval.

## introduction recent studies in zero-shot retrieval have demonstrated remarkable advancements, significantly improving the effectiveness of retrievers with the use of encoders like bert @xcite and contriever @xcite . with the emergence of large language models (llms) @xcite @xcite , research focused on how to leverage llms for information retrieval tasks, such as zero-shot retrieval. early zero-shot ranking with llms relied on methods that score each query-document pair and select the top-scoring pairs @xcite . researchers have attempted to boost these methods by enriching contextual information to help llms understand the relationships between queries and documents. this often involves using llms to generate additional queries, passages, or other relevant content @xcite . these enhancements have significantly improved retrieval performance, especially for unseen (zeroshot) queries. in a typical retrieval setting, as shown in figure 1a , queries and documents are embedded in a shared representation space to enable efficient search. the success of the entire approach depends strongly on the quality of the results of the retrieval step. however, llms can generate potentially non-factual or nonsensical content (e.g. "hallucinations"), and their performance is susceptible to factors like prompt order and input length, which can hurt the performance of the retriever @xcite . to address this problem, some studies @xcite propose employing llms as relevance assessors, providing individual relevance judgments for each query-document pair. these approaches aim to enhance trustworthiness by leveraging the llm's strengths in un-derstanding nuances and identifying potentially irrelevant content. additionally, recent work @xcite suggests incorporating re-ranking models into the retrieval process. such models process a ranked list of documents and directly produce a reordered ranking. however, most existing methods focus solely on retrieval without a separate relevance assessment step, which could be beneficial. to address this gap, our approach utilizes rank aggregation techniques to combine individual rankings generated by separate retrieval and relevance assessment sub-processes. this allows our method to combine the strengths of the two stages, leading to a more refined and accurate final ranking of documents. while combining multiple rankings (rank aggregation) has proven highly effective in various domains, like bio-informatics @xcite and recommendation systems @xcite , its use with llms in zero-shot retrieval has not been explored thus far. our method (figure 1b ), named genra, first utilizes the llm to generate informative passages that capture the query's intent. these passages serve as query variants, guiding the search for similar documents. next, we leverage the llm's capabilities to further refine the initial retrieval. this can be achieved through either direct relevance assessment (generating 'yes' or 'no' judgments) or by employing a re-ranking model to optimize the document order and select the top-ranked ones. this step acts as a verification filter, ensuring the candidate documents can address the given query. using each verified document as a query, we retrieve new documents from the corpus, generating documentspecific rankings that capture diverse facets of the query. by combining these individual rankings through a rank aggregation method, we mitigate potential biases inherent in any single ranking and achieve a more accurate final ranking. thus, the main contributions of the paper are the following: genra can be combined with different llms and different rank aggregation methodologies. ## related work zero-shot retrieval has experienced great advancements in recent years, largely driven by the development and adoption of pre-trained models @xcite @xcite . researchers have explored a diverse range of approaches, including contrastive pre-training @xcite , contextualized models @xcite , and hybrid settings @xcite . with the emergence of llms, research focused on the capabilities of generative models to improve the query representation, through query generation and rewriting @xcite @xcite , or context generation @xcite . in the same line of work @xcite and @xcite , llms are used to create synthetic queries for documents. these artificial query-document pairs serve as training data for retrievers or re-rankers, aiming to enhance retrieval effectiveness. similarly, hyde @xcite employs llms to enrich queries by generating hypothetical documents. beyond retrieval, llms have also been employed for relevance assessment, helping to filter out irrelevant or off-topic documents generated at the retrieval stage. the goal of llm assessors is to provide a relevance label to each query-document pair. such methods @xcite and @xcite have been used to refine the retrieved document sets and enhance relevance. other recent work @xcite proposes the use of more fine-grained relevance judgements instead of binary filters. moreover @xcite suggest to incorporate these fine-grained relevance judgements into the llm prompting process. taking fine-grained relevance judgments one step further, re-ranking models @xcite have been shown to achieve improved retrieval performance with the use of popular generative models like chatgpt and gpt-4 @xcite . in the same line of work @xcite utilize passage relevance labels, obtained either from human judgments or through a gpt-based teacher model. however, the computational cost associated with these models can be significant, requiring substantial resources for both training and inference. furthermore, relying on black-box models poses significant chal-lenges for academic researchers, including substantial cost barriers and restricted access due to their proprietary nature. previous studies have also explored methods for aggregating query or document representations to improve performance in zero-shot document retrieval @xcite . however, the question of how to effectively aggregate per-document rankings has received limited attention. while various rank aggregation techniques exist @xcite , their potential in the context of zero-shot retrieval has not been explored. our study bridges this gap, by incorporating different rank aggregation strategies within the genra pipeline. drawing inspiration from and building upon previous work, genra introduces an effective approach to zero-shot document retrieval, and demonstrates the potential of incorporating rank aggregation techniques for improved retrieval results. ## preliminaries given a query q and a set of documents @xmath0, d 2 , .., d n } the goal of retrieval is to rank d in descending order of relevance to query q. sparse retrieval methods, like bm25 @xcite , rely on keyword-based similarity to estimate relevance. the similarity metric, denoted by s q,d , is typically based on term frequencies and document lengths, ignoring semantic relationships between terms. on the other hand, dense retrieval leverages embedding similarity to assess the relevance between a query q and document d. using an encoder ϵ, queries and documents are converted into vectors, v q and v d , respectively. the inner product of these vectors serves as the similarity metric s q,@xmath1, ϵ(d)⟩. upon inferring the similarity scores for each document, we can readily construct a ranked document list in zero-shot retrieval, the whole process is performed without the aid of labeled training data, posing a significant challenge. ## methodology in figure 2 , we present the proposed genra approach, which consists of three main steps. the initial step aims to retrieve potentially relevant documents from a large collection. the retriever at this stage is assisted by a llm that generates pas-sages, based on the query. in the second step, the relevance of each retrieved document is further assessed and only highly-relevant documents are kept. this is achieved either by asking a llm to filter-out irrelevant documents or by employing a pre-trained model to re-rank the documents and keep the top most-relevant ones. once the relevant documents are selected, similar documents are retrieved, generating a ranking per document. in the third and last step of genra, a rank aggregation method combines the individual rankings into a single more accurate ranking. each of the three steps is detailed in the following subsections. notably, the method relies solely on llm inference, without the requirement for dedicated training. ## conclusion in this work, we have introduced genra, a new approach to information retrieval that leverages llms and rank aggregation to enhance the effectiveness of zero-shot document retrieval. our method involves three main steps. first, genra utilizes llms to create informative passages that serve as refined query representations. then, the retrieved documents undergo a llm-based relevance assessment, keeping only highly relevant ones. finally, multi-document retrieval generates individual rankings for each verified document, which are further refined through rank aggregation, leading to the final ranking. extensive experiments on benchmark datasets demonstrated that genra consistently outperforms existing zero-shot approaches, in some cases achieving considerable improvements. furthermore, the modular nature of genra facilitates further experimentation with different, possibly better, llms and rank aggregation methods. in addition, we will be investigating alternative ways to incorporate relevance judgments into the passage generation instructions. finally, we are interested in ways to improve the computational efficiency of genra, particularly for large-scale retrieval.