FIRST : Faster Improved Listwise Reranking with Single Token Decoding (2024)

Abstract: Large Language Models (LLMs) have significantly advanced the field of information retrieval, particularly for reranking. Listwise LLM rerankers have showcased superior performance and generalizability compared to existing supervised approaches. However, conventional listwise LLM reranking methods lack efficiency as they provide ranking output in the form of a generated ordered sequence of candidate passage identifiers. Further, they are trained with the typical language modeling objective, which treats all ranking errors uniformly–potentially at the cost of misranking highly relevant passages. Addressing these limitations, we introduce FIRST, a novel listwise LLM reranking approach leveraging the output logits of the first generated identifier to directly obtain a ranked ordering of the candidates. Further, we incorporate a learning-to-rank loss during training, prioritizing ranking accuracy for the more relevant passages. Empirical results demonstrate that FIRST accelerates inference by 50% while maintaining a robust ranking performance with gains across the BEIR benchmark. Finally, to illustrate the practical effectiveness of listwise LLM rerankers, we investigate their application in providing relevance feedback for retrievers during inference. Our results show that LLM rerankers can provide a stronger distillation signal compared to cross-encoders, yielding substantial improvements in retriever recall after relevance feedback.

## introduction given their vast linguistic knowledge and strong zero-shot capabilities @xcite , there has been a natural push to incorporate large language models (llms) into the search stack @xcite . one of the core applications of llms in search involves ranking candidate passages for their relevance to a given rank the passages below based on their relevance to the search query. all the passages should be listed using identifiers in descending order of relevance. b: <candidate @xmath0: <candidate @xmath1: <candidate @xmath2 ## methodology in this section, we first discuss the fundamentals of listwise llm reranking ( §3.1). we then present first, our own novel approach to the task ( §3.2). ## experiments we first demonstrate in §4.2 that the proposed ranking loss improves the accuracy of listwise llm reranking. next, in §4.3, we measure the improvement in latency of inference from using first. finally, we show in §4.4 that leveraging listwise llm rerankers for relevance feedback improves the recall of retrievers. and instruction-tuned on chat datasets @xcite . we finetune zephyr β for listwise reranking for three epochs with an effective batch size of 32, a learning rate of 5e-6 using bfloat16 precision, and leverage noisy embeddings @xcite . training takes approximately 7 hours on four 40gb nvidia a100 gpus when used with deepspeed @xcite . we randomly sample 300 queries from ms marco as our development set, and use λ = 10 for scaling the weighted ranknet loss datasets: we use 40k gpt-4 labeled instances from @xcite for fine-tuning llm rerankers, which were created using 5k queries from ms marco @xcite . examples contain a variable number (≤ 20) of candidate passages that need to be reranked. for evaluation, we use the beir benchmark @xcite , which comprises test instances from ms marco and out-of-domain evaluation data from several scientific, biomedical, financial, and wikipedia-based retrieval datasets foot_1 . ## conclusion in this work, we introduce first, a novel strategy for listwise llm reranking. first leverages the output logits of the first generated identifier to obtain a ranking for the candidates, as opposed to the typical approach of generating the entire ranked ordering sequence of candidate passage identifiers. we demonstrated that our single-token decoding approach reranks a considerably larger number of candidates compared to inference with ordered sequence generation in the same time, leading to larger gains when reranking under a latency constraint. first also demonstrates ranking performance benefits from incorporating a learning-torank loss during training, allowing for prioritizing more important ranks. by addressing both the training and inference inefficiencies of existing llm listwise reranking approaches, first represents a significant step forward in the development of advanced re-ranking techniques using llms.