RRAD istill: Distilling LLM s’ Passage Ranking Ability for Long-Tail Queries Document Re-Ranking on a Search Engine (2024)

Abstract: Large Language Models (LLMs) excel at understanding the semantic relationships between queries and documents, even with lengthy and complex long-tail queries. These queries are challenging for feedback-based rankings due to sparse user engagement and limited feedback, making LLMs’ ranking ability highly valuable. However, the large size and slow inference of LLMs necessitate the development of smaller, more efficient models (sLLMs). Recently, integrating ranking label generation into distillation techniques has become crucial, but existing methods underutilize LLMs’ capabilities and are cumbersome. Our research, RRADistill: Re-Ranking Ability Distillation, propose an efficient label generation pipeline and novel sLLM training methods for both encoder and decoder models. We introduce an encoder-based method using a Term Control Layer to capture term matching signals and a decoder-based model with a ranking layer for enhanced understanding. A/B testing on a Korean-based search platform, validates the effectiveness of our approach in improving re-ranking for long-tail queries.

## introduction large language models (llms), such as chat-gpt (openai, 2022) and gpt-4 @xcite , have shown remarkable potential across diverse search tasks, including query rewriting @xcite , query and document expansions @xcite @xcite . as llms advance in complex tasks, they also show potential for passage ranking, which involves understanding relationships between queries and multiple documents. recent studies @xcite @xcite show that instruction-tuned llms like gpt-4, flan-t5 @xcite and vicuna @xcite effectively handle passage ranking in zeroshot settings. motivated by these studies, we also conducted zero-shot re-ranking with our in-house llms, hyperclova x (hcx) @xcite which comprises large (hcx-l) and small (hcx-s) variants. we put a query and a list of document snippet texts to hcx, using a korean translation of list-wise prompt from rankgpt @xcite . we found that hcx-l effectively returns document identifiers in the desired order, excluding low-relevance ones, as depicted in figure 1. unlike short-head queries, which are popular and short like keywords, long-tail queries are complex and involve longer, specific phrases (a.1.1). these queries benefit more from relevance than feedback, and semantic than syntactic matching due to their rich semantic content but lack of user feedback. thus, llms' ability to rank complex long-tail queries by relevance is highly valuable. however, the slow inference speed challenges the direct use of llms in search engine. to address this, we trained a much smaller language model (slm) to retain hcx-l's ranking ability, following a trend known as llm distillation, similar to rankgpt and twolar @xcite . this involves two stages: 1) generating ranking label using llms, and 2) training the slm ranker. when generating ranking label with llms, previous studies utilize the list-wise permutation generation method, which inputs a query and a set of documents to llm and receives an ordered list of document identifiers. however, these approach require sliding windows, which infer multiple times on partial lists due to the prompt length constraints of llms, causing a burden. moreover, previous studies viewed the missing phenomenon as a problem, where llms fail to include all input documents in the output. however, we observed that in most cases, excluded documents due to missing are significantly irrelevant to the query, making it a valuable signal. consequently, we reframed the missing and highlighted its impact. we developed our own label generation pipeline to address these issues, including two key techniques: 1) pre-rank to filter documents, retaining only those effective to train slm rankers and bypass the sliding window, 2) consider missing as useful signals, and utilize excluded documents as hard negatives, to train slm rankers. our pipeline speeds up labeling and provides compact yet effective training data. in training slm rankers, we explored both bert @xcite and gpt @xcite styles, incorporating our training techniques. for bert ranker, we integrate a term control layer into the training process to utilize specific term matching signals. for gpt ranker, we developed techniques to effectively utilize classification (whether relevant or irrelevant) and reasoning (rationale for relevant or irrelevant) during training, with a light-weight ranking layer. both rankers incorporate additional training layers, but only specific parts of the model architecture (encoder plus a classification head for bert, decoder plus a dense layer for gpt) are utilized during inference, reducing the burden for service applications. in this paper, we provide various experiments on our bert ranker (rra-bert) and gpt ranker (rra-gpt), trained to mimic llms' relevance ranking, aiming to improve long-tail search quality. we tested the effectiveness of our methodology through rigorous online and offline evaluations. ## experiment we tested the effectiveness of our label generation and training method, which leverages bert and gpt structures, alongside following baselines, hcx-l (yoo et al., 2024), bm25 @xcite , monobert @xcite , monot5 @xcite and rankgpt @xcite . monobert and monot5 utilize our labeled dataset for training. rankgpt's training labels are generated using sliding windows, without our pre-ranking process. while exact parameter sizes of backbones are undisclosed, they follow the order: t5 (small) < @xmath0, all of which are below 1 billion parameters. for evaluation, we used our custom naver testset along with korean-translated public testsets for passage re-ranking: ms marco @xcite , miracl @xcite , dl19 @xcite , and dl20 @xcite . more detailed settings about dataset, metrics, baselines, and implementation are outlined in a.2. ## conclusion in this paper, we present an efficient label generation pipeline using llms that utilizes a preranker to select effective documents and leverages llms' missing phenomenon as a useful signal. we also present effective training methods to capture long and complex query-document relevance in both bert and gpt. however, only key parts of the model structure are employed during inference, minimizing the service deployment burden. through extensive experiments, including a/b testing on naver search, we demonstrate the effectiveness of our approach for long-tail queries reranking.