Discrete Prompt Optimization via Constrained Generation for Zero-shot Re-ranker (2023)

Abstract: Re-rankers, which order retrieved documents with respect to the relevance score on the given query, have gained attention for the information retrieval (IR) task. Rather than fine-tuning the pre-trained language model (PLM), the large-scale language model (LLM) is utilized as a zero-shot re-ranker with excellent results. While LLM is highly dependent on the prompts, the impact and the optimization of the prompts for the zero-shot re-ranker are not explored yet. Along with highlighting the impact of optimization on the zero-shot re-ranker, we propose a novel discrete prompt optimization method, Constrained Prompt generation (Co-Prompt), with the metric estimating the optimum for re-ranking. Co-Prompt guides the generated texts from PLM toward optimal prompts based on the metric without parameter update. The experimental results demonstrate that Co-Prompt leads to outstanding re-ranking performance against the baselines. Also, Co-Prompt generates more interpretable prompts for humans against other prompt optimization methods.

## introduction information retrieval (ir) is the task of searching for documents relevant to a given query from a large corpus. as re-ranking the fetched documents from the retriever effectively enhances the performance and the latency, recent studies have suggested several kinds of re-rankers by fine-tuning pre-trained language models (plm) @xcite . furthermore, @xcite show that large-scale language models (llms) such as gpt-3 @xcite can be exploited as a zero-shot reranker with the prompt describing the task. they also highlight the importance of an appropriate prompt to elicit the full performance of llms, rather than updating the parameters. they choose an optimal prompt among the handcrafted candidates by cross-validation. however, such a manual search for the discrete prompts is highly expensive and sub-optimal in transferability. to resolve the issue, several methods are proposed for automatically optimizing the discrete prompt. they focus on text classification or maskfilling task while underestimating the open-ended generation @xcite @xcite . recently, @xcite address the discrete prompt optimization applicable to generation tasks with reinforcement learning by designing the reward function, which measures the generated text belonging to a discrete label. since there are tasks that are still not aligned, requiring a continuous score of output, we aim at a prompt optimization for one of such tasks: re-ranking. in this paper, we propose constrained prompt generation, co-prompt, as left-to-right discrete prompt optimization without additional model training. by defining the metric of prompt optimum for re-ranking, we interpret the searching process of the optimal prompt as constrained generation with two modules: a zero-shot re-ranker as a discriminator and any decoder-only plm as a generator. the discriminator calculates the likelihood (i.e., metric) that the prompt sequence is optimal for guiding an llm to distinguish relevant documents among the large set for a given query. the generator samples the prompt tokens having a high prior from the previous prompt sequences for effectively restricting the prompt candidates for discriminator to evaluate. an overview of co-prompt is shown in figure 1 . we validate our method, co-prompt, against other optimization baselines on two llms, t0 @xcite and opt @xcite , with two benchmark datasets, ms-marco @xcite and natural question @xcite . experimental results show that co-prompt consistently generates well-performing prompts regardless of llms and datasets over the baselines. the qualitative analyses also support the interpretability of the prompts generated by co-prompt, similar to human language patterns. our contributions in this work are threefold: prompt optimization as prompting is considered a key variable when exploiting llms for various nlp tasks, finding the optimal prompt has become important to get the best performance out of the llms @xcite . recently, the prompt optimization work has focused on discrete prompt search @xcite @xcite or soft prompt learning over a continuous space @xcite @xcite . while the existing optimization methods mainly consider text classification or mask-filling task, their applicability to re-ranking is yet underexplored. in this paper, we target at optimizing discrete prompts for zero-shot re-ranker to get higher relevance scores for more relevant pairs via constrained generation. constrained generation constrained generation aims at deriving the text sequences that follow a certain constraint @xcite . utilizing a discriminator for guiding the generation toward the constraint via the bayes' rule is one of the widely used constraint generation methods @xcite @xcite . inspired by the effectiveness of the discriminator-based method, we adopt the zero-shot re-ranker as a discriminator when generating optimal discrete prompt sequences. ## experimental setups we describe the experimental setups for validating the performance of the prompts. our code is publicly available at github.com/zomss/co-prompt. datasets we employ two information retrieval datasets: 1) ms-marco @xcite , collected from the bing search logs, and 2) natural question (nq, @xcite al. (2019) fetched from google search engines. we only use the document data of the dataset for evaluation. more information is shown in appendix a.1. evaluation metrics we evaluate the results by two metrics, acc and ndcg. 1) acc is the percentage of the relevant documents in the total retrieved ones. 2) ndcg, normalized discounted cumulative gain, reflects that the more relevant documents should record higher ranks. retriever & re-ranker we select two widely used sparse and dense retrievers as our retrievers, which are 1) bm25 @xcite and 2) dpr @xcite , respectively. for the zero-shot re-ranker, we use 1) t0 @xcite and 2) opt @xcite . we describe more detailed information in appendix a.3 and a.4. ## result in this section, we show the overall results of our method, co-prompt, with a detailed analysis. overall results as shown in ## conclusion in this paper, we propose co-prompt, left-to-right prompt optimization for zero-shot re-ranker via constrained generation. co-prompt effectively restricts prompt candidates and evaluates the optimum of these prompts without any parameter updates. we experimentally show that our method achieves consistently outperforming performance across all experiments. also, the impact of prompt optimization including baselines on the zero-shot re-ranker highlights its importance. we also present an interesting outcome in that the optimal prompt is interpretable for human. for future work, we plan to expand our method to other open-ended generation tasks using llms.