Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search (2023)

Abstract: Precisely understanding users’ contextual search intent has been an important challenge for conversational search. As conversational search sessions are much more diverse and long-tailed, existing methods trained on limited data still show unsatisfactory effectiveness and robustness to handle real conversational search scenarios. Recently, large language models (LLMs) have demonstrated amazing capabilities for text generation and conversation understanding. In this work, we present a simple yet effective prompting framework, called LLM4CS, to leverage LLMs as a text-based search intent interpreter to help conversational search. Under this framework, we explore three prompting methods to generate multiple query rewrites and hypothetical responses, and propose to aggregate them into an integrated representation that can robustly represent the user’s real contextual search intent. Extensive automatic evaluations and human evaluations on three widely used conversational search benchmarks, including CAsT-19, CAsT-20, and CAsT-21, demonstrate the remarkable performance of our simple LLM4CS framework compared with existing methods and even using human rewrites. Our findings provide important evidence to better understand and leverage LLMs for conversational search.

## introduction conversational search has been expected to be the next generation of search paradigms @xcite . it supports search via conversation to provide users with more accurate and intuitive search results and a much more user-friendly search experience. unlike using traditional search engines which mainly process keyword queries, users could imagine the conversational search system as a knowledgeable human expert and directly start a multi-turn conversation with it in natural languages to solve their questions. however, one of the main challenges for this beautiful vision is that the users' queries may contain some linguistic problems (e.g., omissions and coreference) and it becomes much harder to capture their real search intent under the multi-turn conversation context @xcite . to achieve conversational search, an intuitive method known as conversational query rewriting (cqr) involves using a rewriting model to transform the current query into a de-contextualized form. subsequently, any ad-hoc search models can be seamlessly applied for retrieval purposes. given that existing ad-hoc search models can be reused directly, cqr demonstrates substantial practical value for industries in quickly initializing their conversational search engines. another type of method, conversational dense retrieval (cdr), tries to learn a conversational dense retriever to encode the user's real search intent and passages into latent representations and performs dense retrieval. in contrast to the two-step cqr method, where the rewriter is difficult to be directly optimized towards search @xcite , the conversational dense retriever can naturally learn from session-passage relevance signals. however, as conversational search sessions are much more diverse and long-tailed @xcite @xcite , existing cqr and cdr methods trained on limited data still show unsatisfactory performance, especially on more complex conversational search sessions. many studies @xcite @xcite have demonstrated the performance advantages of using de-contextualized human rewrites on sessions which have complex response dependency. also, as reported in the public trec cast 2021 benchmark @xcite , existing methods still suffer from significant degradation in their ef-fectiveness as conversations become longer. recently, large language models (llms) have shown amazing capabilities for text generation and conversation understanding @xcite @xcite . in the field of information retrieval (ir), llms have also been successfully utilized to enhance relevance modeling via various techniques such as query generation @xcite , query expansion @xcite , document prediction @xcite , etc. inspired by the strong performance of llms in conversation and ir, we try to investigate how llms can be adapted to precisely grasp users' contextual search intent for conversational search. in this work, we present a simple yet effective prompting framework, called llm4cs, to leverage llm as a search intent interpreter to facilitate conversational search. specifically, we first prompt llm to generate both short query rewrites and longer hypothetical responses in multiple perspectives and then aggregate these generated contents into an integrated representation that robustly represents the user's real search intent. under our framework, we propose three specific prompting methods and aggregation methods, and conduct extensive evaluations on three widely used conversational search benchmarks, including cast-19 @xcite , cast-20 @xcite , and cast-21 @xcite ), to comprehensively investigate the effectiveness of llms for conversational search. in general, our framework has two main advantages. first, by leveraging the powerful contextual understanding and generation abilities of large language models, we show that additionally generating hypothetical responses to explicitly supplement more plausible search intents underlying the short rewrite can significantly improve the search performance. second, we show that properly aggregating multiple rewrites and hypothetical responses can effectively filter out incorrect search intents and enhance the reasonable ones, leading to better search performance and robustness. overall, our main contributions are: query encoder context & question search intent interpreter (llm) rew rtr rar passage vectors (offline encoded) rewrite response rewrite llm rewrite response × 𝑁 aggregation maxprob sc mean . ## llm4cs: prompting large language models for conversational search in this section, we introduce our llm4cs framework, which leverages llm as a text-based search intent interpreter to facilitate conversational search. figure 1 shows an overview of llm4cs. in the following, we first describe our task formulation of conversational search, and then we elaborate on the specific prompting methods and aggregation methods integrated into the framework. finally, we introduce the retrieval process. ## human evaluation the retrieval performance is influenced by the adhoc retriever used, which implies that automatic search evaluation metrics may not fully reflect the model's capability to understand contextual search intent. sometimes, two different rewrites can yield significantly different retrieval scores, even though they both accurately represent the user's real search intent. to better investigate the contextual search intent understanding ability of llm, we perform a fine-grained human evaluation on the rewrites generated by our llm4cs (rew + maxprob). specifically, we manually compare each model's rewrite with the corresponding human rewrite and label it with one of the following four categories: (1) good-h: the model's rewrite is nearly the same as the human rewrite. (2) good-a: the expression of the model's rewrite is different from the human rewrite but it also successfully conveys the user's real search intent. (3) bad-c: the rewrite has coreference errors. (4) bad-o: the rewrite omits important contextual information or has other types of errors. furthermore, we apply the same principle to label the rewrites of t5qr for comparison purposes. a few examples of such categorization are presented in appendix b.2. the results of the human evaluation are shown in figure 3 , where we observe that: ## conclusion in this paper, we present a simple yet effective prompting framework (i.e., llm4cs) that leverages llms for conversational search. our framework generates multiple query rewrites and hypothetical responses using tailored prompting methods and aggregates them to robustly represent the user's contextual search intent. through extensive automatic and human evaluations on three cast datasets, we demonstrate its remarkable performance for conversational search. our study highlights the vast potential of llms in conversational search and takes an important initial step in advancing this promising direction. future research will focus on refining and extending the llm4cs framework to explore better ways of generation to facilitate search, improving aggregation techniques, optimizing the llm-retriever interaction, and incorporating reranking strategies.