Effective and Efficient Conversation Retrieval for Dialogue State Tracking with Implicit Text Summaries (2024)

Abstract: Few-shot dialogue state tracking (DST) with Large Language Models (LLM) relies on an effective and efficient conversation retriever to find similar in-context examples for prompt learning. Previous works use raw dialogue context as search keys and queries, and a retriever is fine-tuned with annotated dialogues to achieve superior performance. However, the approach is less suited for scaling to new domains or new annotation languages, where fine-tuning data is unavailable. To address this problem, we handle the task of conversation retrieval based on text summaries of the conversations.A LLM-based conversation summarizer is adopted for query and key generation, which enables effective maximum inner product search. To avoid the extra inference cost brought by LLM-based conversation summarization, we further distill a light-weight conversation encoder which produces query embeddings without decoding summaries for test conversations. We validate our retrieval approach on MultiWOZ datasets with GPT-Neo-2.7B and LLaMA-7B/30B. The experimental results show a significant improvement over relevant baselines in real few-shot DST settings.

## introduction dialogue state tracking (dst) is one of the most crucial components in task-oriented dialogue systems. the goal of dst is to track users' intents, slots and values at every turn of a dialogue based on a predefined schema @xcite . the challenge of training a supervised dst model lies in the cost of dialogue state annotations, which is not scalable to new schemas, domains or annotation languages. to address these challenges, recent works @xcite adopt in-context learning with pre-trained large language models (llm) for few-shot dst. in the few-shot setting, similar dialogue exemplars are retrieved * * work done during an internship at apple. based on the test sample and then these exemplars are added to the llm prompt for target generation. this approach is attractive since no domain-specific fine-tuning is required for the llm but it can still generalize to unseen domains. one challenge in few-shot dst is how to retrieve salient conversation exemplars (e.g., in a set of 3 to 5) from the support set, which serves as demonstrations for the llm. ideally, a retrieved exemplar should carry both the same dialogue history and state change as the test sample. however, in a practical few-shot setting (e.g., with at most 100 annotated support examples), it is likely that no exemplar in the support set satisfies the above requirement. consider a test example with two user turns: in this work, we propose a new solution for conversation retrieval starting with the introduction of a llm-based conversation summarizer. for each exemplar to be indexed and also each test dialog, the summarizer produces a text summarizing what the user wants at this point of the conversation. in section 2, we provide a discussion of this specific summarization choice and how it compares to dialogue state. the summaries are then used as condensed search keys and queries applicable to pre-trained dense retrievers with standard nearest neighbor search. we empirically show that in the few-shot setting, using summaries as retrieval keys and queries is more effective than using raw dialogues. notably the conversation summarization task described above can be easily handled by state-ofthe-art llms via prompt learning, as we will show in an ablation study. however, the deployment of such a retrieval system also introduces extra model parameters and inference cost. unlike search keys, which can be pre-built offline, a search query needs to be auto-regressively decoded for each test dialogue right during inference. to improve the efficiency of this conversation retriever, our second contribution in this work focuses on distilling a light-weight conversation encoder which embeds a raw dialogue directly into a vector space similar to the embedding of its summary. the light-weight conversation encoder enables efficient conversation search over a vector database without explicit query generation. when evaluated on the mul-tiwoz dataset with gpt-neo-2.7b @xcite , llama-7b, and llama-30b @xcite for few-shot dst, we find that the distilled conversation encoder is not only more ef-ficient, but also more effective than a cascaded conversation retriever with explicit query generation. our approach also significantly outperforms relevant baselines, which use annotated dialogues for retriever fine-tuning. ## conversation retrieval with summaries for llm-based dst in the context of task-oriented dialogues with multiple turns of interactions between a user and a system, the objective of dst is to predict the accumulated intents, slots and values at each user turn. in a llm-based approach, the generation of a dialogue state is conditioned on a task-specific prompt. the prompt includes at least the test conversation and a set of k demonstration examples, from which we expect the llm to learn to generalize. considering the size limit of the prompt, k is expected to be small (3-5 examples). each of the retrieved examples is an annotated conversation sharing similar features as the test conversation. we expect to retrieve these exemplars with a dense retriever from a "support set" (e.g., 100 annotations) that can be constructed with minimum effort for domain scaling. there are two major challenges of conversation retrieval for llm-based dst described above. first, a good representation of search keys and queries need to be found. as we analyze in section 1, the similarity of two dialogues is not directly quantifiable by semantic distance, but rather requires more sophisticated structural matching mechanism or a higher-order similarity function. this requirement leads to the second challenge as to how to train an effective conversation retriever that can scale across domains. previous works @xcite mainly fine-tune pre-trained dense retrievers with annotated dialogues obtained from the support set. however, fine-tuning is not realistic in a few-shot setting and for every domain. as shown in figure 1-(a) , our work introduces a query/key generation step in the llm-based dst. the generation is performed with another llm which transforms the raw dialogue context into a text summary whose similarity can be evaluated more easily with pretrained retrievers. specifically, the text summary represents the user's intent up to the current point of the conversation. it grounds the latest user input onto the dialogue history, keeping only information related to the current user intent. note that the summary is a contextual rewriting of the current user intent that is possibly expressed in multiple turns, with applied ellipsis recovery @xcite and co-reference resolution @xcite . examples of the conversation summary are in with the introduction of an explicit query/key generation step, we expect that the conversation retrieval becomes easier and the search index can be built more efficiently. to construct the search index, an offline process can be triggered to generate text summaries for every example in the support set. note that search key generation does not add any inference cost. however, the query generation step comes at an extra cost since the generation needs to happen in an online process. in the next section, we describe how to make the conversation retrieval more efficient by stepping away from explicit query generation. ## conversation encoder distillation note that in the proposed conversation retriever, the llm-based conversation summarizer needs to be invoked for every test sample to generate the search query as shown in figure 1-(a ). to eliminate the extra inference cost, we propose to distill a light-weight conversation encoder which directly embeds a dialogue into a vector space similar to its summary, by maximizing their embedding similarity. the encoder is trained with large-scale dialogue-summary pairs generated by the conversation summarizer in an offline process. after training the model, as shown in figure 1-(b) , we can directly encode each dialogue into a query embedding for maximum inner product search. we call our conversation encoder converse, stand-ing for conversation embeddings for versatile retrieval with implicit summaries. next we explain the structure and training objective of con-verse. ## conclusion the contribution of this work is twofold. first, we proposed an effective way of retrieving conversations in llm-based dst with conversation summaries as search keys and queries. we then improved the efficiency of the retrieval system by distilling a conversation encoder capable of embedding a conversation into a vector space similar to its summary. this eliminates the cost of decoding an actual summary for each test sample during inference. we validated our converse encoder for llm-based dst in a real few-shot setting with 100 conversations in the support set. results showed that converse consistently improved both the efficiency and the performance of few-shot dst when using different llms, outperforming previous llm-based dst baselines that rely on annotated dialogues for retriever fine-tuning.