Generalizing Conversational Dense Retrieval via LLM -Cognition Data Augmentation (2024)

Abstract: Conversational search utilizes muli-turn natural language contexts to retrieve relevant passages. Existing conversational dense retrieval models mostly view a conversation as a fixed sequence of questions and responses, overlooking the severe data sparsity problem – that is, users can perform a conversation in various ways, and these alternate conversations are unrecorded. Consequently, they often struggle to generalize to diverse conversations in real-world scenarios. In this work, we propose a framework for generalizing Conversational dense retrieval via LLM-cognition data Augmentation (ConvAug). We first generate multi-level augmented conversations to capture the diverse nature of conversational contexts. Inspired by human cognition, we devise a cognition-aware prompting process to mitigate the generation of false positives, false negatives, and hallucinations. Moreover, we develop a difficulty-adaptive sample filter that selects challenging samples for complex conversations, thereby giving the model a larger learning space. A contrastive learning objective is then employed to train a better conversational context encoder. Extensive experiments conducted on four public datasets, under both normal and zero-shot settings, demonstrate the effectiveness, generalizability, and applicability of ConvAug. The code is released at https://github.com/haon-chen/ConvAug.

## introduction conversational search is anticipated to become the leading form of ad-hoc search engines in the future @xcite . this approach, utilizing multi-turn natural language interactions, offers a user-friendly experience, particularly for complex information-seeking tasks. there are two typical approaches for conversational search. one way is conversational query rewriting (cqr) @xcite . cqr models convert a conversational query into a de-contextualized search query suitable for ad-hoc retrieval. however, cqr models either perform poorly because they cannot be optimized by downstream retrieval task @xcite , or have unacceptable search latency when using large language models (llms) during inference @xcite . another approach is to perform conversational dense retrieval (cdr) in an end-to-end manner. it typically uses the entire conversational context to train the context encoder within cdr models for passage retrieval. this approach has been demonstrated to be more effective than cqr models on the downstream retrieval task of conversational search @xcite . existing cdr approaches typically utilize conversations as fixed multi-turn natural language texts to train the context encoder. however, in real-world scenarios, users can express conversations in various ways. the conversational search data often lack the diversity to support training for such variability due to the severe data sparsity issue. in other words, numerous alternative conversations with the same intent (or with similar expressions but different intents) as a specific data sample are unrecorded. as a result, cdr models trained on these limited and fixed data often struggle to adapt to diverse real-world conversations. some works have tried to compensate for the deficiency of multiturn texts. however, these efforts often rely on basic rule-based strategies @xcite or human annotations to augment conversations @xcite . furthermore, comprehending turn dependencies in multi-turn conversations poses a significant challenge for simple language models. to tackle these problems, we propose an llmbased data augmentation framework to mimic how users perform diverse conversations. specifically, we design multi-level augmentation strategies to generate positive (similar intents but different expressions, denoted as + + +) and hard negative conversations (similar expressions but different intents, denoted as ---): (1) token level. to mitigate the model's overreliance on specific tokens, we randomly mask some tokens of conversations (+ + +). besides, we identify and replace the entities (---) to help the model focus on key information. (2) turn level. to prevent the model from depending on specific turns or the order of turns within conversations, we mask (+ + +) and reorder (+ + +) turns to generate diverse conversations. we also generate a noisy turn (+ + +) to enhance the model's denoising ability. to avoid generating false positives, we identify the turn dependency structure to guide the turn-level augmentations. (3) conversation level. we paraphrase the conversation (+ + +) to introduce linguistic variations. we also shift the intent of conversations to help the model detect subtle intent changes (---). however, llms may generate false positives or negatives and be prone to generate texts with hallucinations @xcite . to produce high-quality conversations, we propose a three-step prompting process inspired by human cognition. initially, we prompt an llm to get a comprehensive understanding of the conversation (e.g., its intent and theme) in the first step @xcite . subsequently, the llm associates existing elements, such as expressions, intents, and entities, with new yet related ones @xcite . finally, the llm can conclude final outputs based on former outputs. these outputs are less prone to be false positives, false negatives, or hallucinations, as the llm has a deeper understanding of the original conversation subsequently, we employ contrastive learning to bring together augmented positive conversations and push them away from negative ones. through this, we aim to train a more robust and generalized conversational context encoder, capable of accurately interpreting users' search intents of diverse conversations. to enhance the contrastive learning process, we go beyond basic random sampling methods @xcite , and introduce a difficulty-adaptive sample filter to select more challenging augmented samples for more difficult conversations. we believe that complex conversations offer a larger learning space for the model. more challenging data can thus provide the model with richer information, enabling it to understand these complex conversations better. extensive experiments on four public datasets demonstrate that convaug can consistently im-prove the performance of various conversational dense retrievers across various complexity levels of conversational turns. the contributions of our work are as follows: ## related work conversational search. cqr models usually utilize the context to rewrite the conversation into a standalone query @xcite @xcite . some researchers attempt to connect the downstream retrieval task to the rewriting task @xcite @xcite . on the other hand, cdr models try to utilize the whole conversation to train a conversational context encoder. some works use a few-shot manner to train the cdr model @xcite @xcite . some design delicate denoising approaches for better cdr models @xcite @xcite . however, none of these models focus on developing a context encoder that can comprehend diverse conversations smoothly. ## methodology: convaug in this section, we present our two-stage framework convaug, as illustrated in figure 1 . in the first stage, we leverage an llm to perform our data augmentation strategies tailored for conversational search. a three-step cognition-aware prompting process is developed to guide the llm to generate multi-level augmented conversations. the second stage is to utilize the augmented data to optimize the conversational context encoder. we propose to select more challenging samples for more complex conversations to facilitate model learning. ## conclusion in this work, we present convaug to augment conversational search data with llms. we design a three-step cognition-aware prompting process to generate multi-level augmented conversations. we also develop a difficulty-adaptive sample filter to assign challenging samples to difficult conversations for larger learning space. a contrastive learning objective is employed to train a generalized conversational context encoder. extensive experiments on four public datasets at both normal and zero-shot settings validate the effectiveness, generalization ability, and applicability of convaug.