STYLE : Improving Domain Transferability of Asking Clarification Questions in Large Language Model Powered Conversational Agents (2024)

Abstract: Equipping a conversational search engine with strategies regarding when to ask clarification questions is becoming increasingly important across various domains. Attributing to the context understanding capability of LLMs and their access to domain-specific sources of knowledge, LLM-based clarification strategies feature rapid transfer to various domains in a post-hoc manner.However, they still struggle to deliver promising performance on unseen domains, struggling to achieve effective domain transferability.We take the first step to investigate this issue and existing methods tend to produce one-size-fits-all strategies across diverse domains, limiting their search effectiveness.In response, we introduce a novel method, called STYLE,to achieve effective domain transferability.Our experimental results indicate that STYLE bears strong domain transferability, resulting in an average search performance improvement of 10% on four unseen domains.

## introduction recent research in conversational search systems has highlighted the potential of utilizing large language models (llms) to address ambiguities present in user queries @xcite . a key focus has been on investigating the strategies regarding when to ask clarification questions during conversations @xcite . when developing a search system for a specific domain without prior training, its clarification strategy may be limited, since ambiguities are influenced by the domain-specific background knowledge @xcite . for example, financial terminology may be perceived as ambiguous by the conversational search systems that are not confident with the finance jargon. to address * corresponding author. to investigate this, we conduct an in-depth analysis to examine off-the-shelf llm-based methods (detailed in section 2). our findings reveal that while llm-based methods, armed with domainspecific knowledge sources, somewhat develop strategies that could work on unseen domains, they often produce a one-size-fits-all strategy, such as asking increasingly more questions as the conversation advances. such strategies typically perform less effectively compared to tailored strategies trained on domain-specific data. this highlights the limited ability of llm-based methods to adapt to unseen domains effectively. the challenge lies in the fact that strong domain transferability can not be achieved by training on single-domain data. meanwhile, the mismatched distribution of domainspecific representations poses a significant obstacle to effective domain transfer. affected by the above challenges, existing methods are accustomed to addressing ambiguities solely in a single domain @xcite , potentially leading to models becoming brittle when faced with unseen domains. to this end, we propose a novel method, called style, which features rapid transfer to previously unseen domains via tailored strategies in a post-hoc manner. style comprises the domain-invariant strategy planner (disp) in the conversational search engine and multi-domain training (mdt) paradigm. specifically, the disp is configured to extract domain-invariant information, mitigating the mismatch in the distribution of domain-specific representations and ensuring robustness across domains. we further leverage a retrieval-augmented paradigm to obtain documents that match the user queries in conjunction with their matching scores. the matching score reflects the retrieval quality and confidence of the retrieval module, which is highly correlated to the ambiguous level of the user queries and avoids the introduction of domain-specific semantic representation. moreover, instead of relying on one solitary domain, mdt encourages the domain transferability of disp by training it across multiple diverse domains. this is inspired by the population-based training @xcite , which suggests that the generalization of a collaborative agent to held-out populations can be improved by training larger and more diverse populations @xcite . as such, style enhances domain transferability and adapts its strategies for various domains. we conduct experiments to evaluate style using four domain-specific benchmark datasets in conversational search, including e-commerce, movies, and books. our findings demonstrate that style consistently surpasses existing llm-based baselines in terms of domain transferability, resulting in a significant average performance improvement of ∼10%. further analysis reveals that style tailors its strategies to diversify them in different domains, which laid the foundation for its effectiveness. in summary, we analyze and address the inadequacies of existing methods in determining when to ask clarification questions in scenarios involving previously unseen domains. this paves the way to advance both practical applications and academic research in this area. we present three key contributions: ## preliminary analysis previous analyses indicate that current methods' effectiveness of clarification strategies is compromised when applied in unseen domains @xcite ), yet the underlying cause remains unclear. this section aims to investigate the factors that may hinder the domain transferability of llm-based methods. our analysis involves two factors: search performance, and strategy characteristics. ## the method to achieve effective domain transferability when deciding on when to seek clarification, we propose style. it includes a domain-invariant strategy planner (disp) and multi-domain training paradigm (mdt). we provide the problem formulation in section 3.1. the overall architecture of style is introduced in section 3.2, with detailed explanations of disp and mdt provided in section 3.3 and section 3.4, respectively. ## experiments we conduct experiments to evaluate the domain transferability of style and analyze reasons behind its success. we study three research questions: ## related work determining when to ask clarification questions is critical in conversational search engines @xcite @xcite , which resolves the ambiguous user query @xcite . supervised methods rely on extensive data annotation and training to adapt to specific domain requirements @xcite . building on the success of large language models (llms), developers have created strategies for determining when to ask questions based on llms. these strategies use llms to identify if a user query is ambiguous and generate questions through in-context learning @xcite and chain-of-thought methods @xcite . this approach enables efficient development across various domains in a post-doc manner. however, their performance is still not satisfactory @xcite , especially when applied to unseen domains. for the first time, we reveal that they struggle to establish tailored strategies when transferred to various unseen domains and propose the use of style to address this challenge.