CLAMBER : A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models (2024)

Abstract: Large language models (LLMs) are increasingly used to meet user information needs, but their effectiveness in dealing with user queries that contain various types of ambiguity remains unknown, ultimately risking user trust and satisfaction. To this end, we introduce CLAMBER, a benchmark for evaluating LLMs using a well-organized taxonomy. Building upon the taxonomy, we construct 12K high-quality data to assess the strengths, weaknesses, and potential risks of various off-the-shelf LLMs.Our findings indicate the limited practical utility of current LLMs in identifying and clarifying ambiguous user queries, even enhanced by chain-of-thought (CoT) and few-shot prompting. These techniques may result in overconfidence in LLMs and yield only marginal enhancements in identifying ambiguity. Furthermore, current LLMs fall short in generating high-quality clarifying questions due to a lack of conflict resolution and inaccurate utilization of inherent knowledge.In this paper, CLAMBER presents a guidance and promotes further research on proactive and trustworthy LLMs.

## introduction given well-defined user queries, large language models (llms) have demonstrated remarkable proficiency in facilitating the information search process @xcite @xcite . they provide more precise search results with the help of the inherent knowledge stored within llms. nonetheless, as evidenced by previous studies (kuhn et al., † corresponding author. # both authors contributed equally to this study. @xcite , the practical utility of llms is hindered by unclear and ambiguous user queries in real-world scenarios. for instance, in a query like "what are the strategies for saving?", the term "saving" can have multiple interpretations, such as "saving money" or "saving from sins", depending on the user's actual need. this necessitates llms proactively identifying (i.e., determine if the query is ambiguous or not) and clarifying the ambiguities rather than providing potentially incorrect answers that may not align with the user's true needs, ultimately risking user trust and satisfaction @xcite . driven by this concern, recent works have explored llms' capacity to address ambiguous queries @xcite . however, these investigations have been somewhat fragmented, lacking a comprehensive taxonomy, leading to incomplete and inconsistent handling of ambiguity distributions @xcite ). as a notable example, they are often limited to contextual ambiguity, where the given context is insufficient for producing a definitive answer. in the era of llms, there should be more emphasis on the llm-oriented ambiguity that may occur when inherent knowledge stored within llms have conflict understanding about the query. consequently, it still remains unclear which ambiguities llms can effectively identify and clarify, along with the challenges that llms persistently encounter in this regard. to this end, we introduce clamber (clarifying ambiguous query), a novel benchmark for comprehensively evaluating llms in identifying and clarifying various ambiguities using a well-organized taxonomy. drawing inspiration from the input-process-output framework for evaluating collaborative systems @xcite , we establish a taxonomy that consolidates both input understanding and task completion perspectives into three primary dimensions, as illustrated in with clamber, we comprehensively evaluate strengths, weaknesses, and potential risks of various llms. our findings indicate that chat-gpt (openai, 2022) outperforms other smallscale llms, especially excelling in identifying and clarifying ambiguities in multifaceted queries @xcite , such as "what is the largest manufacturer in china?", which does not specify the type of "manufacturer". however, they still encounter numerous challenges: 1) current llms, despite leveraging chain-of-thought (cot) and few-shot prompting, face challenges in identifying ambiguities. our results suggest that cot and few-shot prompting may lead to the overconfidence issue in small-scale llms, impacting ambiguity identification negatively. even with a large number of shots and cot support, llms only achieve a marginal improvement. moreover, current llms struggle to leverage contextual cues to disambiguate pronouns, highlighting the inadequacy in deducing underlying ambiguities. 2) current llms fail to ask high-quality clarifying questions, due to the inability of knowing their knowledge gap. despite llms recognize a query containing ambiguities, their lack of conflict resolution and inaccurate use of inherent knowledge results in uncertainty about which ambiguity to clarify. this prompts the need of developing effective methods for llms to resolve conflicts and accurately utilize their inherent knowledge. in this paper, clamber stands as a valuable resource to provide guidance and insight into evaluating llms and addressing ambiguous information needs for future improvements. in conclusion, our contributions are threefold: ## related works our research is closely tied to the taxonomy and resolution of ambiguities in llms. we provide a literature review and highlight our differences. ambiguity taxonomy. as evidenced by a recent survey @xcite , there is a lack of a well-organized taxonomy for ambiguity in information retrieval. while previous research attempts to integrate ambiguity taxonomies, their taxonomies are fragmented and underdeveloped @xcite , failing to facilitate comprehensive evaluations. recent taxonomies min et al. (2020); guo et al. (2021) are formulated based on a limited set of factual questions and lack precise definitions for each category. moreover, existing taxonomies were established before the era of llms, disregarding the ambiguity specific to llms that may arise from conflicting interpretations of queries by the inherent knowledge stored within llms. this is evident when llms encounter unfamiliar entities @xcite or potential inconsistencies within queries @xcite . for the first time, we introduce a well-organized taxonomy for categorizing various query ambiguities. our taxonomy draws inspiration from the input-process-output view to evaluate collaborative systems. it combines three primary dimensions that capture potential ambiguities during input understanding and task completion of llms. using this taxonomy, we construct ∼ 12k data for analyzing the pros and cons of llms in resolving different ambiguities. ## clamber benchmark to evaluate llms in an inclusive manner, we present clamber, which introduces a taxonomy encompassing three key dimensions (i.e., epistemic misalignment, linguistic ambiguity, aleatoric output) that capture potential ambiguities during input understanding and task completion. these three dimensions are further divided into eight specific categories. we delve into the taxonomy and data collection process in following sections. each data comprises a user query, a binary ambiguity label, and a clarifying question for ambiguous queries. see details of data collection in appendix c. ## experimental design we consider two tasks to evaluate off-the-shelf llms, including identifying ambiguities (cf. section 5) and asking clarifying questions (cf. section 3 .5-turbo-16k (i.e., chatgpt). these llms are widely used in recent studies of information search @xcite . prompting schemes. following @xcite , we devise four prompting schemes for evaluation: 1) zero-shot w/o cot, where the llm is evaluated directly on the test dataset, 2) zero-shot w/ cot @xcite , where the llm starts with ambiguity analysis before making predictions, 3) few-shot w/o cot @xcite , where the llm is evaluated by providing examples, 4) few-shot w/ cot, where the llm is evaluated by providing examples with their corresponding ambiguity analysis. in the few-shot setting, we provide two randomly selected examples, one is ambiguous and the other is unambiguous. importantly, we carefully selected 3 prompts and test all llms on these prompts. we present the average performance across various prompts to guarantee the statistical significance of the experimental findings. details on prompts are presented in appendix a. ## task 1: identifying ambiguity this section aims to evaluate the ability of llms to identify different categories of ambiguous user queries, focusing on both the overall performance (cf. section 5.1) and performance specific to each category (cf. section 5.2). following @xcite 2023; deng et al., 2023b), we adopt the accuracy and f1 score as metrics. ## conclusion in this work, we introduce clamber, a benchmark for evaluating llms in identifying and clarifying ambiguous user queries through a wellorganized taxonomy. our clamber comprises ∼ 12k high-quality data covering a wide range of ambiguity categories. with clamber, we assess strengths, weaknesses, and potential risks of various off-the-shelf llms. our results indicate that current llms still face difficulties in achieving optimal performance in ambiguity identification and clarification, limiting their practical utility in advanced information search applications. in this paper, clamber acts as a foundation for enhancing the proactive capabilities of llms in addressing ambiguity. moving forward, we plan to integrate more challenging and comprehensive datasets into our clamber based on our taxonomy.