Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models (2023)

Abstract: Questions in open-domain question answering are often ambiguous, allowing multiple interpretations. One approach to handling them is to identify all possible interpretations of the ambiguous question (AQ) and to generate a long-form answer addressing them all, as suggested by Stelmakh et al., (2022). While it provides a comprehensive response without bothering the user for clarification, considering multiple dimensions of ambiguity and gathering corresponding knowledge remains a challenge. To cope with the challenge, we propose a novel framework, Tree of Clarifications (ToC): It recursively constructs a tree of disambiguations for the AQ—via few-shot prompting leveraging external knowledge—and uses it to generate a long-form answer. ToC outperforms existing baselines on ASQA in a few-shot setup across the metrics, while surpassing fully-supervised baselines trained on the whole training set in terms of Disambig-F1 and Disambig-ROUGE. Code is available at https://github.com/gankim/tree-of-clarifications.

## introduction in open-domain question answering (odqa), users often ask ambiguous questions (aqs), which can be interpreted in multiple ways. to handle aqs, several approaches have been proposed, such as providing individual answers to disambiguated questions (dqs) for all plausible interpretations of the given aq @xcite or asking a clarification question @xcite . among them, we adopt that of @xcite , which provides a comprehensive response without bothering the user for clarification: the task is to identify all dqs of the given aq and generate a long-form answer addressing all the dqs (see figure 1 ). there are two main challenges to this task: (1) the aq may need to be clarified by considering multiple dimensions of ambiguity. for example, the aq "what country has the most medals in olympic history" in figure 1 can be clarified with respect to the type of medals-gold, silver, or bronze-or olympics-summer or winter; and (2) substantial knowledge is required to identify dqs and respective answers. for example, it requires knowledge to be aware of the existence of different types of medals and the exact counts for each country. to address the challenges and provide a longform answer to aq, we propose a novel framework, tree of clarifications (toc): it recursively constructs a tree of dqs for the aq-via few-shot prompting leveraging external knowledge-and uses it to generate a long-form answer. more specifically, first, relevant passages for the aq are retrieved. then, leveraging the passages, dqs for the aq are recursively generated via few-shot prompting and pruned as necessary. lastly, a long-form answer addressing all dqs is generated. the tree structure promotes exploring dqs in targeting particular dimensions of clarification, addressing the first challenge, and the external sources offer additional knowledge to cope with the second challenge. experiments demonstrate that our proposed use of llms with retrieval-augmentation and guidance to pursue diverse paths of clarification results in the new state-of-the-art on asqa @xcite -a long-form qa benchmark for aqs. toc outperforms existing baselines on asqa in a few-shot setup across all metrics. in addition, this 5-shot performance surpasses that of the fullysupervised baselines trained on the whole training set by 7.3 and 2.9 in terms of disambig-f1 and disambig-rouge, respectively. the main contribution of this work is proposing a novel framework, tree of clarifications (toc), for generating long-form answers to aqs in odqa, advancing the state-of-the-art on the asqa benchmark. toc introduces two main innovations: ## related work a line of studies @xcite @xcite extends retrieve-andread frameworks dominant in odqa task @xcite @xcite to clarify aq and generate dqs with corresponding answers to them. however, their approaches require fine-tuning models on the large-scale train set. on the other hand, our framework enables llm to generate a comprehensive response addressing all dqs via few-shot prompting. recent studies introduce llm-based methods to generate a long-form answer to the aq. amplayo et al. (2023) suggest optimal prompts specifically engineered for the task. @xcite prompt llms to clarify ambiguous questions selectively. however, the studies do not utilize external information to ensure the factual correctness of the disambiguations, thereby potentially increasing the risk of hallucinations from llms. moreover, the results could be bounded by inherent parametric knowledge of llm. concurrently, @xcite automatically generate clarifying questions to resolve ambiguity. our framework involves the recursive tree architecture, inspired by several prior studies. @xcite propose the tree-decoding algorithm to autoregressively rerank passages in ambiguous qa. @xcite iteratively explore additional interpretations and verify them in a round-trip manner. concurrently, extending chain of thoughts @xcite apply the tree architecture to reasoning tasks for deductive or mathematical problems. on the contrary, toc recursively clarifies questions and introduces a self-verification method to prune unhelpful dqs. ## tree of clarifications we introduce a novel framework, tree of clari-fications (toc), as illustrated in figure 1 . we first devise retrieval-augmented clarification (rac; sec. 3.1), a basic component that clarifies aq and generates dqs based on relevant passages. toc explores various fine-grained interpretations, represented as a tree structure (ts; sec. 3.2) by recursively performing rac and pruning unhelpful dqs. lastly, it aggregates the tree and generates a long-form answer addressing all valid interpretations. ## conclusion in this work, we propose a novel framework, tree of clarifications. it recursively builds a tree of disambiguations for the aq via few-shot prompting with external knowledge and utilizes it to generate a 7 see appendix 7 for failure cases 8 the idea is aligned with the annotation process of am-bigqa @xcite , in which the target question is classified as ambiguous if multiple distinct answers to it were observed.