Answering Ambiguous Questions via Iterative Prompting (2023)

Abstract: In open-domain question answering, due to the ambiguity of questions, multiple plausible answers may exist. To provide feasible answers to an ambiguous question,one approach is to directly predict all valid answers, but this can struggle with balancing relevance and diversity. An alternative is to gather candidate answers and aggregate them, but this method can be computationally costly and may neglect dependencies among answers. In this paper, we present AmbigPrompt to address the imperfections of existing approaches to answering ambiguous questions. Specifically, we integrate an answering model with a prompting model in an iterative manner. The prompting model adaptively tracks the reading process and progressively triggers the answering model to compose distinct and relevant answers. Additionally, we develop a task-specific post-pretraining approach for both the answering model and the prompting model, which greatly improves the performance of our framework. Empirical studies on two commonly-used open benchmarks show that AmbigPrompt achieves state-of-the-art or competitive results while using less memory and having a lower inference latency than competing approaches. Additionally, AmbigPrompt also performs well in low-resource settings.

## introduction recent years have witnessed substantial advances in open-domain question answering (qa) systems @xcite @xcite , which aim to find the answer for the given question from a large knowledge corpus @xcite . while a dominating scenario is the single-answer qa setting, i.e., only one exact answer is required for a given question @xcite , this work focuses which movie was both directed and screen written by kamal haasan? vishwaroopam (titled vishwaroop in hindi;) is a 2013 indian espionage action thriller film written, directed and produced by kamal haasan, who also enacts the lead role. vishwaroopam ii, or vishwaroop ii, is a 2018 indian espionage action thriller film written and directed by kamal haasan, it is the sequel to "vishwaroopam" (2013) and features himself alongside rahul bose, shekhar kapur, pooja kumar and andrea jeremiah, reprising their roles question passages answers vishwaroopam vishwaroopam ii sabaash naidu virumaandi figure 1 : an example of an open-domain question, a subset of its evidential wikipedia passages and multiple answers they lead to. on the more realistic scenario of multi-answer qa, where multiple plausible answers are associated with a user-issued question @xcite , given that questions posed by humans are often openended and ambiguous. 1 a natural approach for answering ambiguous open-domain questions would be to fine-tune a pretrained answer generation model, e.g., t5 @xcite , using supervised data of the form (evidential passages, question, all plausible answers) @xcite . however, this approach often leads to sub-optimal solutions since it requires the model to balance the relevance and diversity of the generated multiple answers within a single-round decoding procedure, which is non-trivial. to manage the relevance-diversity trade-off, another approach is to decompose multianswer qa into candidate answer prediction and answer post-processing. this typically requires a high-capacity model with billions of parameters to construct candidate answers and sophisticated answer aggregation pipelines to obtain the final results @xcite , incurring high computational costs. in addition, this approach suffers from the dilemma of having to predict diverse candidate answers before knowing which answer has been predicted, which is unnatural and intricate. for example, in figure 1 , given the question "which movie was both directed and screenwritten by kamal haasan?," with the existence of the answer vishwaroopam, the model excludes its eponymous translation version vishwaroop and deduces that vishwaroopam ii is another potential answer. when facing an ambiguous question, people are capable of providing multiple valid answers by introspectively composing new content on the basis of what has already been devised, usually in an iterative manner. inspired by this observation, in this paper, we conceptualize ambigprompt as an approach to mimic this mechanism by iteratively guiding the answering model with a lightweight prompting model. as shown in figure 2 , this prompting model steers the answering model to progressively generate valid answers whose content the prompting model will then condition on for the next-round prompt construction. essentially, our proposed framework comprises two key components: (i) an encoder-decoder answering model and (ii) an interleaving answer-conditional prompting model. by conditioning on preceding generated contents, the proposed framework introspectively perceives which answer has been predicted before updating the hidden activation for the generation of subsequent answers. furthermore, we devise a task-adaptive post-pretraining strategy, in which pseudo multi-qa training instances are constructed to facilitate the training of the proposed framework. we carry out extensive experiments on the am-bigqa @xcite and @xcite datasets. the results demonstrate that ambigprompt attains superior performance despite having a significantly smaller parameter scale, 14 times less than state-of-the-art models. furthermore, as a lightweight approach, ambigprompt improves the answer relevance and diversity with a tiny fraction of the memory footprint and inference latency of competing approaches. notably, ambig-prompt achieves the best performance in the lowresource setting. the effectiveness of the proposed method is also verified by ablation experiments and ## ambigprompt conventionally, the question answering model generates the desired answer given the input context in a single pass (izacard and grave, 2021b). while it suffices to tackle the single-answer qa scenario, managing ambiguous questions with multiple answers can be more nuanced -the answering model is required to balance the relevance and diversity of the generated answers in a single pass, and precisely modeling dependencies among the answers can be non-trivial. in this paper, we propose am-bigprompt, a question-answering model that answers ambiguous questions via iterative prompting, inferring more accurate answers progressively. figure 2 gives an overview of the proposed method. overall, ambigprompt decomposes the generation of answers a into multiple steps instead of one single pass, i.e., p(a|q, c) = n @xmath0, q, c), ## conclusions in this paper, we have proposed ambigprompt for multi-answer qa. ambigprompt is a simple yet effective model that answers ambiguous questions by iterative prompting. we have proposed an answerconditional prompting model for prompt generation, and a task-adaptive post-pretraining scheme for model training. extensive experiments suggest that ambigprompt achieves comparable performance as high-capacity models and achieves the best results in a low-resource setting.