Query Refinement Prompts for Closed-Book Long-Form QA (2023)

Abstract: Large language models (LLMs) have been shown to perform well in answering questions and in producing long-form texts, both in few-shot closed-book settings. While the former can be validated using well-known evaluation metrics, the latter is difficult to evaluate. We resolve the difficulties to evaluate long-form output by doing both tasks at once – to do question answering that requires long-form answers. Such questions tend to be multifaceted, i.e., they may have ambiguities and/or require information from multiple sources. To this end, we define query refinement prompts that encourage LLMs to explicitly express the multifacetedness in questions and generate long-form answers covering multiple facets of the question. Our experiments on two long-form question answering datasets, ASQA and AQuAMuSe, show that using our prompts allows us to outperform fully finetuned models in the closed book setting, as well as achieve results comparable to retrieve-then-generate open-book models.

## introduction interest in large language models (llms) has exploded in recent years due to their wide range of abilities improving the state-of-the-art with just a few or no examples @xcite @xcite . one task that benefited greatly is closed-book question answering @xcite in a few-shot setting, i.e., to produce correct answers to questions without access to passages to read and find the answer. most of the impressive results, however, are limited to generating short answers, and while previous work has utilized llms to generate long-form texts @xcite , most of these outputs are difficult to evaluate since they are subjective in nature (i.e., they may have multiple correct and distinctive answers). in this paper, we attempt to evaluate the long-form generation ability of llms through long-form question answering @xcite in the closed-book setting. unlike question answering with short answers, questions with long-form answers are naturally multifacted, covering many aspects that are required to fully answer the questions (see our study is based on two long-form question answering benchmarks, asqa @xcite and aquamuse @xcite , which focus on queries that may be ambiguous and/or require information from multiple sources. using these two datasets, our work is the first to show that llms are capable of generating long-form answers to complex questions of various types in a closed-book setting. we devise novel query refinement prompts that encourage llms to express multiple facets of input question and generate multifaceted answers discussing all identified interpretations coherently. specifically, we identify several different types of multifacetedness in questions and produce a labeled set of query refinement prompts for question answer pairs with a balanced coverage over different asqa question: when was the first car made in america? ## experiments on ambiguous question answering we conducted experiments on two question answering tasks that require long-form answers: ambiguous question answering (asqa; @xcite and query-focused multi-document summarization (aqua-muse; @xcite . in this section, we present results on asqa; the following section presents results on aquamuse. ## conclusions in this paper, we investigated the ability of large language models to answer questions in a long-form manner through two question answering datasets: asqa and aquamuse. we introduced query refinement prompting that improves over standard few-shot prompting and prompt tuning methods by encouraging the model to explicitly express the multifacetedness in questions. with the use of query refinement prompts on both few-shot closed book prompting and prompt tuning settings, we are able to outperform systems trained using the full training data in both asqa and aquamuse. while we also achieve comparable results with open book systems in asqa, we acknowledge that these systems still perform better when they have access to more retrieved passages. for future work, we plan to explore ways to few-shot prompt large language models in the open book setting and ways to augment a retrieval component into large language models.