F an O ut QA : A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models (2024)

Abstract: One type of question that is commonly found in day-to-day scenarios is “fan-out” questions, complex multi-hop, multi-document reasoning questions that require finding information about a large number of entities. However, there exist few resources to evaluate this type of question-answering capability among large language models. To evaluate complex reasoning in LLMs more fully, we present FanOutQA, a high-quality dataset of fan-out question-answer pairs and human-annotated decompositions with English Wikipedia as the knowledge base. We formulate three benchmark settings across our dataset and benchmark 7 LLMs, including GPT-4, LLaMA 2, Claude-2.1, and Mixtral-8x7B, finding that contemporary models still have room to improve reasoning over inter-document dependencies in a long context. We provide our dataset, along with open-source tools to run models to encourage evaluation.

## introduction in real-world production deployments, large language models (llms) are often asked "fan-out" questions: questions that require models to find a list of entities and then consult a large number of documents to aggregate information about those entities to answer a user's question. this pattern of question can be found commonly in dayto-day scenarios, such as performing a literature review (fan-out over research papers), planning a trip (fan-out over attractions), or choosing where to eat (fan-out over nearby restaurants). the fan-out task is particularly challenging because it requires multi-hop reasoning across multiple documents, and the combined length of the documents needed to answer the question typically exceeds the length of a model's context window. existing questionanswering benchmarks like hotpotqa @xcite figure 1 : the fanoutqa dataset contains multi-hop, multi-document "fan-out" questions along with humanwritten decompositions (bottom). we formulate three challenge settings for llms to answer these fan-out questions to test capabilities of llms (top). 2018), longbench @xcite , and zero-scrolls @xcite focus on intradocument dependencies or dependencies between a small number of documents, which does not sufficiently evaluate models' performance on this type of task. in this paper, we present fanoutqa, a high quality dataset of 1,034 information seeking questions, 7,305 human-written decompositions, and their answers, along with a multi-hop, multi-document benchmark using english wikipedia as its knowledge base. compared to other question-answering benchmarks, fanoutqa requires reasoning over a greater number of documents, with its main focus being on the fan-out style of question (figure 1 ). we formulate three distinct challenge settings over the dataset. the closed-book setting requires the model to answer fan-out questions without external knowledge, testing its general knowledge. the open-book setting gives models access to retrieval tools, testing their ability to retrieve relevant articles and reason across multiple long documents. finally, the evidence-provided setting provides the models with relevant articles, testing their longcontext and multi-hop reasoning capabilities. we find that the closed-and open-book settings are difficult for modern systems, with the best performing models scoring below 50%. in the openbook setting, retrieved documents outgrow models' context lengths. in the evidence-provided setting, models' performance correlates strongly with their context length. human volunteers completing the open-book task score 85% accuracy, showing room to improve llm systems. ## fanoutqa dataset fanoutqa consists of three parts: questions, answers, and evidence. each question includes a decomposition into sub-questions that can be answered with a single wikipedia article. the answers to the sub-questions can then be combined to answer the top-level question. we provide these sub-questions, answers, and associated wikipedia articles as an additional resource for decomposing complex queries. we provide sample questions in appendix a, and the dataset's topic distribution in appendix b. ## benchmarking study we benchmarked seven large language models on fanoutqa: gpt-4, gpt-4-turbo, gpt-3.5-turbo, llama 2 70b chat, mistral-7b, mixtral-8x7b, and claude 2 (more details in appendix d). all models generated text with greedy decoding; all local models were run with fp16 precision. ## conclusions fan-out question answering presents several challenges for llms, including decomposing complex questions into simpler sub-questions, retrieving documents, extracting relevant information, and multi-hop reasoning over a large number of documents. we developed a dataset called fanoutqa for this ambitious task in response to the rapidly improving reasoning abilities and context management strategies in large language models, and we formulate three challenge settings over the dataset. we benchmarked the performance of seven state-ofthe-art models on our challenge settings, and find that the requirement of fan-out question-answering challenges even the long context capabilities of modern models. accuracy correlated with context length in the open book and evidence-provided but not in the closed book settings, suggesting that more information helps performance. the correlation was stronger in the evidence-provided setting, further suggesting that the quality of information matters as well. in our experiments, our main goal was to evaluate llms' answers to the top-level questions in the three settings we present. as there may be multiple valid decompositions to achieve a final answer, we don't evaluate on the similarity between the human-written question decompositions and strategies used by llms (most relevant in the open book setting). however, we would like to highlight its usefulness for imitation learning (e.g. fine-tuning a function-calling-capable model) as a direction for future work. we also encourage exploration of additional decompositional prompting strategies, such as decomposed prompting @xcite and gendec @xcite . we encourage researchers to use fanoutqa to evaluate new retrieval-augmented models, longcontext models, and other novel llm systems with our open-source resources. foot_1