MDB ench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance (2025)

Abstract: Natural language processing evaluation has made significant progress, largely driven by the proliferation of powerful large language mod-els (LLMs). New evaluation benchmarks are of increasing priority as the reasoning capabilities of LLMs are expanding at a rapid pace. In particular, while multi-document (MD) reasoning is an area of extreme relevance given LLM capabilities in handling longer-context inputs, few benchmarks exist to rigorously examine model behavior in this setting. Moreover, the multi-document setting is historically challenging for benchmark creation due to the expensive cost of annotating long inputs. In this work, we introduce MDBench, a new dataset for evaluating LLMs on the task of multi-document reasoning. Notably, MDBench is created through a novel synthetic generation process, allowing us to controllably and efficiently generate challenging document sets and the corresponding question-answer (QA) examples. Our novel technique operates on condensed structured seed knowledge, modifying it through LLM-assisted edits to induce MD-specific reasoning challenges. We then convert this structured knowledge into a natural text surface form, generating a document set and corresponding QA example. We analyze the behavior of popular LLMs and prompting techniques, finding that MDBench poses significant challenges for all methods, even with relatively short document sets. We also see our knowledge-guided generation technique (1) allows us to readily perform targeted analysis of MD-specific reasoning capabilities and (2) can be adapted quickly to account for new challenges and future modeling improvements.

## introduction the rapid advancements in natural language processing (nlp) have been largely driven by the development and deployment of large language models (llms). these models have showcased remarkable improvements in various tasks, including understanding, generating, and reasoning over text. however, despite these advancements, evaluation frameworks for nlp systems have struggled to keep pace @xcite , notably for tasks involving reasoning over multiple documents @xcite . multi-document (md) reasoning involves synthesizing and inferring information across multiple diverse texts @xcite , posing unique challenges not addressed by traditional single-document benchmarks. while llms are increasingly capable of handling longer-context multi-document inputs, there is a scarcity of benchmarks that rigorously examine the specific reasoning characteristics that are prominent in this setting. in addition, many existing benchmarks consist of static, hand-crafted datasets, which are labor-intensive to produce. these datasets are often susceptible to data contamination @xcite over time, e.g., llms are exposed to public benchmarks during training. this can compromise the integrity of the evaluation. in this work, we address these limitations with mdbench foot_0 , a benchmark using a novel generation technique for multi-document reasoning evaluation. our benchmark is generated through a synthetic process that leverages structured knowledge as seed information. this process uses a strong llm (gpt-4o) to augment structured knowledge by injecting complexities that require advanced reasoning skills, then generates text documents from the augmented knowledge. synthetic generation allows for efficient example creation and can be easily adapted to incorporate new reasoning skills and objectives. our benchmark generation pipeline (overviewed in figure 1 ) begins with a structured knowledge table generation "create a new table using the provided instructions: {orig. table} {edit plan}" edit plan 1) create question+answer: "which team won the series…" 2) add context to by associating buffalo introduce soft knowledge dependency with blue and gold in april 17 and april 21 . . 5) add style/prose for each row to simulate a real-world document step 4: discard examples w/ poor "oracle" consistency orig. table edit plan new table question orig. table edit plan docs question orig. table edit plan new table docs question correct? correct? correct? verdict ✅ ✅ ✅ keep ✅ ✖ ✅ reject figure 1: mdbench generation pipeline overview. we 1) source structured knowledge, then use in-context multi-document reasoning demonstrations to 2) intentionally modify the existing knowledge with challenging dependencies. we then 3) map this seed knowledge into document form to produce the multi-document qa example. finally, we 4) employ an added validity check to filter inconsistent examples that yield erratic outputs in an oracle setting where example construction knowledge is provided. ## related work evaluating the capabilities of llms is a critical aspect of nlp research. as llms continue to improve rapidly, existing evaluation frameworks often lag behind, particularly in assessing complex reasoning abilities such as multi-document (md) reasoning. as llms rapidly increase in reasoning capacity, there is a pressing need to develop evaluation methods that can capture these higher-order reasoning skills. multi-document reasoning md reasoning involves synthesizing and inferring information across multiple texts. existing work in this area includes datasets targeting specific phenomena such as temporal reasoning @xcite , summarization @xcite @xcite , multi-hop question answering with hotpotqa, musique and 2wiki-multihop @xcite @xcite and ambiguous entity resolution @xcite . notably, many of these md datasets are publicly-sourced and often reliant on significant human effort to curate. for example, @xcite introduce fanoutqa, a recent multi-hop, multi-document question answering dataset, which targeted decomposable qa examples sourced from public wikipedia knowledge and relied on thousands of manual annotations. our work seeks to use knowledge-controlled generation to offer a scalable alternative for producing nuanced and unseen multi-document reasoning examples. tabular reasoning with llms llms have demonstrated strong performance in tasks involving structured knowledge, such as tabular data or knowledge bases @xcite . recent studies have observed success in applying llms to table reasoning, manipulation, and augmentation @xcite . while there are limitations in llm pre-training which can lead to formatting sensitivities and limitations with handling large tables, nahid and rafiei (2024) find improved performance by decomposing the tabular knowledge into a digestible size. similarly, leveraging tabular knowledge within reasoning chains allows for compact and effective representation of complex problems, as explored in the chainof-tables framework @xcite . these insights highlight the potential of using condensed knowledge as a foundation for generating challenging reasoning tasks. although our work draws on structured data to simplify the creation of examples, our ultimate goal is to produce longer, more nuanced natural-language text document sets. ## mdbench generation pipeline in this section, we motivate and overview the generation process, and provide details on the components and steps taken to produce the mdbench evaluation benchmark. our design decisions for the mdbench benchmark are guided by several key objectives to ensure the resulting examples test a broad and relevant range of reasoning skills: answer rationale: turkey had 200 showings and belgium had 600. 600 > 200, therefore turkey had the most showings. commentary: this is a simple reasoning process as it requires a simple comparison of two values with no additional reasoning required. harder example: which country had the most showings and how many total? answer: turkey had the most with 700 showings. answer rationale: turkey had showings on two different days, so the total is 200+500=700 showings. 700 > belgium's 600, therefore turkey had the most. commentary: by adding a new row with complementary information, we necessitate an additional reasoning hop to correctly answer the question. note that this table was edited specifically such that the answer (turkey) is flipped from the original answer (belgium) in the simple example. figure 2 : example skill description on "multi-hop reasoning". during knowledge augmentations, we demonstrate the multi-document skills relevant to the document sets. ## experimental setup to assess the challenges of mdbench, we test the performance of many popular llms in combination with conventional prompting setups. concretely, we test open-weights llms with meta's llama-3 (dubey et al., 2024), using the 8b-instruct and 70b-instruct variants. for api-based table reasoning is when evaluated with the intermediate table qa examples. document reasoning refers to the performance on the final task of multi-document reasoning. document reasoning poses a harder challenge, speaking to the importance of surface form in reasoning performance. proprietary methods, we use models from the anthropic claude, openai gpt, and google gemini families, which represent current state-of-the-art llms. for claude we evaluate claude-3-opus 20240229 and claude-3.5-sonnet 20240620 foot_1 . for gpt we test gpt-3.5-turbo-16k 0613 foot_2 (ouyang et al., 2022), gpt-4o 2024-08-06 foot_3 , and gpt-o1 2024-12-17 foot_4 . for gemini we include gemini-1.5-pro 0514 (team et al., 2024b) and the recent gemini-2.5-flash-preview 05-20 foot_5 . we explore both zero-shot and one-shot qa prompting scenarios, noting that when prompting in the one-shot case we use a single representative demonstration across models for consistency. we use a conventional question-answering prompt, and also further instruct the models to 'think step by step' to additionally produce chain-of-thought (cot) rationales. examples of these prompt formats are provided in appendix b. to evaluate on the qa task, we use gpt-4o as a reference-based scorer, first parsing the final answer from each output, then comparing the similarity of the predicted answer with the ground-truth answer (conditioned on the original question). we calculate ## results and analysis overall findings figure 3 overviews the overall task performance on the multi-document and intermediate tabular versions of the dataset, while overall, we see relatively strong performance from one or more models from each family, with claude-3.5-sonnet performing best among the claude family. the use of cot prompting yields performance gains in larger models like gemini-flash-2.5, gpt-o1 and gpt-4o, although the effect varies. we see open-weight models struggle overall, although llama-3-70b bests gpt-3.5-turbo, with the 8b variant comparable in performance to gpt-3.5. 50.2 41.7 ## conclusion we present mdbench, a novel benchmark for evaluating multi-document reasoning in llms. leveraging structured seed knowledge and augmenting it with nuanced reasoning dependencies, mdbench enables the systematic creation of challenging qa examples while addressing data contamination and annotation costs. our results reveal significant challenges for current models, and our approach facilitates scalable, targeted evaluation of multi-document reasoning capabilities, paving the way for more rigorous evaluation of models' abilities to handle real-world, multi-source information. xanh ho, anh-khoa duong nguyen, saku sugawara, and akiko aizawa. 2020. constructing a multihop qa dataset for comprehensive evaluation of reasoning steps. in proceedings of the 28th international conference on computational linguistics, pages 6609-6625, barcelona, spain (online). international committee on computational linguistics. mehran kazemi, quan yuan, deepti bhatia, najoung kim, xin xu, vaiva imbrasaite, and deepak ramachandran. 2023. boardgameqa: a dataset for natural language reasoning with contradictory information. preprint, arxiv:2306.07934. yoonsang lee, xi ye, and eunsol choi. 2024. ambigdocs: reasoning across documents on different entities under the same name. preprint, arxiv:2404.12447. hongxin li, jingran su, yuntao chen, qing li, and zhaoxiang zhang. 2023a. sheetcopilot: bringing software productivity to the next level through large language models. preprint, arxiv:2305.19308. zhuoyan li, hangxiao zhu, zhuoran lu, and ming yin. 2023b. synthetic data generation with large language models for text classification: potential and limitations. in proceedings of the 2023 conference on empirical methods in natural language processing, pages 10443-10461, singapore. association for computational linguistics. gili lior, avi caciularu, arie cattan, shahar levy, ori shapira, and gabriel stanovsky. 2024. seam: figures 7, 8, 9, 10, 11 overview the five reasoning skills we demonstrate during the creation of mdbench. figure 6 demonstrates an edit plan provided to inspire the table augmentation. b model evaluation prompts simple qa prompt "you will be presented with a question and a context. you should answer the question based on the context. the last thing you generate should be answer:[your answer here]" chain-of-thought qa prompt "you will be presented with a question and a context. you should answer the question based on the context. explain your reasoning step by step before you answer. the last thing you generate should be an-swer:[your answer here]" c example post-processing and scoring prompts these prompts are applied after example construction to (a) normalise the ground-truth format, (b) extract concise model predictions after inference, and (c) assign a scalar correctness score to the processed predictions. qa example formatting prompt "adjust the question and answer format such that the question includes output formatting instructions like "output the answer as a json with @xmath0, and ensure the answer follows a corresponding json structure. question: {question} answer: {answer}" predicted answer extraction prompt "take this response (which may include verbose reasoning) and concisely extract only the final answer to the provided question. question: {question} answer: {model_response}" scoring prompt "on a scale of 0-10, how consistent is the predicted answer with the ground-truth? partial credit is acceptable if a portion of the response matches the ground-truth. re-turn only an integer. question: {question} predicted answer: {predicted_answer} ground-truth answer: {ground_truth}"