MDCR : A Dataset for Multi-Document Conditional Reasoning (2024)

Abstract: The same real-life questions posed to different individuals may lead to different answers based on their unique situations. For instance, whether a student is eligible for a scholarship depends on eligibility conditions, such as major or degree required. ConditionalQA was proposed to evaluate models’ capability of reading a document and answering eligibility questions, considering *unmentioned* conditions. However, it is limited to questions on single documents, neglecting harder cases that may require *cross-document reasoning* and *optimization*, for example, “What is the maximum number of scholarships attainable?” Such questions over multiple documents are not only more challenging due to more context to understand, but also because the model has to (1) explore all possible combinations of unmentioned conditions and (2) understand the relationship between conditions across documents, to reason about the optimal outcome. To evaluate models’ capability of answering such questions, we propose a new dataset MDCR, which can reflect real-world challenges and serve as a new test bed for complex conditional reasoning that requires optimization. We evaluate this dataset using the most recent LLMs and demonstrate their limitations in solving this task. We believe this dataset will facilitate future research in answering optimization questions with unknown conditions.

## introduction answers to many real-life questions depend on geographical or temporal situations (min et al., 2020; zhang and choi, 2021; stelmakh et al., 2023) or the facts about the person who asked the question (sun et al., 2022). consider a high school senior student reading a scholarship document that describes its eligibility conditions. the answer to whether the student is eligible for the program depends on, for instance, whether the student is "planning to pursue a degree in a us post-secondary education". the answer is "yes" if this condition is satisfied and "no" otherwise. this condition, despite being unmentioned in the question, is necessary for the "yes" answer to hold. scholarships, internships, and government benefits, among others, are popular domains where (eligibility) conditions frequently appear in documents and eligibility questions naturally occur. conditionalqa (sun et al., 2022) was proposed to examine the performance of models for such questions on single benefit documents in the public policy domain. yet, besides binary yes/no questions on single documents, users are also interested in asking maximization questions over multiple benefits that require optimization. for instance, students can ask questions about application strategies to maximize the number of scholarships attainable to cover tuition. low-income families can ask about qualifications to maximize their social benefits (e.g., tax credit, housing allowances) to improve living standards. existing datasets that target questions on single documents neglect these common questions over multiple documents (kulkarni et al., 2020; boni et al., 2021; bolotova-baranova et al., 2023). these questions that need to consider unmentioned conditions over multiple documents jointly pose several new challenges. first, it requires a fine-grained multi-document understanding. in particular, models not only need to understand the conditions in each document, but also the relationships between conditions across documents, which are critical to answering the multi-document questions correctly. figure 1 shows three common relationships: conflicting, equivalent, and inclusive. secondly, answering questions over multiple documents may require more complex reasoning capabilities for optimization. specifically, models may need to reason about a plan with additional, unmentioned conditions to achieve the best feasible outcome, based on their understanding of condition relationships and satisfiability. consider the user in figure 2 whose objective is to maximize the number of scholarships attainable. the model could decide to use plan 1, which only leads to a suboptimal solution, or it needs to look holistically at all three benefits, go through all possible combinations of the three scholarships (one trio, three pairs, and three singles), and leverage condition relationships to decide the group of conditions (plan 2) that results in the optimal outcome (here, the coca-cola scholarship and the microsoft scholarship). for instance, the model needs to understand the conflicting relationships between the croucher stu-dentship and both the coca-cola scholarship and the microsoft scholarship to conclude that no condition group can lead to an outcome of all benefits. reasoning with unmentioned conditions to identify optimized groups of conditions is not required for conditional reasoning over single documents. lastly, the multi-document setting magnifies the challenges in the single-document setting. in particular, it introduces more information to the context that could be relevant or irrelevant, which could potentially mislead models or lead to underutilization, as shown in recent studies (liu et al., 2024), thereby making scenario and document understanding and reasoning more difficult. to reflect the real-life challenges above, we propose a new dataset called mdcr (multi-document conditional reasoning). we collected documents from two domains (scholarships and jobs) and constructed questions that assess models' abilities to reason about different numbers of documents. we benchmarked on mdcr, show its challenges to recent llms, and present insights for improvement. most llms (including gpt-4o and llama3-70b) consistently achieve around 69% short answer accuracy and only around 40% conditional answer f1 (around 50% f1 for the relaxed version), demonstrating the difficulty of this task. ## task description in this section, we elaborate on the task of multidocument conditional reasoning, describe its inputs and outputs as well as how we evaluate the outputs. ## data collection in this section, we provide an overview of the data collection process. more details are provided in appendix a. ## benchmark in this section, we evaluate recent llms on our dataset and elaborate on their performance under different settings. ## conclusion conditional reasoning is crucial in many domains, such as scholarships, job applications, and government benefits. it involves understanding the eligibility conditions and determining optimal outcomes based on users' satisfiability of conditions. however, existing works focus only on such reasoning over single documents, neglecting situations where users want to search for optimized outcomes that span through multiple documents. in this paper, we introduced mdcr to address this gap and benchmarked recent large language models on this dataset. as evidenced by the results, the multidocument setting brings significant challenges to models and suggests their limited understanding and reasoning capabilities. we hope this serves as the foundation for future work that examines complex conditional reasoning.