CORG : Generating Answers from Complex, Interrelated Contexts (2025)

Abstract: In a real-world corpus, knowledge frequently recurs across documents but often contains inconsistencies due to ambiguous naming, outdated information, or errors, leading to complex interrelationships between contexts. Previous research has shown that language models struggle with these complexities, typically focusing on single factors in isolation. We classify these relationships into four types: distracting, ambiguous, counterfactual, and duplicated. Our analysis reveals that no single approach effectively addresses all these interrelationships simultaneously. Therefore, we introduce Context Organizer (COrg), a framework that organizes multiple contexts into independently processed groups. This design allows the model to efficiently find all relevant answers while ensuring disambiguation. COrg consists of three key components: a graph constructor, a reranker, and an aggregator. Our results demonstrate that COrg balances performance and efficiency effectively, outperforming existing grouping methods and achieving comparable results to more computationally intensive, single-context approaches.

## introduction in real-world documents-ranging from blog posts and news articles to official records or usergenerated content-the same knowledge often appears in multiple forms, sometimes with consistency but often with variations or conflicts. these discrepancies can arise from ambiguous phrasing, outdated data, or simple errors. when analyzing these contexts, we find relationships between them generally fall into four categories as shown in figure 1 : distracting, ambiguous, counterfactual, or duplicated. each entity consists of a surface name, a general term that can be ambiguous, and a descriptor that provides specificity. for instance, in "the simpsons (season 2) contains 22 episodes", "the simpsons" is the surface name, and "season 2" is the descriptor that specifies the entity. based on these attributes, we classify contexts as distracting (same surface name, different descriptors), ambiguous (same surface name, only one with a descriptor), counterfactual (same entity with differing answers), or duplicated (same entity with identical answers). current research often simplifies real-world complexities by treating corpora as unified sources (e.g., fixed wikipedia versions), where knowledge appears consistently without cross-document conflicts. many studies also address isolated factors rather than considering such complex interrelationships between contexts @xcite @xcite . additionally, finding relevant contexts using webbased retrieval methods often introduces search engine biases, resulting in limited diversity @xcite . to bridge this gap, we expand existing corpora to better reflect such complex interrelation between contexts in realworld conditions.specifically, we introduce ambig-docs+ and conflictqa+ -extensions of ambig-docs @xcite and conflictqa @xcite )-where we construct additional contexts based on the (question, answer, context) pairs, ensuring coverage of all four conditions. when analyzing the effect of each factor, we observe a performance drop, particularly when am-biguous or counterfactual contexts are added, consistent with prior findings @xcite @xcite . our investigation reveals that no single simple solution addresses such contexts in complex relationships simultaneously; although simple methods exist for individual factors, they often fail to generalize across scenarios. interestingly, we found that a straightforward solution for distracting contexts is to modify the question to a plural form. for ambiguous contexts, adding or replacing missing descriptors to create a distracting relationship can improve clarity. however, these approaches are less effective for counterfactual contexts, where separating them into different forward passes appears to yield better performance. to address these challenges, we introduce con-text organizer (corg), a framework designed to improve performance on real-world corpora through a simple, efficient approach based on insights from individual solutions. corg prioritizes three objectives: (1) high answer recall, (2) accurate disambiguation, and (3) minimal inference runs. for cases with multiple answers, corg generates responses that include all relevant answers with citations, allowing users to review and filter information as needed. corg comprises three components: the graph constructor, which represents context relationships; the reranker, which organizes contexts into scenario-based groups; and the aggregator, which generates responses with citations for each group. we conduct experiments using eight language models of different sizes and observe that con-text organizer consistently improves performance over six baselines on both ambigdocs+ and conflictqa+, which contain multiple factors, as well as on ambigdocs and conflictqa, which each contain only a single factor. context or-ganizer notably excels in entity recall, measuring the model's ability to identify disambiguated entities. even large models show low disambiguation performance when simply processed without corg. additionally, grouping similar contexts without structured processing, as in corg, tends to reduce performance: groups with only similar contexts seem to confuse the model more than simply appending diverse contexts together. we hope our analysis of these real-world corpus scenarios encourages the community to explore the unique effects and solutions for each factor in greater depth. ## complex, interrelated contexts we analyze real-world contexts and observe that the relationship between contexts can be categorized into four types: distracting, ambiguous, counterfactual, and duplicated. in section 2.1, we define these four categories, followed by an analysis of their occurrence in real-world web corpora in section 2.2. in section 2.3, we describe how we extend an existing dataset to incorporate all four relationship types, and in section 2.4, we share details of evaluation metrics. ## analyzing solution for each factor in this section, we analyze how adding the context of each factor to the input affects model performance (section 3.1) and investigate a simple solution when looking at each factor individually (section 3.2). all evaluation in this section is performed with the llama2 7b model @xcite . ## context organizer in this section, we introduce context orga-nizer (corg), a framework designed for realworld corpora, based on observations in section 3.2. as shown in figure 3 , corg consists of three components: a graph constructor, a reranker, and an aggregator. the framework aims to achieve (1) high answer recall, (2) strong disambiguation, and ## experiments in this section, we share the experimental setup (section 5.1), six baselines (section 5.2), experimental results (section 5.3), and analysis over efficiency (section 5.4). ## related works various studies highlight incorporating external knowledge as a solution for hallucination and expanding the model's capability. however, performance is dependent on the relationships among input contexts. some works focus on ambiguous questions involving multiple relevant contexts @xcite , while others examine counterfactual or conflicting contexts relative to model knowledge @xcite @xcite @xcite . yet, real-world corpora often feature more complex relationships, which this paper categorizes into four types, revealing a gap in prior solutions for managing such complexities efficiently. some research leverages web-based corpora that might capture these relationships. however, we noticed that many of these works often rely on single sources, like wikipedia, where each fact is likely to appear only once @xcite @xcite or find relevant contexts using search engine api, which introduce bias to retrieve similar rather than diversely related contexts @xcite @xcite . in this work, we focus on the realworld challenge of complex relationships among input contexts, and analyze the impact of each factor. we further propose corg, a framework that achieves high performance and efficiency across diverse real-world cases without additional training. unlike previous works that use graph-based methods @xcite , corg constructs a graph where each node represents an entire document, and edges capture relationships between documents. in contrast, prior approaches focus on building graphs with nodes representing entities and edges capturing relationships within a single document. this distinction arises because our objective is to help llms better understand and process interconnected documents, whereas previous works focus on modeling relationships within individual documents. to the best of our knowledge, this is the first work to explore graph-based modeling for complex, interrelated documents. ## conclusion this work investigates the characteristics of realworld corpora, categorizing them into four types and examining their effects and solutions. we find that no single solution addresses all four factors. based on the observations, we introduce context organizer (corg), a framework consisting of three components-graph constructor, reranker, and aggregator-designed to achieve high answer recall, effective disambiguation, and efficiency in real-world corpora with complex relationships between contexts. it is applicable to any model without requiring additional training. experiments across four datasets with eight models demonstrate that corg consistently enhances performance, outperforming six baselines and achieving the best trade-off between efficiency and effectiveness.