MAGIC : A Multi-Hop and Graph-Based Benchmark for Inter-Context Conflicts in Retrieval-Augmented Generation (2025)

Abstract: Knowledge conflict often arises in retrieval-augmented generation (RAG) systems, where retrieved documents may be inconsistent with one another or contradict the model’s parametric knowledge.Existing benchmarks for investigating the phenomenon have notable limitations, including a narrow focus on the question answering setup, heavy reliance on entity substitution techniques, and a restricted range of conflict types. To address these issues, we propose a knowledge graph (KG)-based framework that generates varied and subtle conflicts between two similar yet distinct contexts, while ensuring interpretability through the explicit relational structure of KGs.Experimental results on our benchmark, MAGIC, provide intriguing insights into the inner workings of LLMs regarding knowledge conflict: both open-source and proprietary models struggle with conflict detection—especially when multi-hop reasoning is required—and often fail to pinpoint the exact source of contradictions.Finally, we present in-depth analyses that serve as a foundation for improving LLMs in integrating diverse, sometimes even conflicting, information.

## introduction retrieval-augmented generation (rag) has become the de facto standard for enhancing the performance of large language models (llms) by enabling updates to outdated knowledge and facilitating adaptation to specialized domains @xcite . while effective, rag's heavy reliance on retrieval quality introduces inherent risks. for instance, knowledge obtained from external sources may conflict with the model's parametric knowledge or contain inconsistencies within the retrieved documents themselves. knowledge conflict (kc) is a recent research topic that covers issues related to the aforementioned scenarios and has been receiving attention in the field @xcite ). an ideal llm-based system should be robust against knowledge conflict, capable of integrating information from multiple sources-including those that may be contradictory-and ultimately generating reliable responses. however, its implementation is largely hindered by the challenge of detecting whether disagreements exist across different knowledge sources and, if so, pinpointing exactly where they occur. numerous benchmarks have been introduced to evaluate the performance of llms in knowledge conflict detection @xcite @xcite . nonetheless, we emphasize that existing research in this area has notable limitations. first, previous studies primarily focus on the question answering (qa) task, where conflicts occur only among multiple answer candidates for a given question @xcite @xcite . second, prior research often relies on overly simplistic techniques for dataset construction, e.g., entity substitution @xcite , which are insufficient to capture the complex and subtle nature of knowledge conflicts. third, while some studies attempt to categorize types of knowledge conflict @xcite , systematic analysis distinguishing between forms, such as single-hop vs. multi-hop conflicts, is still lacking. finally, existing benchmarks are largely concerned with exploring conflicts between parametric and external knowledge, while conflicts among multiple input documents remain underexplored @xcite . 1 to alleviate these issues, we propose a framework for constructing a benchmark targeting intercontext knowledge conflict. it leverages knowledge graphs (kgs) as the primary source, from which subgraphs are extracted to serve as the basis for distinct knowledge chunks. these subgraphs are then perturbed by modifying nodes and edges to introduce conflicts. finally, both original and altered graphs are converted into corresponding text passages using kg-to-text generation algorithms. by design, the proposed framework offers several advantages. leveraging the relational structure of kgs enables greater diversity, complexity, and control in inducing conflicts within documents. moreover, compared to text-based strategies, our approach improves interpretability and supports structured analysis by representing conflicting entities and relations in graph form (see figure 1 ). lastly, we conduct extensive analyses using magic (a multi-hop and graph-based benchmark for inter-context conflicts), a novel dataset constructed through our framework. 2 magic features complex inter-context conflict patterns, including simultaneous and multi-hop cases rarely seen in existing benchmarks. experiments on magic yield key insights into how llms perceive knowledge conflicts: (1) most models remain imperfect at detecting conflicts, especially when multi-hop reasoning is required; and (2) even when contradictions are detected, models often fail to localize the exact point of conflict. 1 knowledge conflict is commonly studied in three setups @xcite : (1) context-memory conflict, involving inconsistencies between parametric and external knowledge; (2) inter-context conflict, where contradictions exist among multiple input documents; and (3) intra-memory conflict, where inconsistent responses reflect variation in an llm's training data. in this work, we focus on inter-context conflict. ## magic: multi-hop and graph-based benchmark for inter-context conflicts in this section, we introduce a new framework for constructing an inter-context knowledge conflict detection dataset through the collaboration of kgs and llms. compared to existing benchmarks, the proposed approach offers several advantages: (1) rather than using qa datasets as the source, we adopt kgs, enabling broader domain coverage and a richer range of conflict forms, including multihop cases. (2) it combines the strengths of both manual and automated strategies by incorporating a human-in-the-loop process during llm-based generation, ensuring both high quality and scalability. ## experimental setups with the magic benchmark as a foundation, we conduct experiments to examine how llms handle inter-context knowledge conflicts. we evaluate various open-source and proprietary llms without task-specific training, instead prompting them to identify potential contradictions. in the following, we outline the llms, datasets, prompting strategies, and metrics used in our experiments. datasets alongside magic, we employ two existing benchmarks to highlight its strengths. remarkably, the scale of magic surpasses that of existing benchmarks by a significant margin, offering a richer and more challenging resource for evaluating inter-context conflict detection. prompting strategy prior work @xcite typically frames the task as a binary classification problem, relying on minimal prompts (see appendix b). in contrast, we introduce a stepwise prompting strategy (refer to figure 17 for details) designed to more fully probe the capabilities of llms for inter-context conflict detection. metrics to account for the stochasticity of llms, all models perform three independent inference runs. we use two metrics for fine-grained evaluation, with scores averaged across all instances in each dataset. these metrics are manually computed by participating researchers, as automatic methods-such as llm-as-a-judge-remain insufficiently reliable for this task. 13 lower scores indicate higher complexity. we observe that models struggle more with pinpointing the exact location of a conflict than with detecting its presence. the lowest value in each row is shown in bold. ## experimental results the main experimental results are shown in overall results llms tested on magic consistently show lower id and loc scores compared to those on econ and wikicontradict, with average scores decreasing by up to 10% and 17%, respectively. this indicates that models struggle more to identify conflicts in our dataset, and even when they do, they have difficulty pinpointing the exact portions where the conflict occurs. id scores per llm loc scores per llm performance by conflict types figure 7 presents the average performance of all llms across four conflict complexity settings: single-hop vs. multihop and 1-conflict vs. n-conflict, as in section 3.2. detailed results are provided in appendix d.1. single-hop conflicts, often involving entity or relation substitutions, are relatively easier, with models performing well in both identification and localization. in contrast, multi-hop conflicts introduce greater complexity, as contradictions become more indirect, leading to lower id and loc scores. localization is especially difficult in multi-hop cases, as conflicts often span multiple locations. meanwhile, a higher number of conflicts reflects a stronger contradiction between the two contexts, making conflict detection relatively easier for models. in other words, while more conflicts facilitate identification, they also complicate precise localization. when multiple conflicts occur, pinpointing all the individual conflicting sentences becomes more difficult, thereby leading to lower loc scores. a similar trend has also been observed in econ in appendix e supports this finding. ## further analysis and discussion taxonomy-based analysis to systematically analyze conflict patterns, we apply our proposed conflict typology to econ and wikicontradict. representing these datasets as graphs allows us to highlight their characteristics. a key challenge is the lack of predefined ontologies or domain structures, which hinders the use of traditional ontology-based methods @xcite . to address this, we use langchain @xcite to construct reliable, schema-free kgs that support structured conflict representation. figure 3 shows that 1-single-hop conflicts-the typically easiest case-are the most prevalent in prior datasets, accounting for 78% in econ and 76% in wikicontradict. in contrast, magic exhibits a balanced distribution across conflict types, with substantial proportions of 1-multi-hop (28%) and n-multi-hop (27%), underscoring its robustness as a benchmark. length-based analysis in conflict detection, a reasonable hypothesis is that context length positively correlates with dataset difficulty, as longer documents tend to involve more complex linguistic structures. to validate this, we present lengthoriented anlysis in figure 9 . the left panel visualizes the total context length distribution among the three datasets, with magic containing more longcontext examples. we suspect magic's multi-hop and description-rich design contributes to this. further, we group magic into four bins based on context length and report their respective performance in the right panel of figure 9 . to balance group sizes, we use quantile-based binning: q1 contains the shortest contexts, and q4 the longest. as context length increases, both scores decline, with loc dropping more sharply-indicating the growing difficulty of pinpointing conflicting spans in longer inputs. comparison with existing kc datasets in addition to the results discussed in §5, we compare magic with existing datasets by focusing on the 1-single-hop type-the only category shared across 55.61 59.22 78.57 83.61 82.52 88.1 gpt-4o-mini 34.42 45.63 50.6 73.83 78.79 81.41 magic wikicontradict econ llama 3.1 70b binary multi-step (ours) figure 12: comparison of id scores for binary and multi-step prompts, tested on two models. we further conduct a comparative study between the challenging subset of wikicontradict (implicit) and the corresponding subset from our dataset (multi-hop in magic). 17 figure 11 reports that magic proves more difficult for llms than wi-kicontradict. magic yields an id score up to 9% lower and a loc score up to 16% lower compared to wikicontradict. this highlights magic's intrinsic complexity, setting it apart from existing datasets. impact of prompts while prior work @xcite typically employs binary (yes/no) prompts for conflict detection-often oversimplifying the task-we use a multi-step technique that helps improve performance. to compare the effectiveness of these two distinct prompting strategies for conflict detection, we conduct experiments, with results reported in figure 12 . across all cases, the results show that our multistep prompting approach outperforms the naïve prompt, achieving improvement up to 39.41%. this implies that although we employ a method superior to those commonly used in the literature, there remains substantial room for addressing the challenges posed by magic. ## conclusion we propose a kg-based benchmark, magic, for inter-context knowledge conflict detection with greater diversity and complexity. experimental results reveal the strengths and limitations of llms in handling knowledge conflicts. despite recent progress, llms continue to struggle with conflict detection in complex cases, e.g., those requiring multi-hop reasoning. as a future direction, we aim to develop an optimized method to help models overcome these limitations.