Evaluating LLM s’ Assessment of Mixed-Context Hallucination Through the Lens of Summarization (2025)

Abstract: With the rapid development of large language models (LLMs), LLM-as-a-judge has emerged as a widely adopted approach for text quality evaluation, including hallucination evaluation. While previous studies have focused exclusively on single-context evaluation (e.g., discourse faithfulness or world factuality), real-world hallucinations typically involve mixed contexts, which remains inadequately evaluated. In this study, we use summarization as a representative task to comprehensively evaluate LLMs’ capability in detecting mixed-context hallucinations, specifically distinguishing between factual and non-factual hallucinations. Through extensive experiments across direct generation and retrieval-based models of varying scales, our main observations are: (1) LLMs’ intrinsic knowledge introduces inherent biases in hallucination evaluation; (2) These biases particularly impact the detection of factual hallucinations, yielding a significant performance bottleneck; and (3) the fundamental challenge lies in effective knowledge utilization, balancing between LLMs’ intrinsic knowledge and external context for accurate mixed-context hallucination evaluation.

## introduction large language models (llms) generate coherent text and follow instructions across diverse tasks @xcite . however, a critical challenge in scaling llm applications is hallucination, where the generated content lacks factual grounding @xcite or deviates from the intended discourse context @xcite . this issue makes hallucination evaluation essential for improving llm reliability and developing more robust models @xcite @xcite . recent studies have explored the 1 code and data available at https://github.com/ cece00/fhsumbench . source … the woman was arrested by detectives investigating the child's death following an incident in belfast in march. she was charged with murder and then held under the mental health act… a woman charged with the murder of a girl has been remanded in custody at the royal, victoria hospital in belfast. a woman charged with the murder of a girl has been remanded in custody at the royal, victoria hospital in belfast, a city in america. ## fhsumbench to evaluate models' ability to detect mixed-context hallucinations in summaries, an appropriate test set is essential. to the best of our knowledge, the xent dataset @xcite is the only one providing factual hallucination annotations for summaries. however, among the 240 samples in the test set, only 29% of entities are annotated as factual or non-factual hallucinations, which is insufficient for assessing llms. another related dataset is m-xsum 3 @xcite , which is also built on the xsum dataset and includes annotations for separate labels: faithfulness and factuality. the faithfulness annotations classify hallucinations as either intrinsic or extrinsic, while the factuality annotations are based on world knowledge. after filtering out samples with inconsistent annotations, m-xsum remains an imbalanced dataset with 92% non-factual hallucination samples, as shown in figure 2. therefore, to obtain more diverse evaluation data, we propose an automated pipeline capable of accurately and effectively constructing both factual and non-factual hallucinations. using this pipeline, we construct the factual hallucination in summarization benchmark (fhsumbench) to evaluate the performance of existing factuality and faithfulness evaluators, as well as llm-as-ajudge approaches. the core component of this pipeline is fact injection: injecting either factual or non-factual information into correct summaries. we use xent @xcite and factcollect @xcite as our seed datasets, which provide annotated correct summaries, defined as being both faithful and factual. 4 the first step of the pipeline is to obtain correct summaries. for xent dataset, we remove spans annotated as "non-factual hallucinations," while for the factcollect dataset, we use summaries labeled as "correct." the second step in-3 also referred to as xsumfaith. we use the name of m-xsum to prevent potential misunderstanding. we use both its faithfulness and factuality annotations. 4 the relationship between fhsumbench and previous summarization datasets can be seen in as a result, fhsumbench contains 1,336 samples, evenly distributed across three categories of summaries, providing a balanced dataset for comprehensive evaluation. this approach is more efficient than manual annotation, as it does not need experts or extensive time. additionally, it is more controllable than data synthesis by llms, as it relies on precise descriptions for each entity. while preserving these advantages, this benchmark can effectively evaluate the ability of current llms and evaluators to assess factual hallucinations. ## llm-based evaluators on mixed-context hallucination in this section, we describe widely used llmbased evaluators for hallucinations. based on whether they retrieve knowledge from external sources, we categorize the evaluators into two types: direct generation evaluators ( §4.1) and retrieval-based evaluators ( §4.2). we will evaluate the performance of these approaches by analyz-ing their effectiveness in assessing mixed-context hallucination. ## experiment and result this section presents the experiment settings and offers a comprehensive analysis of llm-based evaluators' performance in detecting mixed-context hallucinations. through carefully designed experiments, we investigate our research questions while addressing critical challenges in hallucination evaluation using llms, while also proposing potential solutions. ## case study we choose the representative larger llm, gpt-4o, as the case study model to investigate the reasons for the errors in hallucination detection (examples available in appendix c.2). intrinsic knowledge bias of llm judge since the cot method exhibits the lowest score in detecting factual hallucinations using gpt-4o among direction generation llm methods, we selected it to investigate the causes of error cases. we randomly selected and analyzed 30 error cases from the cot outputs of gpt-4o. among these cases, 63.3% were misclassified as "no hallucination" due to the llm either relying on its intrinsic knowledge for reasoning or inherently assuming the injected information to be faithful. additionally, 30% of the samples were mistakenly categorized as "nonfactual hallucination" due to insufficient evidence being provided. the remaining 6.7% of errors resulted from issues in the reasoning process. these cases include instances where unfaithful content was correctly flagged as hallucination during the cot process but was later misinterpreted as faithful in the final judgment. conversely, some cases involved injected factual knowledge being accurately recognized during reasoning but ultimately misjudged as false in the factuality assessment. this indicates that, despite well-defined evaluation criteria, the model still struggles to some extent to distinguish between faithfulness and factuality. the influence of retrieval since the intrinsic knowledge of the llm significantly influences its decision-making, retrieval-based experiments demonstrate that it can partially rectify erroneous classifications. to further investigate this, we examined the results of the cr and rr methods on gpt-4o using the aforementioned samples. overall, evidence retrieval helps mitigate intrinsic knowl-edge bias. among the analyzed error cases, the cr method successfully corrects 47.3% of samples misclassified as "no hallucination", while the rr method achieves a correction rate of 57.9%. however, the effectiveness of this approach still depends on the appropriateness of the knowledge base and the accuracy of retrieval. although the rr method improves performance, there remains room for further enhancement. ## conclusion and future direction in this paper, we systematically evaluate the ability of llms to assess hallucinations in mixed-context scenarios within summarization. the evaluation covers both open-source and closed-source models across various scales. specifically, the distinction between factual and non-factual hallucinations is analyzed across different models and methods. our findings reveal that llms still struggle to differentiate between faithfulness and factuality, highlighting the benefit of explicitly incorporating external knowledge to aid judgment. while larger models generally exhibit better performance, the trend suggests the existence of a performance plateau. drawing insights from our experimental findings, several directions deserve further investigation: reducing intrinsic knowledge bias our experiments reveal that intrinsic knowledge bias significantly impacts llm evaluation results, particularly for larger llms. while we have conducted preliminary investigations, deeper exploration is needed to understand fundamental aspects such as training data updates, internal information flow, and mechanisms for integrating internal and external knowledge. effective knowledge integration mixedcontext hallucination evaluation requires the fusion of multiple sources of information for decision-making. our experiments demonstrate that specific information integration methods enable smaller llms to achieve performance comparable to larger llms. this highlights the urgent need for research into more effective information fusion approaches. more flexible scenarios while this work primarily investigates mixed-context hallucination evaluation through the lens of summarization tasks, real-world scenarios often present greater complexity and flexibility. these include extended dialogues and more intricate task workflows that ex-tend beyond knowledge-based hallucination. further research is needed to address these diverse applications.