CHARPEVAL : Benchmarking Large Language Models’ Contextual Reasoning in Knowledge-Grounded Dialogue (2025)

Abstract: This paper presents CHARPEVAL, a challenging benchmark specifically designed to evaluate the ability of Large Language Models (LLMs) to perform contextualized reasoning in knowledge-grounded dialogue scenarios. The task involves selecting the correct response from 6 options, including 5 manually crafted distractors, given the conversation history. Extensive benchmarking experiments with a diverse set of state-of-the-art open-weight LLMs show poor performance on CHARPEVAL due to their inability to effectively reason over discontinuous chunks of text across the input. Our analysis reveals systematic error patterns across models with different properties, highlighting the need to improve LLMs beyond simply scaling-up data and compute. CHARPEVAL is publicly available at https://huggingface.co/datasets/huawei-noah/CHARP.

## introduction there have been ongoing efforts to develop new benchmarks that challenge the rapid advancements of llms @xcite @xcite , by introducing more complex @xcite , reasoning-intensive @xcite , and domain-specific @xcite tasks. knowledge-grounded dialogue @xcite @xcite is a task that requires responding to user queries while staying faithful to the provided knowledge. in this paper, we propose charpeval, a challenging benchmark that evaluates llms ability to identify discontinuous spans of relevant information and perform complex reasoning on them. charpeval is build on top of charp @xcite , which is a testbed designed for probing the conversation history awareness of knowledge-grounded dialog systems. specifically, we expand charp annotations with 5 hand-crafted challenging distractor responses, each designed to test a specific potential limitation in llms contextual reasoning. as shown in figure 1 , charpeval not only presents a serious challenge to modern llms, but also enables a self-contained and reproducible evaluation process by eliminating the need for external evaluation tools. therefore, charpeval is more tailored to llm evaluation compared to existing dialog benchmarks -including charp itself-by addressing issues such as misalignment with true human performance @xcite , reproducibility challenges with api deprecated @xcite , outdated judge llms @xcite , and costly human expertise. these issues are common in approaches relying on lexical overlap/semantic similarity scorers @xcite , closed-source llm apis @xcite , llm-as-judge @xcite , or human expert as evaluation methods, respectively. in contrast to our charpeval, which is a human-generated benchmark emphasizing contextual reasoning in dialog, the recently proposed dialogbench @xcite is automatically generated by @xcite and focuses on evaluating llm instruction-following in dialog tasks. a comprehensive evaluation using a diverse set of recent open-weight llms @xcite reveals that these models perform poorly on charpeval, with the best models scoring around 50%. our extensive analysis reveals systematic error patterns, regardless of the model size, tuning process, or organization behind the model. it also highlights an inherent bias that favors irrelevant responses, with highly frequent text segments encountered during pretraining being favored over task-relevant ones. i like to bake bread in my spare time. do you? bots cannot bake, but did you know that this method of cooking can use hot ashes or hot stones? i didn't know that. i also like cooking different things like cake, stews, and broths. have you ever tried baking at a community center activity? they are well-equipped. well, you need flour, water, yeast and salt to make bread. ## conclusion we introduce charpeval, a benchmark that is both challenging and has a reproducible evaluation method to test modern llms' contextual reasoning abilities in knowledge-grounded dialogue tasks. we hope that our charpeval can guide the development of future llm upgrades to address the limitations identified in this paper.