CHARP : Conversation History A wa R eness Probing for Knowledge-grounded Dialogue Systems (2024)

Abstract: In this work, we dive deep into one of the popular knowledge-grounded dialogue benchmarks that focus on faithfulness, FaithDial. We show that a significant portion of the FaithDial data contains annotation artifacts, which may bias models towards completely ignoring the conversation history. We therefore introduce CHARP, a testbed, designed for evaluating supposedly non-hallucinatory models trained on the FaithDial dataset. Our extensive analysis reveals that models primarily exhibit poor performance on CHARP due to their inability to effectively attend to and reason over the conversation history. Furthermore, the evaluation methods of FaithDial fail to capture these shortcomings, neglecting the conversational history. Our findings indicate that there is substantial room for contribution in both dataset creation and hallucination evaluation for knowledge-grounded dialogue, and that CHARP can serve as a tool for monitoring the progress in this particular research area. Data, models, and source code will be publicly available upon acceptance.

## introduction despite the success of general purpose large language models (llms) @xcite , the utility of the generated texts rests in its relevance and knowledge grounding. the task of information seeking dialogue @xcite is a touchstone for knowledge grounded generation. the task evaluates a system's ability to respond to user queries while it remains faithful to the knowledge. a system response not adhering to this would be deemed unfaithful. this topic has received considerable attention resulting in several diagnostics and mitigation techniques for texts that lack knowledge france has approximately 34,000 bakeries, and in germany, where bread is also an important part of the cuisine, there are about 10,000 bakeries. ## related work constructing diagnostic sets with curated adversarial or counterfactual examples, has been shown to be an effective approach across nlp tasks to capture such artifacts that standard evaluation sets fail to detect. for instance, hans @xcite , fever @xcite , paws @xcite , core @xcite , nrb @xcite , and nature @xcite datasets are vital in identifying biases of models solving tasks like textual entailment, fact verification, paraphrase identification, relation extraction, named entity recognition, and intent detection respectively. studies @xcite @xcite on the recent trend of largescale pretraining @xcite show that data quality affects the models in inheriting biases from the artifacts embedded in data. especially in information seeking dialogues, @xcite show that unfaithfulness to the given knowledge is a dominant type of hallucination. @xcite show that most information-seeking dialogue datasets like @xcite , topicalchat @xcite , and wizard of wikipedia (wow; @xcite ) contain a high ratio of hallucinations, with wow dataset being the least affected. @xcite propose faithdial built through replacing hallucinatory wow annotations with with knowledge faithful responses. @xcite show that the models trained on faithdial show significant reduction in hallucination. further studies such as the one from @xcite recently demonstrated that flan-t5base @xcite can fetch further reduction in hallucination by training on faithdial over t5-base @xcite as reported in @xcite . @xcite also propose elastic weight removal (ewr) hallucination mitigation method to reduce the rate of unfaithfulness response generation. the authors report high bertscore similarity @xcite between the model response and both the ground truth and the provided knowledge suggesting the faithfulness of the response generated. as training an lm to attend to the knowledge could inadvertently result in ignoring the history of turns leading to poor reasoning of the model, we, in this work, propose charp that serves as a diagnostic set to measure this phenomenon. ## charp charp, the proposed diagnostic set, exclusively assess whether information-seeking dialogue systems effectively attend to and use the conversation history. charp is built by modifying examples from the faithdial validation set to ensure maximum domain alignment with faithdial and to minimize annotation costs. that is, we edit faithdial examples to make their response dependent on the conversation history analogously to faithdial's editing of wow annotations to make them hallucination-free. it is important to note that the faithdial validation and test sets are sampled from the same distribution and exhibit similar result patterns. we create two variants of charp: hcharp ( § 4.1) for examples where addressing the last seeker's inquiry requires reasoning over the conversation history, and echarp ( § 4.2), where the last inquiry can be addressed without such reasoning. we annotate 42% of the faithdial validation set (after excluding examples without conversation history) containing 2, 160 examples, split equally between hcharp and echarp. ## analysis 6.1 on faithdial data artifact we conduct ablations on the behavior of models to estimate the resulting errors due to the artifacts in the faithdial training data by comparing the performances of models with and without fine-tuning on faithdial. we use the few-shot learning technique via prompting to ablate for not training with faithdial dataset. specifically, we compare the performances of llama-2-7b model that we tuned on faithdial against the same model with 3 in-context examples (3-shot). in addition, we evaluate the 3-shot performances of chatgpt (openai, 2022) and mixtral (jiang et al., 2024) llms. implementation details of these experiments can be found in appendix b.3, as well as an example of the models' responses in figure 5 . in first, we observe that w1 is the dominant error category, where models add extra information after addressing the user inquiry. we attribute this behavior to the verbose nature of llms, which is challenging to mitigate without further tuning @xcite . second, we notice that the error ratio for w2 is significantly lower, by at least 20%, across configurations compared to fine-tuned models. additionally, we observe near-zero error ratios for w3 and w4, strongly suggesting that models not tuned on faithdial are not affected by their ability to reason over the history. the error trend of llama-2-7b trained on faithdial strongly mimicking the results in third, non-tuned models suffer significantly more from severe hallucinations (w6), a wellknown issue with llms @xcite . however, this issue tends to be mitigated as the models get larger; for instance, w6 drops from 32% in llama-2-7b to 5% in mixtral on hcharp. while the smaller llama-2-7b llm performs worse than its fine-tuned version, the larger models, mixtral and chatgpt, significantly outperform all reported fine-tuned models by at least 30% on c1. despite their high performances, we observe that hcharp remains more challenging than echarp for even the most advanced models, indicating that charp can additionally serve as a measure of reasoning capability for such models. finally, the fact that mixtral outperforms chatgpt in our tasks serves as an additional indicator that high performance is achievable through community-shared, open-source models. ## conclusion in this work, we examine the impact of annotation artifacts on information-seeking dialogue models tuned on faithdial, a well-established, hallucination-free annotation benchmark. we introduce charp, a diagnostic set designed to evaluate the ability of models to reason over the conversation history, while also staying grounded on the knowledge. our analysis with charp reveals a strong correlation between training on faithdial to models' ignoring reasoning over the conversation history. further, proprietary llm apis can be a proxy to human evaluation, and a better hallucination estimator to automatic metrics. in similar vein to @xcite , we note that while it is important to ensure hallucination-free annotations, including examples to cover reasoning over context and other pretraining knowledge is necessary to preserve models' reasoning capabilities.