Emphasising Structured Information: Integrating A bstract M eaning R epresentation into LLM s for Enhanced Open-Domain Dialogue Evaluation (2025)

Abstract: Automatic open-domain dialogue evaluation has attracted increasing attention, yet remains challenging due to the complexity of assessing response appropriateness. Traditional evaluation metrics, typically trained with true positive and randomly selected negative responses, tend to assign higher scores to responses that share greater content similarity with contexts. However, adversarial negative responses, despite possessing high lexical overlap with contexts, can be semantically incongruous. Consequently, existing metrics struggle to evaluate such responses effectively, resulting in low correlations with human judgments. While recent studies have demonstrated the effectiveness of Large Language Models (LLMs) for open-domain dialogue evaluation, they still face challenges in handling adversarial negative examples. We propose a novel evaluation framework that integrates Abstract Meaning Representation (AMR) enhanced domain-specific language models (SLMs) with LLMs. Our SLMs explicitly incorporate AMR graph information through a gating mechanism for enhanced semantic representation learning, while both SLM predictions and AMR knowledge are integrated into LLM prompts for robust evaluation. Extensive experiments on open-domain dialogue evaluation tasks demonstrate the superiority of our method compared to state-of-the-art baselines, particularly in discriminating adversarial negative responses. Our framework achieves strong correlations with human judgments across multiple datasets, establishing a new benchmark for dialogue evaluation. Our code and data are publicly available at https://github.com/Bernard-Yang/SIMAMR.

## introduction open-domain dialogue systems have garnered substantial attention owing to their broad applicability @xcite across various domains, including personal medical assistance and biomedical telecommunications @xcite . traditional evaluation approaches, such as n-gram-based metrics @xcite @xcite and embedding-based metrics @xcite , assess the semantic similarity between response candidates and gold references. these methods correlate poorly with human evaluation due to their limited capacity to incorporate conversational context @xcite . while recent advances in trainable evaluation frameworks @xcite have improved context-response relationship modeling, they face fundamental limitations stemming from their training . these models, typically trained with true positive and randomly sampled negative examples, tend to assess responses primarily through surface-level content similarity. although some approaches have attempted to address this by incorporating adversarial examples @xcite , they either require extensive pre-training on large-scale conversational corpora or demand adaptation to specific datasets, incurring substantial computational overhead. moreover, their exclusive reliance on surfaceform features compromises robustness when evaluating adversarial examples that deviate from the training distribution. the vulnerability to adversarial attacks further compounds this challenge. @xcite demonstrated that even simple synonym substitutions can lead to misclassification in text analysis tasks. for instance, a positive review stating "the characters, cast in impossibly contrived situations, are totally estranged from reality" would be misclassified as negative when minimally modified to "the characters, cast in impossibly engineered circumstances, are fully estranged from reality", despite maintaining semantic equivalence. recent advances in large language models (llms) have shown promise across a variety of tasks @xcite @xcite . however, these models still exhibit suboptimal performance when evaluating adversarial negative responses. to address these limitations, we propose integrating llms with domain-specific language models (slms) enhanced by abstract meaning representation (amr) graph information, specifically aimed at improving evaluation robustness for adversarial examples. amr graphs serve as powerful tools for capturing dialogue system states and providing complementary semantic knowledge @xcite . consider the following example: given the context "would you recommend some places for sightseeing? how about great canyon? is it worth seeing?", and an adversarial negative response "the movie was really good, it was worth watching it", existing metrics might erroneously classify this as positive due to lexical overlap. amr graphs help address this by modeling semantic relationships between concepts (e.g., "worth" and "canyon") through explicit edge relations (e.g., ":mod" and ":arg1"). our approach introduces an amr graphenhanced slm that effectively identifies adversarial negative examples in open-domain dialogue. the framework integrates both the slm's predictions and amr graph information into the llm's prompt, creating a robust automatic evaluator that leverages domain-specific knowledge during inference. the slm architecture comprises two key components: sentence and graph encoders. the sentence encoder processes surface-form knowledge from conversational contexts and responses, while the graph encoder models amr structural information, capturing both conceptual elements and their interrelations. these complementary representations are unified through a sophisticated gating mechanism and optimised via contrastive learning, encouraging alignment between textual and structural features for positive context-response pairs. the final evaluation integrates both the slm's prediction score and amr graph information into the llm's prompt. comprehensive empirical evaluation across three public datasets demonstrates our model's superior performance compared to state-of-the-art baselines, including llm-based methods. our key contributions include: our contributions can be summarised as follows: ## conclusion in this paper, we presents a novel evaluation framework for open-domain dialogue systems that integrates amr graph-enhanced slms with llms. comprehensive experimental results across multiple datasets demonstrate that our method consistently outperforms existing approaches, including state-of-the-art llm-based methods, in the challenging task of open-domain dialogue evaluation.