C - PMI : Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation (2023)

Abstract: Existing reference-free turn-level evaluation metrics for chatbots inadequately capture the interaction between the user and the system. Consequently, they often correlate poorly with human evaluations. To address this issue, we propose a novel model-agnostic approach that leverages Conditional Pointwise Mutual Information (C-PMI) to measure the turn-level interaction between the system and the user based on a given evaluation dimension. Experimental results on the widely used FED dialogue evaluation dataset demonstrate that our approach significantly improves the correlation with human judgment compared with existing evaluation systems. By replacing the negative log-likelihood-based scorer with our proposed C-PMI scorer, we achieve a relative 60.5% higher Spearman correlation on average for the FED evaluation metric. Our code is publicly available at https://github.com/renll/C-PMI .

## introduction evaluating dialogues is a multi-faceted task that demands consideration of diverse dimensions, which distinguishes it from the evaluation of task-oriented dialogue systems. traditional n-gram-based evaluation metrics, such as rouge @xcite and bleu @xcite , demonstrate weak correlation with human-annotated judgments due to the broad spectrum of potential responses in dialogues. as a result, researchers often resort to human evaluations to ascertain the quality and effectiveness of their generated system responses, especially for knowledge-guided dialog systems @xcite @xcite . substantial research has been conducted on automatic evaluation metrics for dialogue @xcite . these metrics can be classified into reference-based and reference-free categories. reference-based metrics, which depend on com-paring the system response to a human-written reference response, are generally inadequate for dialogue evaluation due to the inherent one-to-many nature of dialogues. the reference-free metric instead uses a computational model to generate a score for the system response with a given context. early models predominantly focus on a limited set of general features of dialogue generation quality, such as context coherency and fluency. subsequent evaluation metrics investigated additional dimensions, such as usl-h @xcite , which combines relevance evaluation with factto-response selection. holistic-eval @xcite assesses content coherence, language fluency, self-consistency, and semantic appropriateness. d-score @xcite and predictive engage @xcite introduce response diversity and engagement scores. the recent fed @xcite ) metric encompasses 18 turn-level and dialogue-level metrics, including interestingness, likeability, and response flexibility. however, all of these methods do not model the interaction between the turn-level response and the dialogue history and regard them as an integrated context for score calculation. in this paper, we focus on directly modeling user-system interactions through the lens of mutual information @xcite and propose a novel scorer based on conditional pointwise mutual information (c-pmi), which effectively captures the turn-level interactions between the system and user with respect to a given hypothesis. we demonstrate that our approach results in a reference-free, training-free, automatic turn-level dialogue evaluation that significantly outperforms state-of-the-art methods with a comparable number of model parameters. our contributions in this work are three-fold: ## related work developing automatic evaluation metrics for dialog is challenging for several reasons: 1) dialogues often have a one-to-many nature, rendering wordoverlap metrics ineffective. to address this issue, metrics should be designed to be reference-free. 2) given the limitless nature of conversation topics in open-domain dialogues, the dialogue evaluation metrics are expected to understand the semantic meaning of both the dialogue context and the generated responses. this necessitates a metric that can leverage pre-trained large language models and self-supervised training objectives. 3) training dialogue evaluation metrics solely on labeled data can significantly restrict the metric's range, risking over-fitting to the training data in terms of conversation topics and response generation models. as such, recent metrics have started to incorporate selfsupervised training objectives designed to capture various aspects of a dialogue, such as relevance, fluency, and interestingness among others. given the aforementioned challenges, large language models have become an integral part of dialogue evaluation. dialogrpt @xcite employs an extended gpt-2 model trained on 147 million conversation-like interactions from reddit. usr @xcite ) is an unsupervised, reference-free tool that takes advantage of the roberta @xcite model. usr employs a dialogue retrieval metric for assessing dialogue, where the metric is trained to differentiate between a ground truth response and a randomly sampled response. the fed metric @xcite utilizes dialogpt @xcite due to its capacity for capturing knowledge, specifically within the context of conversations. it ignores the interaction between the user and the system and consider the dialogue history and the system response as an integral context, while our method explicitly captures such interaction through conditional mutual information. ## conditional pointwise mutual information based turn-level metric for each of the dialogue turn t, our pointwise mutual information (pmi) based metric is considering the dependencies between the following three random variables: the full dialogue history r @xmath0, x 0 , u 1 , x 1 , ..., u t } ∼ r (where u t is the user utterance), the system response x t ∼ x and a hypothesis h ∼ h. ideally, we want to know how much correlation between the dialogue history and the system response causes the hypothesis to be a plausible entailment of the combined history, {r t , x t }. we measure such correlation by calculating the conditional mutual information (cmi) between the response and the history with a given hypothesis, i.e., intuitively, if i(r, x|h) is large, the hypothesis is less likely to be caused by the interaction (i.e., the shared information) between r and x. since sampling the history on a turn-by-turn basis needs exponentially increasing computation, an accurate estimation of the cmi between these random variables is intractable. therefore, we propose to measure the cmi by calculating the pointwise mutual information contained between the observed dialogue history and the system response when the hypothesis is appended to the combined history. formally, we define our conditional pmi (c-pmi) score between the observed dialogue history, the system response, and the hypothesis as follows, in practice, we estimate the probability of each sequence using the averaged log-likelihood (ll) obtained from a language model p θ , i.e., for integrating our scorer with the existing evaluation system such as fed, we simply replace its nll scoring function with our c-pmi scorer, and follow the original pipeline to get the final score for each of the data samples. ## results & analysis the results indicate that the proposed c-pmibased turn-level metrics are capable of providing a more accurate evaluation of dialogue system responses compared to existing state-of-the-art methods. moreover, our metric is unreferenced and training-free, which makes it particularly suitable for practical applications, such as responses selec-tion and re-ranking. ## conclusion in this paper, we introduce a novel dialogue evaluation metric based on conditional pointwise mutual information (c-pmi) that captures turn-level interactions between the system and user across various evaluation dimensions. the proposed metric is reference-free and training-free, outperforming state-of-the-art methods with a comparable number of model parameters. for turn-level dialogue evaluations, our experimental results demonstrate that this metric can serve as a generalized alternative to the negative log-likelihood scorer for multi-dimensional evaluation metrics. we plan to extend our approach to other dialogue evaluation methods and explore its applicability to general text generation problems. we are also interested to see if our measure can improve the factual consistency evaluation for document-grounded dialogue or conversational question answering. additionally, we will investigate incorporating our c-pmi-based metric into the fine-tuning process of llms.