Relevance in Dialogue: Is Less More? An Empirical Comparison of Existing Metrics, and a Novel Simple Metric (2022)

Abstract: In this work, we evaluate various existing dialogue relevance metrics, find strong dependency on the dataset, often with poor correlation with human scores of relevance, and propose modifications to reduce data requirements and domain sensitivity while improving correlation. Our proposed metric achieves state-of-the-art performance on the HUMOD dataset while reducing measured sensitivity to dataset by 37%-66%. We achieve this without fine-tuning a pretrained language model, and using only 3,750 unannotated human dialogues and a single negative example. Despite these limitations, we demonstrate competitive performance on four datasets from different domains. Our code, including our metric and experiments, is open sourced.

## introduction the automatic evaluation of generative dialogue systems remains an important open problem, with potential applications from tourism @xcite to medicine @xcite . in recent years, there has been increased focus on interpretable approaches @xcite often through combining various sub-metrics, each for a specific aspect of dialogue @xcite @xcite . one of these key aspects is "relevance" (sometimes called "context coherence"), commonly defined as whether " @xcite esponses are on-topic with the immediate dialogue history" @xcite . these interpretable approaches have motivated measures of dialogue relevance that are not reliant on expensive human annotations. such measures have appeared in many recent papers on dialogue evaluation, including usr @xcite , usl-h @xcite , and others @xcite . additionally, dialogue relevance has been used directly in training dialogue models @xcite . despite this work, comparison between these approaches has been limited. aggravating this problem is that authors often collect human annotations on their own datasets with varying amounts and types of non-human responses. consequently, direct comparisons are not possible. it is known that metrics of dialogue quality often perform poorly on new test sets of quality ratings @xcite , but it remains an open question whether poor generalization also plagues the much simpler dialogue relevance task. we address this problem by evaluating and comparing six prior approaches on four publicly available datasets of dialogue annotated with human ratings of relevance. we find poor correlation with human ratings across various methods, with high sensitivity to dataset. based on our observations, we propose a simple metric of logistic regression trained on pretrained bert nsp features @xcite , using "i don't know." as the only negative example. with this metric, we achieve state-of-the-art correlation on the humod dataset @xcite . we release our metric and evaluation code to encourage comparable results in future research. our primary contributions are: (i) empirical evidence that current dialogue relevance metrics for english are sensitive to dataset, and often have poor correlation with human ratings, (ii) a simple relevance metric that exhibits good correlation and reduced domain sensitivity, and (iii) the counterintuitive result that a single negative example can be equally effective as random negative sampling. ## prior metrics prior metrics of relevance in dialogue can generally be divided into more traditional approaches that are token-based, and more current approaches based on large pretrained models. these metrics are given the context (i.e., the two-person conversation up to a given point in time), as well as a response (i.e., the next speaker's response, also known as the 'next turn' in the conversation). from these, they produce a measure of the response's relevance to the context. the ground-truth response (i.e., the 'gold response') may or may not be available. ## datasets used for analysis a literature review reveals that many of these methods have never been evaluated on the same datasets. as such, it is unclear both how these approaches compare, and how well they generalize to new data. for this reason, we consider four publicly available english datasets of both human and synthetic dialogue with human relevance annotations. all datasets are annotated with likert ratings of relevance from multiple reviewers; following @xcite , we average these ratings over all reviewers. due to variations in data collection procedures, as well as anchoring effects @xcite , likert ratings from different datasets may not be directly comparable. consequently, we keep the datasets separate. this also allows us to observe generalization across datasets. altogether, our selected datasets cover a wide variety of responses, including human, lstm, transformer, meena @xcite , and mitsuku 2 generated responses, and random distractors. see ## evaluating prior metrics for each of the aforementioned datasets, we evaluate the following relevance metrics: to prevent an unfair fitting to any specific dialogue model, and to better reflect the evaluation of a new dialogue model, only human responses were used at train time. all hyperparameters were left at their recommended values. nup-bert performance is averaged over 3 runs. note that we also evaluate grade @xcite and dyna-eval @xcite ; however these do not measure relevance, but rather dialogue coherence: "whether a piece of text is in a consistent and logical manner, as opposed to a random collection of sentences" @xcite . as relevance is a major aspect of dialogue coherence, we include these baselines for completeness. as both metrics are graph neural networks intended for larger train sets, we use checkpoints provided by the authors. grade is trained on dailydialogue @xcite , and dynaeval on empathetic dialogue @xcite . both are trained with negative sampling, with grade constructing more challenging negative samples. a summary of the authors' stated purpose for each metric can be found in the appendix c. ## idk: a metric for dialogue relevance based on these results, we propose a number of modifications to the nup-bert metric to produce a novel metric that we call idk ("i don't know"). the architecture is mostly unchanged, however the training procedure and the features used are altered. first, based on the observation that the amount of training data has little impact, we freeze bert features and do not fine-tune to the domain. additionally, whereas the nup-bert baseline uses max-pooled bert word embeddings, we use the pre-trained next sentence prediction (nsp) features: "(classification token) further processed by a linear layer and a tanh activation function [...] trained from the next sentence prediction (classification) objective during pre-training" foot_2 . second, to improve generalization and reduce variation in training (particularly important as the practitioner typically has no annotated relevance data), and operating on the assumption that relevance is captured by a few key dimensions of the nup features, we add l1 regularization to our regression weights (λ = 1). note that experiments with l2 regularization yielded similar validation set performance (see appendix, third, in place of random sampling we use a fixed negative sample, "i don't know". this allows us to train the model on less data. additionally, we simplify the model, using logistic regression in place of 2-class softmax. we train for 2 epochs using bce loss -the same as the nup-bert baseline. we use the adam optimizer @xcite with an initial learning rate of 0.001, and batch size 6. compared to nup-bert, our proposed metric provides strong improvement on the humod dataset and equivalent or stronger performance on usr-tc and fed, at a cost of performance on p-dd. in particular, idk (tc-s) performance on the fed datasets is considerably stronger than nup-bert (tc-s). as the performance drop on p-dd is less than the performance gain on humod, and as humod is human data rather than lstm data, we consider this tradeoff to be a net benefit. compared to grade in particular, we have reduced performance on p-dd, equivalent performance on humod, and stronger performance on usr-tc and fed (in particular, correlation on the usr-tc dataset is non-zero). it is worth noting that, in general, our approach does not out-perform the baselines in all cases -only the majority of cases. as such, when annotated human data is not available for testing, it would appear that our approach is the preferred choice. our metric is also preferable, as it is less sensitive to domain. to numerically demonstrate this, we measure the domain sensitivity of the evaluated metrics as the ratio of best spearman's correlation to worst spearman's correlation -this value should be positive (i.e., there is no dataset where the metric becomes negatively correlated), and as close to 1 as possible (i.e., there is no difference in performance). looking at prior metric ratio fed-correct -0.7 fed-relevant -0.7 norm-prob -2.7 cos-nsp-bert -7.5 cos-ft -13 grade ∞ dyna-eval ∞ nup-bert (tc-s) 11.6 nup-bert (tc) 10.3 cos-max-bert 6.7 nup-bert (h) 6.2 idk (h) ## related work in addition to the prior metrics already discussed, the area of dialogue relevance is both motivated by, and jointly developed with, the problem of automatic dialogue evaluation. as relevance is a major component of good dialogue, there is a bidirectional flow of innovations. the nup-bert relevance metric is very similar to bert-ruber @xcite ; both train a small mlp to perform the next-utterance-prediction task based on aggregated bert features. both of these share a heritage with earlier self-supervised methods, such as adversarial approaches to dialogue evaluation that train a classifier to distinguish human from generated samples @xcite . another example of shared development is the use of word-overlap metrics such as bleu @xcite and rouge @xcite that have been imported wholesale into both dialogue relevance and overall quality from the fields of machinetranslation and summarization, respectively. simultaneously, metrics of dialogue evaluation have been motivated by dialogue relevance. there is a long history of evaluating dialogue models on specific aspects; finch and choi (2020) performed a meta-analysis of prior work, and proposed dimensions of: grammaticality, relevance, informativeness, emotional understanding, engagingness, consistency, proactivity, and satisfaction. new approaches to dialogue evaluation have emerged from this body of work, seeking to aggregate individual measures of various dimensions of dialogue, often including relevance @xcite @xcite . these approaches also share heritage with earlier ensemble measures of dialogue evaluation such as ruber @xcite -although in the case of ruber, it combined a referenced and unreferenced metric rather than separate aspects. metrics of dialogue relevance and quality also share common problems such as the diversity of valid responses. our findings that existing relevance metrics generalize poorly to new domains is consistent with previous findings about metrics of dialogue quality @xcite . thus, our work suggests that this challenge extends to the subproblem of dialogue relevance as well. at the same time, it must be remembered that measuring holistic dialogue quality is a very different task from measuring dialogue relevance -it is well established that aspects of dialogue such as fluency, and interestingness are major components of quality @xcite , and these should have no impact on relevance. with respect to prior work comparing relevance metrics, we are aware of only one tangential work. @xcite performed a comparison of various metrics of dialogue quality; within this work they dedicated three paragraphs to a brief comparison of how these quality metrics performed at predicting various dialogue qualities, including relevance. they reported results on only two of the datasets we used (p-dd and fed). interestingly, the authors found that the fed metric performs well on p-dd (reporting a spearman's correlation of 0.507), however our results demonstrate that the components of fed that are meant to measure relevance (i.e. fed-rel and fed-cor) are significantly negatively correlated with human relevance scores. additionally, as @xcite focus on quality, they do not compare performance between the two relevance datasets. instead they compare performance on quality against performance on relevance, and use the discrepancy to conclude that measuring relevance alone (as done by norm-prob) is insufficient to determine quality. although we agree that relevance alone is insufficient for dialogue quality evaluation, our work provides a richer understanding. our finding that norm-prob performs poorly across a range of relevance datasets suggests that the poor performance of norm-prob in the quality-prediction task is also caused by the poor relevance generalization in addition to the insufficiency of relevance to measure overall quality. ## discussion our experiments demonstrate that several published measures of dialogue relevance have poor, or even negative, correlation when evaluated on new datasets of dialogue relevance, suggesting overfitting to either model or domain. as such, it is clear that further research into new measures of dialogue relevance is required, and that care must be taken in their evaluation to compare against a number of different models in a number of domains. furthermore, it is also clear that for the current practitioner who requires a measure of relevance, there are no guarantees that current methods will perform well on a given domain. as such, it is wise to collect a validation dataset of human-annotated relevance data for use in selecting a relevance metric. if this is not possible, then our metric, idk, appears to be the best option -achieving both good correlation and the lowest domain sensitivity, even when trained on different domains. furthermore, when training data is scarce, our results suggest that the use of strong regularization allows for the use of a single negative example, "i don't know", in the place of randomly sampled negative samples. if that is still too data intensive, then our results suggest that our metric is fairly agnostic to the domain of the training data; therefore training data can be used from a different dialogue domain in place of the domain of interest. having said this, it is clear that further research into what exactly these metrics are measuring, and why they fail to generalize, is merited. the results are often counter-intuitive; our demonstration that 99% of the bert nsp features can be safely discarded is just one striking example. similarly, although our empirical results suggest that use of a single negative example generalizes across domains, there is no compelling theoretical reason why this should be so. more generally, all the metrics outlined are complex, dependent on large corpora, and created without ground truth annotations. as a result, they are all dependent on either surrogate tasks (i.e., nup), or unsupervised learning (e.g., fasttext embeddings). consequently, it is especially difficult to conclude what exactly these metrics are measuring. at present, the only strong justification that these metrics are indeed measuring relevance is good correlation with human judgements -poor generalization across similar domains is not an encouraging result. although the metric outlined is not appropri-ate for final model evaluation (as it risks unfairly favouring dialogue models based on the same pretrained bert, or similar architectures), our aim is to provide a useful metric for rapid prototyping and hyperparameter search. additionally, we hope that our findings on the domain sensitivity of existing metrics will spur further research into both the cause of -and solutions to -this problem. ## conclusion our work demonstrates that several existing metrics of dialogue relevance are problematic as their performance varies wildly between test-domains. we take a first step towards resolving this issue by proposing idk: a simple metric that is less sensitive to test domain and trainable with minimal data. we reduce idk's data requirements through the novel use of a fixed negative example, provide evidence that the underlying bert nsp features are low-dimensional, and propose that this fact (combined with idk's lack of feature learning) allows for the counter-intuitive use of a single negative example. beyond this, we call for better evaluation of future relevance metrics, and thus release our code for processing four diverse, publicly available, relevance-annotated data sets.