Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluation (2023)

Abstract: Despite significant research effort in the development of automatic dialogue evaluation metrics, little thought is given to evaluating dialogues other than in English. At the same time, ensuring metrics are invariant to semantically similar responses is also an overlooked topic. In order to achieve the desired properties of robustness and multilinguality for dialogue evaluation metrics, we propose a novel framework that takes advantage of the strengths of current evaluation models with the newly-established paradigm of prompting Large Language Models (LLMs). Empirical results show our framework achieves state of the art results in terms of mean Spearman correlation scores across several benchmarks and ranks first place on both the Robust and Multilingual tasks of the DSTC11 Track 4 “Automatic Evaluation Metrics for Open-Domain Dialogue Systems”, proving the evaluation capabilities of prompted LLMs.

## introduction automatic dialogue evaluation has largely been focused on evaluating select few languages. the main reason for this constraint is the lack of linguistic diversity in dialogue corpora, which leads to a lack of chatbots that cover other languages. as a result, the need for multilingual metrics has also been limited. a possible solution to this issue is to leverage the latest batch of large language models (llms) to synthetically generate multilingual dialogues. some research has already been conducted to study the capabilities of these models @xcite and the consensus appears to be that these models have achieved a proxy of a formal linguistic competence in the most studied languages. that is, its responses follow linguistic conventions and are fluent and grammatical, but they might be inaccurate or even hallucinate @xcite . more importantly, pertaining to dialogue, they also show signs of functional linguistic competence in its responses, i.e., discursive coherence, narrative structure and linguistic knowledge, even if not fully consistent (sometimes they do not consider context or situated information, and fail to adapt to users and domains). irrespective of these models' limitations, it is clear their emergent capabilities allow for the development of chatbots with capabilities vastly beyond what earlier models were able to achieve. yet, an interesting research question lingers: if these models are able to write responses that follow formal and functional linguistics rules, are they also capable of evaluating responses/dialogues in terms of these same rules? prior work has confirmed the language understanding capabilities of instructionbased llms for dialogue evaluation @xcite . however, we are the first to study the evaluation capabilities of the newest batch of llms in terms of multilinguality and paraphrase robustness. this paper presents our contribution to the dstc11 track on robust and multilingual automatic evaluation metrics for open-domain dialogue systems @xcite , where we participated in both the multilingual and robustness tasks. this track is an excellent venue to benchmark the capabilities of these new llms for dialogue evaluation, as it evaluates properties that have been observed in these models. we propose a comprehensive framework, incorporating earlier encoder-based approaches and chatgpt, as illustrated in figure 1 . by combining multiple models and submetrics through ensembling, our approach aims to improve the performance and robustness of dialogue evaluation, ultimately contributing to the advancement of dialogue system research and development. overall, our contributions are the following: ## problem formulation the main goal of this track was to develop and benchmark automatic open-ended dialogue evaluation metrics. two tasks were proposed this year, metrics for multilingual data and robust metrics. for the metrics for multilingual data task, participants were asked to construct quality metrics that perform well on a multilingual setup. for the the robust metrics task, the goal was to develop metrics that perform robustly when evaluated over back-translated/paraphrased sentences in english. in both tasks, the proposed metrics were evaluated at the turn and dialogue level, without access to a reference. in a turn-level evaluation setting, the goal is, given prior dialogue history (frequently denoted as context) c of varying amount of turns, and a response r, to learn a scoring function (also known as metric) that assigns a score f (c, r) → s. conversely, in a dialogue-level evaluation setting, the goal is to evaluate the performance throughout the full dialogue. irrespective of the level of evaluation, the proposed metrics' outputs are typically compared against he annotations that use a likert scale, where the lowest value means lowest quality and highest value maximum quality. for this track, the performance of these metrics was evaluated by calculating the pearson correlation between the calculated score and he. ## methodology our framework, which we call dialevalml, can be viewed as a dual layered ensemble which are done at the model and submetric level, and that employ strong multilingual pretrained encoder and decoder models which were finetuned or prompted foot_0 . in this section, we describe the stepby-step process of dialevalml, detailing the various components and methods employed. ## discussions the results from our work on both tasks (section 5.4) reveals that chatgpt vastly outperforms typical encoder approaches that are trained to discriminate positive samples from artificially generated negative ones. it is important to note that, compared to the months worth of research dedicated to optimise our encoder models (including curation, training and selection), we were able to easily outperform all other teams and our own encoder models with a day's worth of prompt engineering. this is, in our opinion, a turning point in the paradigm of dialogue evaluation. in any case, we do find instances where chatgpt fails to accurately evaluate aspects of quality, as identified in section 5.5. future research directions may attempt to tackle the issues of score calibration by providing prompts that include examples and/or explicitly provide guidelines for scoring. however, given the current landscape on dialogue generation, and as our submission suggests, dialogue evaluation, it is important to reflect on the value of current quality estimation frameworks. one might argue performing he or developing metrics that evaluate responses and/or dialogues in terms of linguistic competence (e.g. grammatical correctness or coherence) is no longer informative for the current and future crop of llms. besides becoming ever so clear that these models no longer output responses that are incoherent or incorrect, we are reaching the point where these models are better evaluators than humans themselves @xcite . as such, developing metrics that correlate well with he is becoming increasingly questionable. one of the main contention points w.r.t the deployment of these models to the public pertain to their "safety" and "trustworthiness". but while "trustworthiness" can be evaluated by connecting the outputs to external and verifiable sources, the notion of "safety" is much more ambiguous. @xcite suggests considering positionality, acceptability, and value alignment (pava) as features chatbots should have to fulfil appropriateness requirements. however, automatically evaluating if a chatbot has these features using current dialogue evaluation protocols seems implausible. instead, the development of challenge sets for validation (such as the ones proposed in @xcite appears to be the logical next step for evaluation of future chatbots foot_5 . ## conclusion this paper presents a novel open-domain and reference-free dialogue evaluation framework that leverages strong pretrained llms using finetuning and zero-shot prompting. these models, combined with effective ensembling strategies, substantially outperform the previous automatic evaluation paradigm of only training lms with semisupervised training objectives. in fact, dialevalml ranks 1st on both the robust (1st turn-level, 2nd dialogue level) and multilingual (1st on both levels) tasks of track 4 at dstc11.