x D ial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark (2023)

Abstract: Recent advancements in reference-free learned metrics for open-domain dialogue evaluation have been driven by the progress in pre-trained language models and the availability of dialogue data with high-quality human annotations. However, current studies predominantly concentrate on English dialogues, and the generalization of these metrics to other languages has not been fully examined. This is largely due to the absence of a multilingual dialogue evaluation benchmark. To address the issue, we introduce xDial-Eval, built on top of open-source English dialogue evaluation datasets. xDial-Eval includes 12 turn-level and 6 dialogue-level English datasets, comprising 14930 annotated turns and 8691 annotated dialogues respectively. The English dialogue data are extended to nine other languages with commercial machine translation systems. On xDial-Eval, we conduct comprehensive analyses of previous BERT-based metrics and the recently-emerged large language models. Lastly, we establish strong self-supervised and multilingual baselines. In terms of average Pearson correlations over all datasets and languages, the best baseline outperforms OpenAI’s ChatGPT by absolute improvements of 6.5% and 4.6% at the turn and dialogue levels respectively, albeit with much fewer parameters. The data and code are publicly available at https://github.com/e0397123/xDial-Eval.

## introduction open-domain dialogue evaluation is a long-lasting challenge to dialogue system research @xcite . currently, human evaluation is the most reliable way to holistically judge the quality of the dialogue. due to the high costs and low reproducibility of human evaluation, automatic metrics are proposed to complement it. there are two distinct paradigms of automatic evaluation, reference-based, and reference-free. reference-based metrics, such as bleu @xcite and embedding average @xcite are widely adopted in the research community due to their easy implementation and general applicability to various types of dialogues. yet, they can be misleading due to the poor correlation with human judgment @xcite . on the other hand, reference-free metrics bypass the reliance on references and directly estimate the quality of a single response (turn level) or a multi-turn dialogue (dialogue level). currently, there's a growing interest in creating model-based, reference-free metrics. one line of work focuses on learning a discriminative metric with self-supervised learning -a model is trained to distinguish high-quality responses/dialogues from low-quality responses/dialogues based on weak supervision signals that are automatically constructed from human-human dialogue data @xcite . these metrics benefit from the bertbased language models @xcite and the availability of high-quality dialogue corpora @xcite . with the recent advancement of large language models (llms) @xcite @xcite ), an emerging line of work is to treat the llms as annotators, which judge the quality of responses/dialogues through prompting @xcite @xcite . different from the bert-based metrics, such metrics are generative in nature. a common attribute of both metric categories is that they are not trained on dialogue evaluation data with human quality annotations, yet they exhibit significant potential in simulating how humans perform evaluations at the turn or dialogue level. despite the significant progress in the field, current research predominantly focuses on the en-glish language, other languages receive insufficient attention. to expand the scope of automatic dialogue evaluation research beyond english and thoroughly investigate the language generalization potential of model-based reference-free metrics, we propose a large-scale multilingual open-domain dialogue evaluation benchmark, named xdial-eval. to construct xdial-eval, we curate 12 turn-level and 6 dialogue-level open-source english evaluation datasets. in total, the turn-level and dialoguelevel datasets comprise 14930 human-annotated turns and 8691 human-annotated multi-turn dialogues respectively. then, we translate the english data into nine different languages with commercial machine translation models. in addition, we comprehensively assess the performance of the two metric categories on xdial-eval. pertaining to the discriminative category, previous state-of-the-art (sota) bert-based methods are analyzed, while for the generative category, we evaluate the multilingual dialogue evaluation capability of recent llms, especially the instructiontuned variants, such as chatgpt foot_1 , alpaca @xcite , and vicuna @xcite . in our analysis, we systematically explore the effects of training data, training strategies, and pretrained models on multilingual dialogue evaluation. lastly, we introduce strong multilingual selfsupervised baselines on xdial-eval. specifically, we fine-tune the llms on synthetic instruction data built from human-human dialogues. the finetuned models are found to exhibit much stronger multilingual dialogue evaluation capability than the original llms. motivated by the complementary nature of generative and discriminative metrics, we perform metric ensemble, which yields strong correlations with human evaluation and language generalization capability on xdial-eval, even outperforming the powerful chatgpt, which has been recently proven to be a superior reference-free text evaluator @xcite . ## related work a long-lasting goal of automatic dialogue evaluation is to fully approximate human evaluation, which is quantified by strong correlations (0.8+) with ground-truth human quality annotations @xcite . recent research is dedicated to developing powerful model-based reference-free metrics for automatic dialogue evaluation @xcite . bert-based discriminative metrics -a series of works @xcite @xcite @xcite @xcite targets turn-level evaluation and leverages self-supervised learning. they rely on negative sampling strategies, such as random utterance replacement and word order shuffling, to generate synthetic data for training discriminative models. another group of metrics is learned to holistically judge the quality of multi-turn dialogues @xcite @xcite with a similar idea that a model is trained to distinguish original humanhuman dialogues from negative samples generated by strategies, such as utterance order shuffling. all the metrics rely on either bert @xcite or roberta @xcite for dialogue context understanding and modeling the utterancelevel interactions. although they exhibit good correlations with human evaluation on english benchmark datasets, their transferability across different languages has not been explored. in this study, we select representative discriminative metrics and conduct correlation analyses on the multilingual xdial-eval benchmark. llm-based generative metrics -the evolution of large language models (llm) has drastically changed the nlp landscape @xcite @xcite . especially, the line of works on instruction-tuning of llms @xcite @xcite @xcite @xcite has led to general-purpose ai assistants that exhibit remarkable ability in understanding and following users' instructions. in the context of automatic dialogue evaluation, these llms can serve as unified evaluators of both the response and the multi-turn dialogue quality through prompting with task-specific templates. for example, @xcite introduces instructdial, which consists of 48 diverse dialogue tasks in a unified text-to-text format. models tuned on the instructdial dataset demonstrate good zero-shot performance in the dialogue evaluation task. @xcite conduct a comprehensive analysis of the dialogue evaluation capability of llms with varying model types, sizes, choices of training data, etc. they observe that the smaller language models fine-tuned on instructions and dialogue-specific data can outperform very large language models. we move beyond both works by comprehensively exploring the llms' ability in evaluating multilingual dialogue. more recently, several works @xcite @xcite ) study the dialogue evaluation capability of closed-source instruction-following llms via prompting, such as openai's chatgpt and anthropic claude @xcite . these closed-source llms exhibit strong zero-shot correlations with human evaluation. yet, a major limitation is that it is not always feasible to adapt such models to custom data. our work serves as the first study exploring the dialogue evaluation capability of both the closed-source and the recent open-source instruction-tuned llms on multilingual dialogue evaluation data. additionally, we move beyond prompt engineering by performing task-specific finetuning of the open-source llms to boost the dialogue evaluation ability and language generalization of such models. @xcite with little emphasis on automatic dialogue evaluation. for example, @xcite constructs a multilingual dataset, which contains the ubuntu irc logs in 12 different languages for response selection. zhang et al. (2021d) proposes mrs, a multilingual reply suggestion dataset with ten languages. more related to our work, rodríguez-cantelar et al. (2023) recently held the shared task on "robust and multilingual automatic evaluation metrics for open-domain dialogue systems" at dstc11 3 . they release a series of multilingual evaluation datasets and a multilingual deep am-fm (zhang et al., 2021b) baseline. yet, their released data is only limited to 3 languages. most of the data target turnlevel evaluation with a limited amount of dialoguelevel annotated data. on the contrary, our proposed benchmark, xdial-eval, contains large-scale annotated data at both turn and dialogue levels, spanning 10 different languages. 3 https://dstc11.dstc.community/tracks 3 the xdial-eval benchmark 3.1 benchmark details the xdial-eval benchmark is built on top of opensource english evaluation datasets. the statistics of the datasets are outlined in ## model-based reference-free metrics mathematical formulation -let d i,l denote a dialogue evaluation dataset in xdial-eval with index i and language l. d i,l either comprises n number of multi-turn dialogues (dialogue-level) or j number of context-response pairs (turn-level). each data instance can be either a multi-turn dialogue we denote the mean quality score assigned by human annotators to a specific data instance as h to score d i,l j or r i,l j with the llms, we first need to convert the input instance to the corresponding model-and task-specific instruction-based prompt templates. take alpaca as an example, an additional note is that current llm finetuning often uses human-annotated data. however, due to challenges in scaling up high-quality, humanannotated training data collection for open-domain dialogue evaluation, and the proven success of sota bert-based metrics using automaticallyconstructed synthetic dialogue data, we propose to investigate whether llms can also benefit from finetuning with synthetic data. specifically, we reuse the xpoe-turn and xfined-dial datasets described in the previous section to perform instruction-tuning of the llms. to speed up the experiments, we sample a subset of 100k (10k per language) from each dataset. subsequently, we transform the data into an instruction format using model-specific prompt templates, similar to those in ## experiment setup we explore the effects on multilingual dialogue evaluation with both zero-shot prompting and finetuning the llms on different types of data 12 . for llm finetuning, we first investigate the crosslingual generalization of the llms when only finetuned on the english synthetic data. models with different pretrained backbones are chosen for investigation: poe, fined-eval, alpaca-7b, phoenix-7b, llama-2-7b, and baichuan-2-7b. secondly, we finetune the llms using the syn12 the computation details are outlined in appendix f. ## conclusion this paper introduces xdial-eval, a multilingual dialogue evaluation benchmark featuring 14930 annotated turns and 8691 dialogues in 10 languages. both automatic and human evaluation validate the high quality of xdial-eval. additionally, we examine the performance of bert-based metrics and emerging llms on this benchmark. key takeaways include: (1) sota bert-based metrics, backed by strong cross-lingual encoders and multilingual training data, can effectively handle multilingual dialogue evaluation. (2) recent generalpurpose instruction-following llms show promise as unified multilingual dialogue evaluators. future work could refine optimization techniques and use high-quality, task-specific data for fine-tuning the llms. (3) ensembling bert-based metrics and llms outperforms chatgpt in correlation with human judgment and achieves good language generalization (4) despite the ensemble approach's good correlations, multilingual automatic dialogue evaluation remains unsolved. xdial-eval could serve as a valuable benchmark for tracking future research progress in this field.