Towards Multilingual Automatic Open-Domain Dialogue Evaluation (2023)

Abstract: The main limiting factor in the development of robust multilingual open-domain dialogue evaluation metrics is the lack of multilingual data and the limited availability of open-sourced multilingual dialogue systems. In this work, we propose a workaround for this lack of data by leveraging a strong multilingual pretrained encoder-based Language Model and augmenting existing English dialogue data using Machine Translation. We empirically show that the naive approach of finetuning a pretrained multilingual encoder model with translated data is insufficient to outperform the strong baseline of finetuning a multilingual model with only source data. Instead, the best approach consists in the careful curation of translated data using MT Quality Estimation metrics, excluding low quality translations that hinder its performance.

## introduction open-domain dialogue systems have gained substantial attention in the nlp (natural language processing) and ml (machine learning) fields, thanks to their increasingly human-like behaviour @xcite . their impressive generation capabilities can be attributed to new milestones in model development and scaling @xcite , and the amount of data used during training. despite this research and development effort, advertised generation capabilities were only attainable in a select few languages (typically english or chinese) due to low resources in dialogue for other languages @xcite . more recently, however, the advent of llms (large language models) finetuned with reinforcement learning from human feedback such as chatgpt @xcite has opened the path for high-quality and easily accessible multilingual dialogue generation. similarly, automated open-domain dialogue evaluation has also been largely limited to evaluating a * work conducted as a visiting scholar at cmu. select few languages. word-overlap based metrics from nlg (natural language generation) such as bleu @xcite and meteor @xcite are agnostic to language, only requiring a reference response. however, these metrics are known to correlate poorly with human judgments due to the multifaceted nature of dialogue @xcite . reference-free metrics such as usr @xcite and usl-h @xcite , however, require dialogue data for training. considering most opensource dialogue data is in english, these models are expected to underperform significantly in other languages. additionally, most open sourced dialogue systems are also limited to english, further disincentivising multilingual research. ## problem formulation the goal of reference-free turn-level dialogue evaluation is, given a dialogue history (frequently denoted as context) c of varying amount of turns, and a response r, to learn a scoring function that assigns a score f (c, r) → s. this scoring function is compared against human judgements, which annotate the same context-response pairs. these responses are evaluated using a scaling method, for instance, a binary (0, 1) judgement or a [1, 5] scale, where the lowest value means lowest quality and highest value maximum quality. the notion of quality varies wildly depending on the annotation. in this work, we evaluate dialogue in two dimensions: most automatic evaluation metrics reformulate the problem as regression. performance is then evaluated using pearson and spearman correlations with human annotations. ## cross-lingual transfer learning the goal of the experiments described in this section was to evaluate different basic approaches of cross-lingual transfer for the task of automatic dialogue evaluation. for encoder model training, we leveraged machine translation (mt) by fully translating an english source dialogue dataset and then finetuning monolingual and multilingual models using these translations. ## mt quality-aware finetuning the effects of noise introduced to the training data is a subject of intense research in the literature @xcite @xcite . it is expected that, for this task, noise is introduced by low quality translations, reducing the performance of trained models. this issue was identified in section 4, where for the vsp model in particular, the models trained using translations performed much worse than the baseline approach. our hypothesis is that some translations heavily disrupt morphosyntactic cues used to infer response fluency, as shown in these observations, paired with the fact encoder models only slightly underperform chatgpt (a much larger and expensive model), motivate the work described in this section. we hypothesise that, by ameliorating the mt noise via identifying and filtering low quality translations, the encoder model performance can outperform llms such as chatgpt, at a fraction of the cost. since there are no available references, an mt qe @xcite automatic metric is used for this purpose. formally, an mt qe model is a scoring function that assigns a score given a source sentence and hypothesis translation. the unboundedness and uncalibrated nature of this score across languages results in the need for a cumbersome ## conclusions this paper explored the use of cross-lingual knowledge transfer for the novel task of automatic multilingual dialogue evaluation. we evaluated different strategies for this task, including zero-shot inference, mad-x and machine translation augmentation. empirically we showed that the naive approach of leveraging mt for augmentation is insufficient to outperform the baseline of english finetuning with a multilingual encoder-based lm, let alone a strong llm. instead, by filtering out low quality translations, we were able to reduce the gap of performance on chatgpt, outperforming it on select correlation metrics. experimental results showed that we obtain the best performance when training encoder models with the following proportions of mt-qe: 5% for understandability and 75% for sensibleness. one could argue the notion of quality is intrinsically related to cultural norms. for instance, japanese speakers may prefer a polite conversation, whereas german speakers might prefer a more direct interaction. a future research direction is to evaluate generative model responses in different languages using annotators exposed to the culture associated with a given language. in addition to ensuring the evaluation of the response meets the criteria of "quality" in different cultures, it would also allow for a qualitative analysis of the differences in the notion of quality between languages.