Parallel Corpora Alignment Framework for Multilingual and Robust Automatic Dialogue Evaluation (2023)

Abstract: Open-domain automatic dialogue evaluation plays an important role in dialogue systems. While recent efforts are being put into making learning-based evaluation metrics correlate better with human evaluation, robust metrics for parallel corpora and multiple domains remain unexplored. Parallel corpora refer to corpora that express the same idea in different ways (e.g., translation, paraphrasing and back-translation). In this paper, we propose Parallel Corpora Alignment Framework (PCAF), which improves the consistency and robustness of model evaluation on parallel corpora. Firstly, parallel corpora are aligned in semantic space through parallel-corpora-aligned contrastive learning. Then, parallel-corpora-aligned distillation on multi-dataset is applied to further improve model’s generalization ability across multiple data domains. Our approach ranks second on the final test data of DSTC11 track4 subtask1 (“Multilingual Automatic Evaluation Metrics”, turn-level) and third on the subtask2 (“Robust Automatic Evaluation Metrics”, turn-level), which proves the strong generalization ability and robustness of our proposed approach.

## introduction open-domain automatic dialogue evaluation, which aims to evaluate dialogues efficiently and accurately, plays an important role in dialogue systems. on the one hand, it provides a basis for crossmodel comparison, on the other hand, it points out the direction for model improvement. while recent efforts are being put into making learningbased evaluation metrics correlate better with human evaluation, robust metrics for parallel corpora and multiple domains remain unexplored. parallel corpora refer to corpora express the same idea in different ways (e.g., translation, paraphrasing or back-translation). in dialogue system technology challenge 11 (dstc11) foot_0 , the track4 "robust and multilingual automatic evaluation metrics for open-domain dialogue systems" proposes such a challenge, which consists of two sub-tasks. in sub-task1 ("metrics for multilingual data"), all participants need to develop effective automatic openended and multilingual (i.e., english, spanish and chinese) dialogue evaluation metrics that perform similarly when evaluated over all the languages. in sub-task2 ("robust metrics"), all participants need to develop effective automatic open-ended dialogue evaluation metrics that perform robustly when evaluated over back-translated/paraphrased sentences in english. for both tasks, the developed metrics should be correlated to human judgements well and explainable. current automatic dialogue evaluation metrics include word overlap-based metrics, embeddingbased metrics and learning-based metrics. word overlap-based metrics (e.g., bleu @xcite , rouge @xcite and meteor @xcite ) evaluate candidate response through its overlapping words with reference response. embedding-based metrics (e.g., greedy matching @xcite and vector extrema @xcite ) firstly obtain sentence representation through word embedding (e.g., word vector @xcite ), then, the semantic similarity between candidate response and reference response is calculated by their representation for dialogue evaluation. however, due to the one-to-many nature of open-domain dialogue @xcite , the two referenced based metrics above have been shown to be poorly correlated with human evaluation @xcite . learning-based metrics aim to predict the score of certain quality of candidate response and have shown great correlation with human evaluation @xcite . previous study of learning-based metrics have explored topic transition dynamics in dialogue @xcite , composition of finegrained qualities @xcite @xcite and quantifiable dialogue coherence evaluation @xcite . however, all of the current metrics are tested on a monolingual setup, and fail to consider the robustness of metrics in noisy settings. to address the above issues, we propose parallel corpora alignment framework (pcaf), which improves the consistency and robustness of model evaluation on parallel corpora. parallel corpora refer to corpora that express the same idea in different manners (e.g., translation, paraphrasing or back-translation). firstly, parallel corpora are aligned in semantic space through parallel-corporaaligned contrastive learning. then, parallelcorpora-aligned knowledge distillation @xcite on multiple datasets is applied to improve model's evaluation capability across multiple data domains. our approach ranks second on the final test data of dstc11 track4 sub-task1 ("multilingual automatic evaluation metrics", turn-level) with an average spearman correlation score of 36.57% and third on the sub-task2 ("robust automatic evaluation metrics", turn-level) with an average spearman correlation score of 38.30%. our contributions are summarized as follows: ## related work automatic dialogue evaluation is of great importance to dialogue systems. it can be divided into dialogue-level and turn-level, while dialogue-level pays attention to the overall evaluation of dialogue system, turn-level mainly evaluate the quality of candidate response according to the provided dialogue history. this paper mainly focus on turnlevel automatic dialogue evaluation. word overlap-based metrics and embeddingbased metrics are standard automatic evaluation metrics @xcite . however, these metrics which assess dialogue based on reference response have been shown to be inaccurate for dialogue evaluation @xcite . learning-based metrics was then proposed, which adopts deep learning models and aims to predict human-like scores to input responses @xcite . learning-based metrics can be divided into supervised and self-supervised. supervised metrics @xcite highly depend on humanannotated training data, which are not widely studied due to the lack of such data. selfsupervised metrics utilize human response as positive responses, and negative responses are obtained through negative sampling, thus, positive-negative responses pairs are constructed for model training. as human conversations are readily available (e.g., dailydialogue @xcite ), various self-supervised metrics have been proposed. grade @xcite applies graph reasoning to model topic transition dynamics in dialogue. maude @xcite distinguishes the positive and negative responses using nce loss @xcite . usr @xcite ) trains one language model and two dialogue retrieval models to measure five qualities respectively and regresses them to an overall score. mme-crs @xcite trains five submodels to evaluate five qualities respectively and weighted them to an overall judgement. however, all of the current metrics are tested on a monolingual setup, and fail to consider metrics' robustness to changes in domain and expression. pcaf can effectively alleviate these problems and shows great generalization ability and robustness. ## methodology figure 1 illustrates the pipeline of our proposed pcaf, a two-stage training framework which aligns parallel corpora in semantic space and improves model's generalization ability. in this section, we will introduce parallel-corpora-aligned pretraining and parallel-corpora-aligned knowledge distillation for generalization in detail. ## conclusion in this paper, we propose pcaf, a parallel-corporaaligned training framework for training multilingual and robust turn-level automatic dialogue evaluation metrics. pcaf treats corpora express the same idea in different ways as parallel-corpora, which is aligned during both pcaf pre-training stage and pcaf knowledge distillation stage. experiment results show that pcaf achieves a great performance, which demonstrates the effectiveness of pcaf. the effectiveness of each sub-method of pcaf is also proved through ablation study. besides, despite the dstc11 task4 organizers allow the participants to fine-tune their system over a subset of the development data, our submitted model is not fine-tuned with those humanannotated datasets. while we train our metric model under self-supervised learning framework, fine-tuning it on supervised datasets may improve models evaluation performance, which will be explored in our future work.