EC oh: Turn-level Coherence Evaluation for Multilingual Dialogues (2024)

Abstract: Despite being heralded as the new standard for dialogue evaluation, the closed-source nature of GPT-4 poses challenges for the community. Motivated by the need for lightweight, open source, and multilingual dialogue evaluators, this paper introduces GenResCoh (Generated Responses targeting Coherence). GenResCoh is a novel LLM generated dataset comprising over 130k negative and positive responses and accompanying explanations seeded from XDailyDialog and XPersona covering English, French, German, Italian, and Chinese. Leveraging GenResCoh, we propose ECoh (Evaluation of Coherence), a family of evaluators trained to assess response coherence across multiple languages. Experimental results demonstrate that ECoh achieves multilingual detection capabilities superior to the teacher model (GPT-3.5-Turbo) on GenResCoh, despite being based on a much smaller architecture. Furthermore, the explanations provided by ECoh closely align in terms of quality with those generated by the teacher model.

## introduction with llms showcasing impressive reasoning and dialogue understanding capabilities vastly superior to any prior nlp technologies, human evaluation has more recently been complemented with automatic evaluations using @xcite . however, gpt-4 as an automated evaluator has its downsides. perhaps the main downside is it being a closed source model hidden behind a paid api, making accessibility difficult for those outside the coverage area and lacking extensive financial resources, while also lacking transparency in its development. in contrast, and to the best of our knowledge, the study of open source alternatives to gpt-4 based dialogue evaluation is mostly limited 1 pronounced "echo". context: a: hello. i bought a china dress in your shop this morning. b: yes? a: i bought it one size up by mistake. b: oh, did you? a: i wonder if you can change it to one size down. reference response. b: yes, of course. will you come with the receipt ? "the response acknowledges the request and offers a solution to accommodate the customer's needs. the answer is yes." random negative sample. b: i'd like a book about law. "the response is completely unrelated to the situation discussed. the answer is no." genrescoh positive sample. b: absolutely, if you bring the dress back to the shop with the receipt, we can exchange it for a smaller size for you. "the response acknowledges the request and offers a solution to the problem. the answer is yes." genrescoh negative sample. b: oh, that's great to hear! i hope it fits perfectly. "the response does not acknowledge the request for a size change and instead expresses an unrelated sentiment. the answer is no." to the benchmarking of open source and open access llms or finetuning with dialogue data @xcite @xcite . these works suggest that llms struggle to outperform older encoder-based metrics trained using negative sampling approaches for relevance (e.g. random response selection). however, it is important to point out that these benchmarks have several limitations. first and foremost, the high performance of these encoder-based models can be explained by the fact that the benchmarks themselves are based on old generative models that exhibit relevance issues that are easy to detect. for instance, in these key observations motivate our work. in order to move towards the development of metrics that evaluate dialogue coherence and are multilingual, we propose genrescoh (generated responses targeting coherence), a collection of positive and negative responses focused on coherence. our dataset, generated using strong llms, contains over 130k responses in different languages (english, french, german, italian, and chinese), together with their corresponding explanations (in english). by prompting an llm, we are able to (1) obtain positive samples that are in distribution (llms frequently output more verbose responses than their human counterparts); (2) obtain negative samples that remain semantically relevant but contain coherence and logical consistency issues, which may be more informative during training, and that are more representative of current limitations of llms. with this dataset, we train a family of evaluators we call ecoh (evaluation of coherence) 2 . our results demonstrate that distilling coherence knowledge from a strong llm allows us to obtain multilingual coherence detection performance of .945 f1 score using a 0.5b model, which is superior to both the teacher models' (gpt-3.5-turbo) .910 and a much larger model of the same family (qwen1.5-7b-chat -0.825). furthermore, the explanations provided by ecoh are of higher quality than qwen1.5-7b-chat, scoring an average of over 4 out of 5 on most instances, as reported by gpt-4 evaluations. ## genrescoh responses dataset this section introduces genrescoh, a multilingual, large-scale response collection that targets coherence, seeded from well established dialogue datasets ( §2.1), and generated using llms ( §2.2). ## ecoh this section presents ecoh, our proposed family of response coherence evaluators. we initially present the method of formulating the task of coherence evaluation as explainable qa (question answering) ( §3.1). then, we describe in detail how our evaluator is trained ( §3.2) and evaluate its performance on different settings ( §3.3). ## conclusions this paper presents genrescoh, a large scale collection of positive and negative responses and corresponding explanations covering several languages. genrescoh is generated from xdaily-dialog and xpersona using state-of-the-art llms, which better matches the responses seen by contemporary chatbots. with this dataset, we train a family of evaluators we call ecoh. our smallest model (0.5b) is able to achieve similar performance to that of the teacher model (gpt-3.5-turbo), despite being much smaller. despite this good performance, we note some limitations when using ecoh, especially when evaluating human responses and/or responses that require more robust commonsense reasoning. whilst we argue that including more data that targets commonsense and human responses, or even training a larger model could mitigate these issues, since we include an explanation in the predictions, one could still use our evaluators for an initial evaluation screening and escalate to a human evaluator if necessary.