Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLM s (2024)

Abstract: Although human evaluation remains the gold standard for open-domain dialogue evaluation, the growing popularity of automated evaluation using Large Language Models (LLMs) has also extended to dialogue. However, most frameworks leverage benchmarks that assess older chatbots on aspects such as fluency and relevance, which are not reflective of the challenges associated with contemporary models. In fact, a qualitative analysis on Soda. (Kim et al., 2023), a GPT-3.5 generated dialogue dataset, suggests that current chatbots may exhibit several recurring issues related to coherence and commonsense knowledge, but generally produce highly fluent and relevant responses.Noting the aforementioned limitations, this paper introduces Soda-Eval, an annotated dataset based on Soda that covers over 120K turn-level assessments across 10K dialogues, where the annotations were generated by GPT-4. Using Soda-Eval as a benchmark, we then study the performance of several open-access instruction-tuned LLMs, finding that dialogue evaluation remains challenging. Fine-tuning these models improves performance over few-shot inferences, both in terms of correlation and explanation.

## introduction current evaluation practices of dialogue systems are still highly dependent on human evaluation. whilst evaluation using strong llms has increased in popularity, human evaluation remains the gold standard since it offers a direct view towards human preferences and satisfaction in the interaction, which is the major goal of open domain dialogue. leaderboard platforms such as chatbot arena @xcite rank models by calculating an elo rating obtained from pairwise comparisons of chatbot responses. however, da (direct 1 github.com/johndmendonca/soda_eval. yasmine: i've been feeling really anxious lately about my job. i'm not sure if i can find a new job that pays as much as my current job. wife: why are you feeling anxious about your job? yasmine: i'm just not sure if i have the skillset to find a new job that pays as much as my current one. my current job is pretty secure, but it doesn't pay that well. and with the economy the way it is, i'm just not sure if i can find a new job that pays as much as my current one. overall assessment: 2. the response contains contradictions and repetition, affecting coherence. despite the potential benefits of direct assessments for open-domain dialogues, the evaluation community is constrained to using a limited number of benchmark datasets, many of which have become outdated @xcite . for instance, fed @xcite , a typically used benchmark in dialogue evaluation, annotates responses generated by arguably obsolete chatbots, and targets quality aspects such as fluency or relevance. while the annotation of these quality aspects may have been of interest at the time, it is not clear if contemporary chatbots still suffer from these issues. in this work we conduct a qualitative analysis of the dialogues that constitute the soda dataset @xcite . soda contains dialogues distilled from gpt-3.5 @xcite , which allows us to better understand the limitations of dialogue generation for this model. our findings suggest that most issues pertain to a lack of coherence, commonsense knowledge, and repetitions. in contrast, generation is almost always fluent and relevant to the prior context, thus confirming newer models have mostly achieved human level fluency and relevance. several authors have proposed automated evaluation frameworks to scale such an analysis and/or complement human evaluation. with the introduction of llms for this task, many studies have surfaced, ranging from direct assessment of responses and dialogues @xcite to a more comprehensive analysis @xcite . however, we point out two limitations within current work: firstly, given the complexity of the task, most studies leverage @xcite , which is known to perform as well as human annotators in many tasks @xcite . however, such models have downsides with respect to accessibility. secondly, since the development of frameworks that use these models are mostly informed by quality aspects used in older benchmarks, their performance when evaluating contemporary chatbots remains an open question. given these limitations, we conduct a large scale dialogue quality annotation based on the soda dataset. our annotations, which we call soda-eval, include over 120 thousand turn level assessments covering 10 thousand dialogues. these annotations are conducted by gpt-4, and target a diverse range of quality aspects, as illustrated in with soda-eval as a benchmark, we conduct a study on the performance of several open-access instruction-tuned llms as dialogue evaluators, and show that the evaluation of stronger chatbots is a challenging task. utilizing soda-eval, we also experiment with finetuning these models, and demonstrate an improvement in performance, both in terms of their correlation with gpt-4 assessments and of the validity of their explanations. furthermore, we also assess the impact of finetuning the models by evaluating the resulting models on out-of-domain datasets, where we also observe improved correlation performance. this indicates the models' adaptability to different evaluation guidelines and a diverse set of dialogue responses. overall, our contributions are as follows: ## soda-eval having identified typically found issues in dialogue generation, we move to the curation of a large scale evaluation benchmark dataset based on soda (cc-by 4.0). this dataset, which we call soda-eval, contains annotations by gpt-4 ( §4.1) and covers over 120 thousand responses of diverse quality ( §4.3). human annotations confirm the validity of this annotation ( §4.2). additional details regarding the development of soda-eval, including preprocessing, data selection and an in-depth statistical analysis of the annotations are available in appendix c. ## training and benchmarking of evaluation models as identified in prior sections, the typically employed benchmark datasets focus on the evaluation of responses that do not accurately reflect current generation capabilities. as a result, most evaluator development has been focused on maximising predictive performance of no longer relevant quality aspects, instead of focusing on more complex issues. soda-eval positions itself as both a more realistic training dataset and as an improved evaluation benchmark for open-domain dialogue evaluation. in this section, we examine the performance of several open-access llms for the task of overall assessment of responses with explanations. ## conclusions this paper presents soda-eval, a large-scale open-domain dialogue quality annotation dataset targeting the responses provided by a llm, encompassing over 120 thousand turn-level assessments. this curation was motivated by the pitfalls of current dialogue evaluation, which is limited to assessing responses generated by outdated chatbots and quality aspects. as highlighted in our qualitative analysis of soda, newer models achieve human-level fluency and relevance, but fall short in areas like coherence and commonsense reasoning. with soda-eval, we conducted a comprehensive study on the performance of openaccess instruction-tuned llms when evaluating dialogue responses. our findings show that finetuning these models on soda-eval improves their performance, both in terms of correlation and explanation quality.