Leveraging LLM s for Dialogue Quality Measurement (2024)

Abstract: In task-oriented conversational AI evaluation, unsupervised methods poorly correlate with human judgments, and supervised approaches lack generalization. Recent advances in large language models (LLMs) show robust zero- and few-shot capabilities across NLP tasks. Our paper explores using LLMs for automated dialogue quality evaluation, experimenting with various configurations on public and proprietary datasets. Manipulating factors such as model size, in-context examples, and selection techniques, we examine “chain-of-thought” (CoT) reasoning and label extraction procedures. Our results show that (1) larger models yield more accurate dialogue labels; (2) algorithmic selection of in-context examples outperforms random selection,; (3) CoT reasoning where an LLM is asked to provide justifications before outputting final labels improves performance; and (4) fine-tuned LLMs outperform out-of-the-box ones. In addition, we find that suitably tuned LLMs exhibit high accuracy in dialogue evaluation compared to human judgments.

## introduction evaluating conversational system performance in nlp is challenging. automating effective evaluation is crucial for enhancing dialogue systems. however, automatic metrics like bleu @xcite and rouge @xcite ) fall short in accurately measuring perceived quality due to complex mappings @xcite . advanced methods (usr @xcite , fed @xcite , dialogrpt @xcite ) address this but often require extensive training data and human references, making them costly and limited in generalization to new datasets. recent advancements in large language models (llms) @xcite have demonstrated robust zero-or few-shot capabilities and reasoning skills across a range of tasks @xcite . consequently, researchers have begun to explore the application of llms to classification problems such as dialogue evaluation @xcite . recent studies @xcite demonstrate that llms excel on diverse datasets. however, uncertainties persist regarding the impact of factors like model size and in-context examples on open-sourced llm performance. this paper aims to clarify these influences. additionally, there is a lack of comprehensive literature on deploying llms for dialogue evaluation. to fill this gap, we propose two common evaluation strategies, providing a comparative analysis of their pros and cons. in this paper, we systematically study different aspects of llm-based dialogue evaluation by conducting extensive experiments on two benchmark datasets, one publicly available and the other proprietary, amazon-internal datasets. we initially explore the connection between different attributes such as model size and in-context examples, and their impact on dialogue evaluation performance. additionally, we present a dialogue evaluation that leverages "chain-of-thought" (cot) reasoning abilities of llms @xcite . our experiments demonstrate that larger model sizes and instruction tuning generally helps with zero shot dialogue evaluation. furthermore, in fewshot scenario, we find that algorithmic selection of in-context examples yields better results than random selection. next, we demonstrate that supervised fine-tuning can substantially improve the performance of llms on dialogue evaluation task. finally, we explore and validate a cot-based evaluation framework which is capable of returning not only dialogue labels but comprehensive explanations and justifications, thereby offering a more coherent and holistic evaluation. remarkably, our results indicate that cot-based evaluation is more accurate when the llm is prompted to first analyze the dialogue and then produce labels. combined, our findings confirm the applicability and effectiveness of llm-based automated dialogue evaluation. the g-eval framework @xcite is a notable advancement, synergistically integrating llms with the chain-of-thought (cot) paradigm and a form-filling strategy. notably, using gpt-4 @xcite as its foundational model, g-eval shows strong correlation with human evaluations in summarization tasks. parameter-efficient fine-tuning (peft). as base language models grow in size @xcite , researchers frequently turn to parameter-efficient fine-tuning techniques to tailor models for specific downstream tasks. these fine-tuning approaches typically fall into three main categories: these strategies reflect ongoing efforts to make large-scale models more adaptable and efficient, leveraging their vast capacities while mitigating computational and practical challenges. in-context learning. in-context learning is a prompting technique for llms in which example input-output pairs for some task are injected into the prompt before the target input is presented. the idea is that seeing correct examples of the task will help the model to provide a correct target output. selecting in-context examples is crucial for effectively prompting llms, enabling them to pivot to new tasks without extensive fine-tuning. examples play a pivotal role in guiding llms' predictive capabilities, with research exploring methods such as semantic proximity evaluation @xcite and retrieval mechanisms such as bm25 @xcite , used independently or in an initial training phase for a selector retriever. these selection approaches excel in few-shot nlp tasks. for instance, in @xcite , a bifurcated framework effectively annotates and selects in-context samples from unlabeled repositories, achieving impressive performance across various tasks. similarly, @xcite suggested that choosing examples with congruent sentence embeddings optimizes gpt-3's efficacy. despite positive outcomes, there is a need for deeper explorations to discover more general in-context example retrieval methodologies. dialogue evaluation with generation. in addition to the mentioned method, lin and chen (2023) proposed a novel framework where llms are prompted to directly generate responses for dialogue evaluation. ratings for the dialogue can then be extracted from the produced llm responses. see prompts in section a.2. ## experiment setup model. in this study, we utilize models from the llama family @xcite and the falcon series @xcite . we also incorporate the instruction-tuning variants of these models as proposed in the alpaca study @xcite . temperature is fixed at 0.7 during generation. dataset. we experiment on two datasets: the publicly available uss @xcite dataset includes sgd, multiwoz, redial, and ccpe subsets, with a 1-5 quality score scale. we randomly allocate 10% as test data, using the rest for training (supervised fine-tuning or in-context learning). we also evaluate our methods on two versions of an amazon-internal dialogue-quality dataset, which has a human-annotated quality rating on a scale of 1-5 (similar in format to the data described in @xcite ). the rating distribution is shown in figure 2 , with the smaller training set's rating distribution resembling the test set more closely than the larger one. we binarize the datasets, considering scores of three and below as "defect" (unsatisfactory) and scores of four or five as "non-defect" (satisfactory). this simplified scheme enables us to use standard binary classification metrics to compare the effectiveness of different methods. in-context example selection methods. here we use three methods to select in-context examples. the first is to simply source examples randomly from the training dataset. the second employs a probabilistic algorithm for information retrieval (ir) @xcite , selecting similar examples based on a defined similarity metric. the third relies on bert @xcite , extracting representations from the input dialogue and identifying similar examples using cosine similarity computations. together, these three methods offer diverse strategies for in-context example selection. we conducted three runs for the random selection method and report the mean value. supervised fine-tuning method. in this study, we also explore supervised fine-tuning to adapt llms to the task of dialogue quality evaluation. to manage cost, we adopt the lora setting @xcite , fine-tuning a relatively small number of parameters compared to full-rank fine tuning. lora fine-tuning efficiently enhances the model's effectiveness at the target task while addressing computational challenges. ## results we now describe the results of our experiments designed to analyze various aspects of llm-based dialogue evaluation. ## conclusion this paper explores the application of llms to evaluation of task-oriented dialogue systems. key findings include: the impact of pretrained model size; the importance of instruction fine-tuning; the effectiveness of in-context examples; consistent performance improvement through supervised fine-tuning; and that the chain-of-though paradigm is most effective with the analysis-first approach.