Three Ways of Using Large Language Models to Evaluate Chat (2023)

Abstract: This paper describes the systems submitted by team6 for ChatEval, the DSTC 11 Track 4 competition. We present three different approaches to predicting turn-level qualities of chatbot responses based on large language models (LLMs). We report improvement over the baseline using dynamic few-shot examples from a vector store for the prompts for ChatGPT. We also analyze the performance of the other two approaches and report needed improvements for future work. We developed the three systems over just two weeks, showing the potential of LLMs for this task. An ablation study conducted after the challenge deadline shows that the new Llama 2 models are closing the performance gap between ChatGPT and open-source LLMs. However, we find that the Llama 2 models do not benefit from few-shot examples in the same way as ChatGPT.

## introduction this paper describes the systems submitted by team6 for chateval, the dstc 11 track 4 competition aimed at evaluating open-domain chat. 1 we participated in task 2, which focuses on evaluating multiple criteria on the level of individual dialogue turns. the task of evaluating responses in a chat is challenging because it requires an understanding of the interlocutor's roles (pragmatics), the conversation's context, and the response's meaning (semantics). at the same time, the conversations are often ungrammatical @xcite and vary in style @xcite . the commonly used metrics, such as bleu @xcite , meteor @xcite , or bertscore @xcite , are based on comparison to human references and thus correlate poorly with human judgments on the turn- ## task & data the goal of the dstc11 track 4 task 2 was to predict several turn-level metrics automatically on the test set. for each dialogue turn, considering the preceding dialogue history, the participants were to submit a system to predict the score of the target metrics, defined by the organizers as: the organizers provided the participants with training, development, and test sets @xcite , each coming from different domains and annotated with different metrics: for our experiments, we created a heuristic mapping to the target metrics on a subset of the development set (see section 3). the submitted systems were benchmarked for the quality of their ranking using the spearman correlation coefficients (scc) @xcite computed between the predicted scores and the human judgments. as a secondary measure, the pearson correlation coefficients (pcc) @xcite were used to evaluate the correlation. the measures were computed for each of the target metrics separately. the overall submissions' ranking was determined using the average of the four sccs. ## data preprocessing since no information was provided on how the individual development dataset metrics relate to the target dialogue metrics, we built a heuristic to obtain target metric scores. the heuristic uses a linear combination of one or more dataset-specific metrics to the target metrics, chosen based on individual descriptions from the literature. 2 using the development set and the heuristic, we created a supervised dataset and split it into training and development splits. we used this development dataset for model selection or supervised training, and we use this dataset to develop the three systems described in section 4. dialogue turns appr rich. gram. rel my boss gave me a 10 raise just last month and it was a nice surprise 5 5 5 -it's great and he might think you're doing a great job 5 5 5 5 we have always been very nice he has always been very supportive of me 4 5 5 5 that's a good thing 4 3 5 4 do you have any pets? 5 4 3 -i am retired so i love to travel so pets would slow me down 4 4 3 4 i understand that my idea of traveling is a hot hot bubble bath 3 4 2 2 yes i have dogs and cats i like to take them with me on trips 2 4 2 2 during our experiments, we struggled to find representative labels and input data which could be used as a development set. therefore, we decided to annotate the additional 156 turns from the rehearsal set with the target metrics described in section 2. we used this data to find our submitted systems' optimal hyperparameters. we assumed that this data came from the same distribution as the test set, but this later proved clearly not to be the case, as seen in figure 2 . note that we did not use the training set at all. ## submitted systems inspired by @xcite , we used pre-trained llms with prompts for predicting the individual metrics. we started with the simplest approach possible and manually designed the prompts. ## experiments we experimented with the three methods described in section 4. first, we we experimented with the simple prompting method using the open-source llms (section 4.1). based on the results, we started two independent experiments. section 5.2 describes the ffn training and section 5.3 describes the development of vector storage which we used with chatgpt api. for all three methods, we used the rehearsal set to select the best-performing model-template combination and hyperparameters. ## results & discussion we report positive findings related to method 3 (section 4.3), but we also report lessons learned from implementing the other two methods and, in general, using the data provided for the challenge. first, we summarize observations from our use of the data (section 6.1). then we report negative results from the simple prompting and ffn finetuning (sections 6.2 and 6.3, respectively). we also report our best results from the vector store (section 6.4) and discuss what our best model in the challenge is capable of evaluating. finally, we add an ablation study in section 6.5 performed after the challenge was complete, comparing few-shot capabilities of chatgpt with the newly released llama 2 model. we are aware that llms are trained on large datasets, some of which (e.g., chatgpt) are not public. however, due to the novelty of the test set @xcite , we believe that the test set has not leaked to their training set. ## related work using simple text perturbations targeting the individual qualities, they show that most existing metrics are not robust enough. two very recent works, closely related to ours, propose the usage of instruction-tuned llms to evaluate generated text in various tasks like summarization and dialogue response generation @xcite , or machine translation @xcite . both approaches use in-context learning and multiple prompting techniques to obtain scalar metric predictions or candidate rankings. they achieved good results and correlations with human judgments. however, they used only closed models for the evaluation and did not experiment with few-shot prompting using relevant examples. ## conclusion we presented three simple approaches to using llms for turn-level chat evaluation. we achieved promising results using chatgpt prompting with few-shot example retrieval from a vector score, and ranked as the second-best team. based on the results of our best system, we argue that chat turn evaluation systems based on current state-of-the-art llms are usable only for system-level evaluation but not for segment-level evaluation, i.e., they cannot distinguish between the quality of individual turns, especially for outputs of high-quality latest systems based on llms such as chatgpt and gpt3. we observed that llms are fragile to the prompts, few-shot examples and cannot be used out-of-the-box for chat evaluation. we also report implementing a simple regressor on top of embeddings obtained from the prompted llm decoder. we attribute its poor performance to our incorrect implementation of data preparation. we also presented an ablation study that inves-tigated the influence of the few-shot examples on the performance of llms. we found that few-shot examples help the llms to generalize better to unseen data, especially with respect to fitting the desired output format. however, using examples dynamically obtained from the vector store instead of hand-picked fixed examples did not bring any additional improvements. we reached a new best spearman correlation coefficient of 0.6136 for appropriateness with chat-gpt and fixed few-shot examples in our ablation study. in addition, the llama 2 open model used in our ablation showed significant improvements over the challenge baseline.