MEEP : Is this Engaging? Prompting Large Language Models for Dialogue Evaluation in Multilingual Settings (2023)

Abstract: As dialogue systems become more popular, evaluation of their response quality gains importance. Engagingness highly correlates with overall quality and creates a sense of connection that gives human participants a more fulfilling experience. Although qualities like coherence and fluency are readily measured with well-worn automatic metrics, evaluating engagingness often relies on human assessment, which is a costly and time-consuming process. Existing automatic engagingness metrics evaluate the response without the conversation history, are designed for one dataset, or have limited correlation with human annotations. Furthermore, they have been tested exclusively on English conversations. Given that dialogue systems are increasingly available in languages beyond English, multilingual evaluation capabilities are essential. We propose that large language models (LLMs) may be used for evaluation of engagingness in dialogue through prompting, and ask how prompt constructs and translated prompts compare in a multilingual setting. We provide a prompt-design taxonomy for engagingness and find that using selected prompt elements with LLMs, including our comprehensive definition of engagingness, outperforms state-of-the-art methods on evaluation of engagingness in dialogue across multiple languages.

## introduction dialogue systems are becoming more popular, but their quality is usually evaluated in terms of metrics such as fluency, coherence, or sensibility. recent advancements in large language model-based dialogue systems enable high levels of proficiency, thus shifting the emphasis from fluency evaluation to more nuanced aspects such as engagingness, which has emerged as an important quality of dialogue systems @xcite @xcite figure 1 : overview of proposed methodology, meep, on datasets in english, spanish, and simplified chinese. our prompt is distinguished by its definition of engagingness, drawn from previous research, linguistic analysis, and the study of inclusivity. extensive experiments informed the use of additional prompt design elements. scores shown here are in a range from 0 (not engaging) to 100 (very engaging). @xcite . development of models that produce more engaging responses can be supported by automatic metrics to complement human annotation. despite the ground-breaking work of existing metrics for engagingness, they evaluate the response without the conversation history @xcite , or are designed for a specific dataset @xcite , and are not highly correlated with human annotations @xcite . although multi-dimensional automatic metrics are desirable, they have limited success with complex qualities like engagement @xcite @xcite . enthusiasm for multi-dimensional evaluation is balanced by calls to develop metrics that measure specific dialogue qualities in order to complement existing metrics @xcite @xcite . furthermore, dialogue systems are increasingly available in languages beyond english so it is important to be able to test systems in these languages @xcite . with the rise of virtual assistants, the importance of engaging dialogue @xcite and multilingualism has increased significantly, as they need to actively assist users, facilitate knowledge exploration, and foster effective communication, making engagingness a vital parameter in evaluating dialogues. our research therefore develops an automatic metric for engagingness that can evaluate dialogue responses in multiple languages. we establish a comprehensive definition of engagingness in conversation and construct an extensive range of prompts designed with our dimensions of engagingness and prompt engineering techniques (see figure 1 ). these prompts are employed across several llms, which show impressive capabilities in many tasks and have had limited exploration into their use for automatic evaluation of dialogue. to our knowledge, this is the first work to extensively test prompt design for the dedicated evaluation of engagingness in dialogue in a multilingual setting. we test the prompts first on englishonly datasets, and select the best-performing set of prompting methods to test against four strong baseline metrics on spanish-language and simplified chinese-language datasets. we also test translated versions of these prompts. we find that our proposed framework -meep: metric for engagingness evaluation using prompting -can be effective for evaluation of enganginess in dialogue in multiple languages with the broader implication that llms can be used for evaluation of complex qualities of dialogue in multiple languages through prompting alone. our contributions are as follows: ## engagingness in dialogue engagingness is central to our conception of conversation, as demonstrated by the prevalence of the word "conversation" as a collocate foot_0 of the verb "engage" (coca). what then is engagingness? in prior research, several metrics conflate engagingness with interestingness @xcite @xcite . others provide no definition and the training of these metrics rely instead on the implicit judgements from the training set annotators @xcite . we posit that there is no standard definition of engagingness in nlp because of the subjectivity of human judgment. humans, like models, cannot be relied upon for self reporting @xcite . we put forward a definition of engagingness that is useful for our goal: to create an evaluation tool that will improve dialogue model performance. to ground our definition, we start with the dictionary. dictionary definitions identify three consistent aspects of the quality of being engaging: attention, interest, and participation (merriam-webster, 2023; @xcite . a reply is engaging when it gets the user's attention, interests the user, and incites participation from the user. these three aspects form the core of what we expect an engagingness metric to measure. we propose the following five subdimensions of engagingness: ## meep: metric for engagingness evaluation using prompting we design several categories of prompts. a straightforward naive prompt acts as the basis against which we compare our more developed prompt sets. we adapt dialogue annotator directions into prompts for the human-directed prompt set and our own proposed sub-dimensions of engagingness form the third prompt set. for each prompt set, we create a version casting the llm in a role. we experiment with prompts in english as well as spanish and chinese. naive prompt (naive) we use the naive prompt as a baseline comparator from which to develop more performant prompts. @xcite report best performance from the least constrained prompting method, implying that this naive prompt approach is valid as its own path of inquiry, and a similar prompt is used in @xcite . human-directed prompts (hd) since llms are developed to approximate natural language and natural linguistic interaction, we theorize that an evaluation prompt styled after the same instructions that human annotators were given should lead to increased performance over the naive prompt. prompts with our proposed dimensions of engagingness (meep) we incorporate each of our five subdimensions of engagingness into a word or phrase in order to form the definition of engagingness within the prompt. this is demonstrated in figure 3 . role assignment is theorized to increase the quality of the responses generated by dialogue models @xcite , and we explore its utility in dialogue response evaluation. we incorporate role assignment using methods specific to the model used. when possible, we make the assignment directly in the api call, using the "system" option. if this method is not supported by the model, the role assignment is prefixed to the prompt text. translation ( †) we translate a selection of our prompts from english into spanish and chinese, to match the language of the dialogues evaluated. spanish text was manually translated, and chinese translations are with gpt-4 and checked by a human translator for quality. ## experiments we first test our prompt combinations using two llms on the english-language fed dataset. five examples from these tests are randomly selected, on which we perform a qualitative analysis. we then select the best performing prompts to test over six multilingual turn-level datasets, two in each of english, spanish, and chinese. for these tests, we evaluate the prompts in english as well as the versions translated into the language of the dataset. the scores reported by each llm/prompt style pair over six datasets are correlated with the humanannotated scores to evaluate the proposed methods and four strong baseline methods. a final set of experiments evaluates the performance of the highestperforming prompts on dialogue-level datasets in english and chinese and correlates with humanannotated scores. these are compared with the strongest baseline method from turn-level testing. ## discussion insights from our experiments showcase the power of our proposed llm-based evaluator as follows: ## conclusion and future work we propose a novel method for estimating engagingness in dialogue using llms as automatic evaluators. our method -meep -outperforms stateof-the-art baselines, and when used in prompts containing a 'such as' phrase with examples, leads to better correlation with human annotated scores. this performance is demonstrated in a multilingual context, using prompts and dialogues in english, spanish, and chinese and for both turn-level and dialogue-level evaluation. our findings indicate that there is promise in the evaluation of other complex dialogue qualities -such as a model's ability to provide emotional support -through a similar use of prompting with llms. we see opportunities for future work in the development and use of non-translated datasets in languages other than english, with annotations for a well-defined measure of engagingness. in the future, we would like to continue to explore lighter models like llama @xcite or orca @xcite with our method for a more energy-efficient approach (appendix e).