Just Adjust One Prompt: Enhancing In-Context Dialogue Scoring via Constructing the Optimal Subgraph of Demonstrations and Prompts (2023)

Abstract: The use of modern Large Language Models (LLMs) as chatbots still has some problems such as hallucinations and lack of empathy. Identifying these issues can help improve chatbot performance. The community has been continually iterating on reference-free dialogue evaluation methods based on large language models (LLMs) that can be readily applied. However, many of these LLM-based metrics require selecting specific datasets and developing specialized training tasks for different evaluation dimensions (e.g., coherence, informative). The developing step can be time-consuming and may need to be repeated for new evaluation dimensions. To enable efficient and flexible adaptation to diverse needs of dialogue evaluation, we propose a dimension-agnostic scoring method that leverages the in-context learning (ICL) capability of LLMs to learn from human scoring to the fullest extent. Our method has three key features. To begin with, rather than manual prompt crafting, we propose automatically generating prompts, allowing the LLM to observe human labels and summarize the most suitable prompt. Additionally, since the LLM has a token limit and ICL is sensitive to demonstration variations, we train a selector to finely customize demonstrations and prompts for each dialogue input. Finally, during inference, we propose to request the LLM multiple times with a subgraph of demonstrations and prompts that are diverse and suitable to maximize ICL from various human scoring. We validate the efficacy of our method on five datasets, even with a small amount of annotated data, our method outperforms all strong baselines. Code is available at https://github.com/iamlxb3/EMNLP2023-ADOROR.

## introduction although modern language models have made significant strides in language fluency, informativeness, and user understanding (taecharungroj, † corresponding author. ## related work in-context learning icl is the process by which models learn from context examples during forward propagation, without parameter updates. icl currently performs very well on traditional nlp tasks @xcite , but icl is still sensitive to how the prompts are written @xcite , the selection and the order of demonstrations @xcite . for automatic labeling, icl has shown good results and low cost on tasks such as word sense disambiguation, text summarization, and question generation @xcite , but using icl for dialogue evaluation is rare. ## problem formulation our goal is to construct a dialogue evaluator that automates human-like scoring. we denote the input of the dialogue evaluator as x, which may contain three parts: the conversation context c (usually contains several rounds of conversation), a response r immediately following the context, and the conversation background b (e.g., knowledge contained, persona information, etc.). dialogue evaluation is categorized into dialogue-level evaluation, where the evaluator scores the overall context c from different dimensions (e.g., topic depth, informative, etc.), and turn-level evaluation, where the evaluator scores the next round of response r succeeding the context c and the possible background b. given an input x, the evaluator is expected to output a score ŷ for a particular evaluation dimension. we adopt the reference-free setting for both turn-level and dialogue-level evaluation due to the one-to-many characteristic of reference-based methods @xcite . exhausting all potential responses for a reference is unfeasible. we denote the labeled dataset as d @xmath0, y 1 ), ..., (x n , y n )), where y is the average score of several human annotators. we define the set of inputs and labels for d train as d x train and d y train . given d x train ∼ p x and d y train , we aim to automate the scoring process of another dataset d x test from the same distribution p x , ensuring as high consistency as possible between the model's scoring d y test and the humans' scoring d y test . ## conclusion our automated dialogue scoring method allows the llm to learn from human labels using in-context learning (icl) in three stages. in the first stage, we let the llm observe human scoring examples, from which the llm induces suitable prompts for dialogue scoring. additionally, the order and selection of demonstrations are crucial for icl, and therefore, we train a demonstration/prompt selector using a new training set created by comparing the llm and the human scoring. during inference, we provide the llm with as diverse and suitable demonstrations and prompts as possible. our adoror approach is flexible and adaptable to new evaluation dimensions or criteria, requiring only a small amount of human-labeled data.