SLIDE : A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluation (2024)

Abstract: The long-standing one-to-many problem of gold standard responses in open-domain dialogue systems presents challenges for automatic evaluation metrics. Though prior works have demonstrated some success by applying powerful Large Language Models (LLMs), existing approaches still struggle with the one-to-many problem, and exhibit subpar performance in domain-specific scenarios. We assume the commonsense reasoning biases within LLMs may hinder their performance in domain-specific evaluations. To address both issues, we propose a novel framework SLIDE (Small and Large Integrated for Dialogue Evaluation), that leverages both a small, specialised model (SLM), and LLMs for the evaluation of open domain dialogues. Our approach introduces several techniques: (1) Contrastive learning to differentiate between robust and non-robust response embeddings; (2) A novel metric for semantic sensitivity that combines embedding cosine distances with similarity learned through neural networks, and (3) A strategy for incorporating the evaluation results from both the SLM and LLMs. Our empirical results demonstrate that our approach achieves state-of-the-art performance in both the classification and evaluation tasks, and additionally the SLIDE evaluator exhibits better correlation with human judgements. Our code is available at https://github.com/hegehongcha/SLIDE-ACL2024.

## introduction open-domain dialogue generation is an important research topic in the field of natural language processing (nlp) @xcite @xcite @xcite . evaluating such dialogues, however, is challenging due to the * equal contribution † corresponding authors prevalent one-to-many issue where one conversational context may have multiple reasonable, yet semantically different responses. additionally, adversarial negative responses, which have word overlap with given contexts, also raise further challenges in adequately evaluating open-domain dialogue @xcite . existing evaluation methods centered on word-overlap, such as bleu @xcite , rouge @xcite , meteor @xcite ) and semantic-embedding based approaches (e.g., bertscore @xcite and bartscore @xcite ), evaluate responses via calculating the similarity between the generated response and contexts or gold references. therefore, they struggle to adequately address the one-to-many problem as a context may permit many different responses. recently, there are some novel approaches proposed to leverage the commonsense reasoning capabilities learned in large language models (llms) for a wide range of nlp tasks, such as dialogue generation, evaluation and sentiment analysis @xcite @xcite @xcite . however, there are still some limitations of llm-based evaluation. for example, @xcite propose llm-eval to evaluate open-domain dialogue systems with a range of llms and identify numerous issues including the significant reliance on prompt phrasing, which may artificially limit the effectiveness of the metrics and the evaluation quality changed with prompt formulation, which makes scoring unreliable and unrobust. additionally, @xcite state that without additional support, llms alone do not constitute equitable evaluators. in addition, llms face significant challenges in generating accurate and dependable assessments when evaluating both positive and adversarial negative responses. although llms can classify and evaluate adversarial negative responses with a degree of proficiency, their performance significantly diminishes when dealing with multiple positive responses, which is the one-to-many nature of open-domain dialogue, a reasonable response that matches well to context could have significantly different semantics to its reference. as for small task-specific model (slm), which is cost-effective alternatives, reveal a propensity towards positive responses. nonetheless, the deployment of slm is challenging, as they require extensive training data. this requirement often leads to issues related to data scarcity and a lack of sufficient examples covering a wide range of topics within open-domain dialogue, presenting obstacles to their effective implementation and scalability. therefore, we combine a slm with llm to solve the one-to-many issue, and we also resolve the data scarcity problem through data augmentation via llms. initially, we train a slm using contrastive learning to minimise the cosine distance between the positive responses and the context embeddings, and enlarge the distance between the embeddings of adversarial negative responses and the context embeddings. after fine-tuning with contrastive learning, these compact models can discern between positive and adversarial negative responses adequately. subsequently, we divide the representations of the responses into two categories: robust and non-robust vectors. through contrastive learning, we only retain robust vectors for subsequent steps while excluding non-robust ones. following this, we proceed to calculate the contextual-response distance, and obtain a probability value from the classification model. finally, an integrated score is composed by combining these two values as the final evaluation score of the slm model. ultimately, the evaluation score produced by the slm and the score from the llm is further integrated as the final evalu-ation score, slide, used for evaluation. a series of experiments are conducted to analyse the effectiveness of our proposed framework, including a classification task and an evaluation task. for the classification task, we use the slm to classify the positive and adversarial negative responses from the dailydi-alog++ dataset. experimental results demonstrate that our slm achieves state-of-the-art (sota) performance, even when compared to the results of some llms (e.g., gpt-3.5). we additionally note that the slm achieves higher accuracy in positive samples, whilst the llm performs better for negative example responses. for the evaluation task, slide also shows a good correlation with human scores. our contributions can be summarised as the following: ## conclusion in this paper, we introduce a novel automatic evaluation metric (slide) that integrates slm with llms for open-domain dialogue evaluation. our approach involves initially training a slm through iterative contrastive learning stages, followed by leveraging the combine strengths of slm and llms to create a superior evaluation metric. this metric exhibits enhanced correlation with human judgments in comparison to those derived exclusively