Combining Multiple Metrics for Evaluating Retrieval-Augmented Conversations (2024)

Abstract: Conversational AI is a subtype of Human Computer Interaction that has gained wide adoption. These systems are typically powered by Large Language Models (LLMs) that use Retrieval Augmented Generation (RAG) to infuse external knowledge, which is effective against issues like hallucination. However, automatically evaluating retrieval augmented conversations with minimal human effort remains challenging, particularly in online settings. We address this challenge by proposing a lexical metric, and a novel method for combining it with other metrics, including semantic models. Our approach involves: (1) Conversational Information Utility (CIU), a new automated metric inspired by prior user studies on web search evaluation, to compute information overlap between conversation context and grounded information in an unsupervised, purely lexical way; and (2) a generalized reward model through Mixture-of-Experts (MoE-CIU) that dynamically ensembles CIU with other metrics, including learned ones, into a single reward. Evaluation against human ratings on two public datasets (Topical Chat and Persona Chat) shows that CIU improves correlation against human judgments by 2.0% and 0.9% respectively compared to the second best metric. When MoE is applied to combine lexical and learned semantic metrics, correlations further improve by 9.9% and 5.0%, suggesting that unified reward models are a promising approach.

## introduction conversational ai is a specific type of human-computer interaction that has been widely studied in recent years @xcite , leading to the development of multi-purpose chat assistants (e.g. chatgpt, claude) based on large language models (llms). however, as more customers interact with such assistants, addressing limitations like hallucination, factual consistency, prompt brittleness and controllability has gained more attention @xcite . one widely-adopted solution is retrieval augmented generation (rag), which allows choosing a context document (d context ) to ground llm responses, and increase truthfulness with respect to the source document @xcite . our work focuses on the task of automatically assessing the quality of retrieval-augmented responses in knowledge-grounded conversations. by examining both the context and the response, we estimate the degree to which the retrieved document was used in generation, in order to identify uninformative or inconsistent responses. our approach is designed for real-time use, where using a large model may be infeasible. compared to offline tasks, online evaluation (e.g., live monitoring of defects) requires efficient solutions. recent work utilizes llms, either through prompt engineering or fine-tuning, to automatically predict evaluation metrics and reduce dependency from human annotators @xcite . despite demonstrated potentials, a large number of parameters, high latency, and potential legal issues significantly limits deploying llm-based solutions for live traffic monitoring. as an alternative, we propose an approach that combines much simpler and scalable metrics to predict user ratings, or potentially other business metrics. our approach can also support offline evaluations, and is relevant to recent trends in reinforcement learning from human feedback (rlhf), which aligns llm responses toward human preferences @xcite @xcite . early attempts on automatic dialog evaluation relied on existing metrics (e.g. bleu) from machine translation literature to evaluate assistant conversations against 'gold' conversations @xcite . however, defining the full space of 'gold' conversations is infeasible due to the nondeterministic nature of dialogs and many existing works simply penalize any response that slightly de-viates from 'gold' @xcite . on the other hand, learned, semantic conversational quality metrics trained on labeled data tend to show higher correlations against human judgements than exact word-overlap metrics, because word embeddingbased approaches can compute overlap in a 'soft' way that accounts for lexical variation, e.g. , @xcite . however, such 'soft' approaches also suffer from different issues, such as over-fitting, performance degradation on longer inputs, or learning similar representations for antonyms. to address these gaps, we ask: is it feasible to unify multiple independent metrics into a single reward? to answer this question, we investigated two research questions: q1 what is the most effective and robust standalone metric (whether lexical or learned) that aligns with human ratings from open-source knowledge-grounded conversations? q2 given a set of independent metrics, how much improvement does a unified reward model gain compared to the best standalone metric? for q1, previous studies on 'exact' word overlap metrics showed they correlate poorly with human preferences, such as question answering accuracy @xcite and response appropriateness @xcite . to address this, we compare our newly proposed lexical metric, conversational information utility (ciu), which is inspired from user-centric studies on web search evaluation @xcite against existing metrics. a key insight is that for the user to gain useful information, they must ask a series of questions, or make statements, that cause the conversational system (or human partner) to respond with information overlapping with d context . our main novelty is how ciu quantifies information overlap to reward relevancy, information novelty and conciseness while penalizing repetitive information and high user effort. experiments validate that ciu improves correlation against human ratings by 2.0% and 0.9% against the second best metric on topical chat @xcite and persona chat @xcite datasets for predicting overall ratings. for q2, we experiment with different ensemble learning strategies to (1) validate whether previously identified strong metrics are considered as strong predictors (metrics); (2) demonstrate the superiority of an unified reward compared to any standalone metric. experimental results in feature selection ratio show that ciu is selected 76.4% across 17 different feature selection approaches, which justifies our findings on q1. when mixtureof-experts (moe) @xcite was applied, the resulting moe-ciu model further improved correlation with human ratings by 9.9% and 5.0% on topical chat and persona chat compared to the best standalone metric. in summary, our contributions are: learned metrics one popular approach is to utilize pretrained contextual embeddings from transformer models to compute a similarity score between two texts. for example, bertscore @xcite computes a token-level similarity matrix and re-weights the scores based on idf scores to boost signals from more novel matches. adem @xcite uses a hierarchical rnn encoder to predict human-annotated ratings on twitter data. while adem requires human judgments, ruber-bert @xcite uses an unsupervised negative sampling strategy to train a model that measures information relatedness between query and response. usr @xcite ) is another transformerbased model that is shown effective for evaluating model generated responses. to evaluate usr, the authors sampled a small number of conversations from topical chat and persona chat datasets to annotate several useful ratings (e.g. overall rating of responses), which we adopt for our evaluations. mixture-of-experts models when there are multiple representations of the same input, e.g. complementary representations computed by different expert modules, the mixture-of-experts (moe) approach @xcite ) can take these independent knowledge sources and conditionally combine them into a joint representation @xcite . it does so by training a gating mechanism that dynamically assigns weights to the experts, depending on the input @xcite . moe has been shown to be effective in various settings such as combining support vector machines @xcite , hierarchical networks @xcite and named entity recognition @xcite . for our use case, each expert is a representation of different metrics. by training a gating network to dynamically weight experts per instance, we expect moe to improve over heuristic-based feature combination strategies, such as sum or mean of different metrics @xcite . ## proposed metrics and models we define our task and usefulness ratings, followed by details on ciu and moe-ciu approaches. ## public datasets with human ratings we used two publicly available dialog corpora to evaluate our approach. topical chat topical chat @xcite , contains 11k conversations that are grounded to web articles (d context ). for human ratings, we use a subset of more recent and reliable ratings obtained on the official test split @xcite , which contains 360 samples. from multiple available ratings, we selected two ratings that are most relevant for our scope: persona chat persona chat @xcite is another popular dataset that contains knowledgegrounded conversations on different personas (d context ). similar to topical chat, mehri and eskenazi (2020) released a more recent human annotations on 300 test samples. for consistency, we will choose the exact same type of ratings for our evaluation. ## experimental settings we first present an overview of our selected baseline metrics and evaluation criteria. for q1, each metric performance is (1) evaluated independently. then for q2, we experiment with two settings for metric unification: (2) static combination where optimal features (metrics) are pre-selected from existing feature selection algorithms; (3) dynamic combination that utilizes moe to automatically learn and weight all input features. ## main results we present the results on each public dataset, followed by ablation study and error analysis. ## conclusion we introduced ciu, a novel utility metric for assessing the quality of retrieval augmented conversations. based on our experiments on two popular retrieval augmented (a.k.a. knowledge-grounded) conversation corpus, we conclude that ciu was the best metric among other lexical baselines. although ruber-bert surpassed ciu performance on persona chat, considering the complexity of ruber-bert (e.g., training and inference), ciu is still an easy-to-use metric that can achieve similar results with no training, which answers q1. for q2, we demonstrated the potentials of unifying multiple independent metrics into a single reward signal without any llm dependency. this was achieved through our moe-ciu model, and experiments confirm its effectiveness over any standalone metric. insights from this study suggest promising directions for applying moe-ciu as a proxy for an unified reward signal to optimize.