MERCY : Multiple Response Ranking Concurrently in Realistic Open-Domain Conversational Systems (2023)

Abstract: Automatic Evaluation (AE) and Response Selection (RS) models assign quality scores to various candidate responses and rank them in conversational setups. Prior response ranking research compares various models’ performance on synthetically generated test sets. In this work, we investigate the performance of model-based reference-free AE and RS models on our constructed response ranking datasets that mirror real-case scenarios of ranking candidates during inference time. Metrics’ unsatisfying performance can be interpreted as their low generalizability over more pragmatic conversational domains such as human-chatbot dialogs. To alleviate this issue we propose a novel RS model called MERCY that simulates human behavior in selecting the best candidate by taking into account distinct candidates concurrently and learns to rank them. In addition, MERCY leverages natural language feedback as another component to help the ranking task by explaining why each candidate response is relevant/irrelevant to the dialog context. These feedbacks are generated by prompting large language models in a few-shot setup. Our experiments show the better performance of MERCY over baselines for the response ranking task in our curated realistic datasets.

## introduction advancements of neural models @xcite @xcite has led to the vast continuous research on open-domain dialog systems. many deployed open-domain dialog systems rely on multiple response generators in order to address the variety of topics within a dialog. accordingly, response ranking is introduced as a major necessity for ranking different responses based on their quality @xcite @xcite . predominantly, rs models have been tested to distinguish between positive and synthetically gen-erated negative candidates following approaches such as random matching @xcite @xcite ), text-level manipulations (shuffling, replacing) @xcite , human-written adversarial irrelevant responses @xcite @xcite . these synthetically curated test sets are not sufficient representations of real-world inference time candidates that are generated by dialog models. @xcite demonstrated the effectiveness of training on machine-generated candidates from real user interactions over using synthetic candidates for response selection. however this data is not publicly available. in this work, we construct the first public dataset of human fine-grained rankings for responses generated by state-of-the-art dialog models in humanhuman dialog contexts, which we denote as static evaluation setup (ses). for our work we also leverage the recently collected interactive evaluation setup (ies) @xcite dataset, which includes human annotations for different machinegenerated responses within human-chatbot conversations. our intention is to conduct a survey to evaluate the performance and generalizability of state-of-the-art model-based reference-free ae and rs on our curated datasets that are closer to deployment time ranking scenarios, where one interlocutor is human and the other is a dialogue system. we show that in these realistic test cases, existing rs and ae models exhibit low performance. to overcome this issue and have a more reliable rs in real case scenarios we propose mercy, which pursues users' behavior of taking different candidates as input and predicting their rankings by relying on their comparable representations (see figure 1 ). there is strong evidence that relying on comparable representations is useful such as human preference modeling to improve the performance of llms @xcite . mercy also augments the input with feedback in the form of natural language that explains why or why not a response is relevant. @xcite introduced an instruction-tuned large language model (llm) to perform a variety of dialog tasks such as determining if a response is relevant or not. we follow a similar approach of prompting a llm to evaluate a response; however, we prompt the model to generate more detailed information by not only asking if a response is relevant but also why it is relevant. we refer to this generated output as feed-back. we leverage bloomz-175b @xcite to generate each candidate's feedback. we train mercy on the train split of ses and demonstrate that it is more accurate in real-case ranking scenarios in comparison to the best performing automatic metric finetuned on the same training set. considering multiple candidates together and augmenting responses with feedback both contribute to mercy's better performance. our contributions are summarized as follows: ## related work due to the vast number of ae/rs models, an indepth comparison of these metrics is critical. @xcite performed a comprehensive survey by comparing multiple ae metrics on publicly available evaluation testsets. in this work, we perform a similar survey of model-based referencefree ae/rs models on response selection testsets. in contrast to their evaluation testsets where responses are annotated on a likert scale, which can lead to annotator bias and could make it difficult for a model to predict the exact scores, we only need to evaluate the relative ordering from the predicted output of these methods. the response selection datasets we leverage are more realistic than previously proposed synthetically generated datasets. prior research proposed to use simple approaches such as random response selection @xcite , corrupting utterances by inserting, substituting and deleting random tokens @xcite , using the mask-and-fill approach @xcite for generating adversarial negative examples or collect human-written negative samples @xcite . previous work also suggest to augment dialog datasets with synthetically generated positive samples @xcite @xcite @xcite . in a study by @xcite , they demonstrated that using a human-chatbot dataset, where responses were generated by multiple response generators and then annotated by humans for training rs (response selection) models, led to improved performance compared to models trained on synthetically generated datasets. unfortunately, the dataset they used could not be made public due to privacy concerns, as it contained real-user dialogs. in contrast, our approach involves collecting a similar and realistic response ranking dataset, which we plan to release for future research purposes. in rs, most models score response candidates independently without considering them together. @xcite proposed a joint matching approach that concurrently accepts exactly four candidates as input and selects the only correct response using log-likelihood as the training objective. our rs model follows a similar training approach but can handle a variable number of responses for ranking. additionally, our metric stands out from previous work as it combines generated natural language feedback with multiple response candidates, providing the model with valuable information in a natural language format. feedback generation has been shown to be beneficial for improving language models. @xcite @xcite @xcite . @xcite al. (2022); hancock et al. (2019); scheurer et al. (2022) focused on improving response generation models using three types of human feedback: binary, modular and natural language. while these studies use natural language feedback collected via human annotation, our work generates feedback from large language models in a few-shot fashion and use them for the ranking task. the closest work to ours is @xcite , which trains an instruction-tuned large language model to conduct evaluation. however, this work treats feedback as a classification task asking if a response is relevant, while we prompt the model to output why it is relevant and accompany that with the candidate. ## data sets to conduct a comprehensive survey on ae/rs models, we look at three response ranking test sets each encompassing different properties: 1) type of dialog contexts, 2) type of candidates for ranking, and 3) type of conversational domains. dailydialog++ sai et al. (2020) composed a dataset consisting of manually created relevant/irrelevant responses for human-human dialog contexts taken from dailydialog @xcite . to create irrelevant responses, annotators were asked to write responses that share similar semantics with the dialog context yet are not acceptable. hence, in dailydialog++ both dialog histories and candidate responses are human-written. ses the responses from dailydialog++ may not match realistic inference time test sets where responses are machine generated. to deal with this, we collect the static evaluation setup (ses) dataset comprising of various model generated responses for contexts sampled from multiple human-human dialog datasets: dailydialog @xcite , blendedskilltalk (smith et al., 2020), personachat @xcite , @xcite . each dialog context contains 8 different responses generated by blenderbot @xcite , gpt2-xl @xcite fine-tuned on blended-skilltalk (smith et al., 2020), plato-2 @xcite , and plato-xl @xcite with different decoding mechanisms. the model training and decoding parameters are provided in section e in the appendix. we collect two sets of data where the rankings are eventuated from two groups of annotators: 1) in-house annotators familiar with the ranking task (ses_internal), 2) amazon mechanical turk (amt) workers (ses_amt). responses in ses_internal are annotated by two internal annotators on the scale of 0 (not an appropriate response) to 2 (a suitable response). we calculate the normalized mean score for each response in the range of 0 to 1 and assign label 1 to the response if its normalized score is greater than 0.5 or 0 otherwise. to better analyze the performance difference of ae/rs models, we remove turns where all the candidates are 1 or 0 and call it ses_internal_filtered. although rs model may face such all good or all bad candidates in realworld scenarios, we exclude them to not mislead the performance of rs with random candidate selection. due to the higher quality of annotations by internal annotators who are more familiar with the task, we leverage this dataset to test our proposed rs model versus baselines. the process of collecting ses_internal is slow due to an insufficient amount of annotators, therefore we use amt workers for faster data collection. for ses_amt dataset, 5 amt workers evaluate each response in the range of 1-5 indicating low-quality to high-quality responses. here we use more fine-grained ratings which allow us to check if amt workers understand the range of how good/bad a response can be. we get the median score of each candidate's ratings and normalize it in the range of 0-1. we assign 0/1 label similar to ses_internal. we remove turns with all good or bad responses, and call it ses_amt_filtered. the statistics of these datasets are shown in a closer look at the ies data shows that some good responses were not marked correctly by amt workers. this could be because amt workers may be taking into account factors besides relevancy when selecting a response such as engagingess. examples of these issues can be seen in section c in the appendix. in order to have a more fair comparison we sample 80 dialogs from ies and ask amt workers to reannotate each response on a scale of [1-5] similar to ses_amt setup, and denote this dataset as ies-v2. ies-v2 includes a part of ies dataset with more fine-grained annotations in a 1-5 scale that allows better training signals for the rs model. we compute the fleiss kappa for interannotator agreement and get a score of 0.41, which indicates moderate agreement. in this work, we use ies/ies-v2 data for only testing. ## method our proposed response selection model mercy evaluates multiple response candidates for a given dialog context simultaneously, and also leverages the feedback generated by llms for candidates. ## results we present results of comparing mercy against deb and mercy with feedback as additional input in ## conclusion we introduce mercy, an rs model that ranks responses by comparing multiple responses synchronously and leveraging natural language feedback. we demonstrate that feedback generated from a llm through a few-shot setup improves the performance of mercy. additionally we introduce the ses dataset, a more realistic rs dataset with human annotated machine generated responses and show the low performance of baseline ae/rs metrics on ses and other existing realistic response raking testsets.