FFAE val: Evaluating Dialogue System via Free-For-All Ranking (2023)

Abstract: Evaluating open-domain dialogue systems is currently an open question. Automatic evaluation metrics have shown poor correlation with human assessment in dialogue generation tasks. Human evaluation, which involves annotators for multi-dimension scoring, is trustworthy but time-consuming. In this work, we propose FFAEval, a reliable and efficient human evaluation framework using Free-For-All ranking approach. By sharing the dialogue history, the framework enables annotators to converse with multiple dialogue systems simultaneously in a single-blind, multi-turn manner. The subsequent free-for-all allows annotators to select the most favourable model in each turn from among all the participating dialogue systems. The final performance of each model is represented by calculating the TrueSkill score derived from the free-for-all competition. Our empirical study on English and Chinese dialogue systems demonstrates that FFAEval achieves a strong correlation with score-based human assessment compared to existing evaluation methods. We further prove the efficiency and stability of our framework in additional experiments. The source code and data are available on Github.

## introduction designing reliable and efficient evaluation methods serves as a beacon for the improvement of opendomain dialogue systems. it becomes increasingly challenging in this era dominated by dialogue systems that are typically built upon generative large language models (llms) @xcite . these systems generate openended responses from various perspectives, each ## ffaeval framework in general, ffaeval treats each multi-turn conversation as a multiplayer competition, in which participating dialogue systems seek to gain recognition from human users. ffaeval comprises three main components: (1) a human-bot interaction mechanism between evaluators and dialogue systems to obtain the competition ranking for each conversation; (2) a dialogue history sharing mechanism to ensure a fair context for response generation; (3) a global ranking scoring algorithm to produce the global ranking by synthesizing all the historical competition rankings. this rest of this section will introduce these modules in detail. the overall framework of ffaeval is demonstrated in figure 2 . ## experiments we conduct extensive experiments to verify the effectiveness of ffaeval in evaluating generative dialogue systems. ## analysis we analyze the characteristics of ffaeval and the feasibility of each designed components. ## related work currently, the evaluation methods for dialogue systems fall into three categories, which is described in detail below. reference-based automatic metrics require human-written references as a gold standard. bleu @xcite calculates the n-gram precision of the model-generated responses using human references. meteor @xcite and rouge @xcite improve bleu by involving stems and synonyms in result calculation and replacing n-gram precision with recall, respectively. adem @xcite trains a recurrent neural network to predict the quality of system responses. bertscore @xcite leverages bert to extract the features of system response and reference, which are then used to compute cosine similarity. reference-free automatic metrics have proliferated in recent years. fed @xcite uses dialogpt and pre-designed questions to evaluate multiple dimensions of dialogue systems. usr @xcite trains several models as sub-metrics to measure different aspects of dialogue. dynaeval @xcite leverages a graph convolutional network to model the holistic quality of dialogue. chatmatch @xcite uses flexible scoring metrics to analyse dialogue records generated by bot-bot conversations. gptscore @xcite employs generative pre-trained models and a prompt framework to evaluate generated texts. human assessments involve annotators to provide subjective scores or preferences. static evaluation uses dialogue context to generate single responses, which are then evaluated using multidimensional scoring @xcite or preference selection @xcite . spot the bot @xcite employs the concept of the turing test to evaluate dialogue systems and uses survival analysis to obtain fine-grained features. smith et al. @xcite compare the sensitivity of various comparative methods for evaluating dialogue systems. liu et al. @xcite evaluate multiple dialogue systems in an interactive setup and require annotators to select one or more appropriate responses. chatbot arena @xcite evaluates llms side-by-side in a single-blind manner and calculates the leaderboard using the elo ranking algorithm. using likert scales to rate chatbots after human-bot interaction is currently the standard method @xcite @xcite . however, this method places a high cost on dialogue generation and evaluation. existing comparative methods either evaluate using low-quality bot-bot dialogues or require a long time to reach a convergent ranking. our framework expedites convergent processes while ensuring the reliability of evaluations. ## conclusion in this paper, we propose a reliable and efficient human evaluation framework using free-for-all ranking, named ffaeval. this framework enables annotators to interact with multiple dialogue systems simultaneously and make a preference selection to credit the most favourable model in each turn of conversation. the final leaderboard is calculated using the results of free-for-all matches and the trueskill ranking algorithm. our experiments demonstrate that ffaeval achieves a strong correlation with the gold standard score. besides, we present that ffaeval outperforms other baselines in time efficiency and is stable towards match order. in the future, we plan to introduce a matchmaking process for the free-for-all to investigate the scalability and flexibility of our framework while handling a large number of models.