A Human-machine Collaborative Framework for Evaluating Malevolence in Dialogues (2021)

Abstract: Conversational dialogue systems (CDSs) are hard to evaluate due to the complexity of natural language. Automatic evaluation of dialogues often shows insufficient correlation with human judgements. Human evaluation is reliable but labor-intensive. We introduce a human-machine collaborative framework, HMCEval, that can guarantee reliability of the evaluation outcomes with reduced human effort. HMCEval casts dialogue evaluation as a sample assignment problem, where we need to decide to assign a sample to a human or a machine for evaluation. HMCEval includes a model confidence estimation module to estimate the confidence of the predicted sample assignment, and a human effort estimation module to estimate the human effort should the sample be assigned to human evaluation, as well as a sample assignment execution module that finds the optimum assignment solution based on the estimated confidence and effort. We assess the performance of HMCEval on the task of evaluating malevolence in dialogues. The experimental results show that HMCEval achieves around 99% evaluation accuracy with half of the human effort spared, showing that HMCEval provides reliable evaluation outcomes while reducing human effort by a large amount.

## introduction conversational dialogue systems (cdss) are often trained to generate responses given unstructured, open-domain dialogues. evaluation of cds responses has drawn broad attention due to its crucial rule for cds development @xcite . broadly speaking, there are two approaches to perform dialogue evaluation: automatic evaluation and human judgements @xcite . automatic evaluation metrics such as appropriateness @xcite , engagement @xcite , are efficient but have low agreement with human judgements due to the diversity of responses @xcite , especially for word-overlap based metrics, such as bleu @xcite and rouge @xcite . more recently, training based methods, e.g., adem @xcite , ruber @xcite and contextualized methods, e.g. @xcite , have been shown to have better agreement with human judgements. however, these methods are still not reliable enough: the pearson correlation with human judgments is 0.44 for appropriateness @xcite and 0.55 for relevance @xcite . to guarantee reliability of evaluation outcomes, our current best practice is to use human judgements. in terms of most evaluation aspects, e.g., appropriateness @xcite , coherence @xcite and empathy @xcite , human judgements simply show the highest reliability. obviously, human judgments are more labor-intensive than automatic evaluation @xcite . the flaws of automatic evaluation and the lack of speed and scalability of human evaluation limits the speed at which the community can develop more intelligent cdss. for example, as part of the daily research and development cycle of cdss, we need to change the model design and retrain the model multiple times, on a daily or even hourly basis. even if there is a minor change, we need to verify its performance again each time. for another example, cds leaderboards are very popular recently as a means to provide platforms for fair comparison @xcite . there are usually dozens of models to evaluate, and new models are introduced everyday. practical scenarios like the above two call for dialogue evaluation methods that are both reliable and efficient. in this paper, we propose the human-machine collaborative evaluation (hmceval) framework for dialogue evaluation with the aim of balancing reliability and efficiency. hmceval formulates the dialogue evaluation task as a sample assignment problem, i.e., if the machine can provide accurate outcomes, most evaluation samples should be assigned to the machine; otherwise, we should assign more samples to human evaluators. as shown in figure 1 , automatic evaluation has low reliability although the efficiency is high; human judgement has high reliability but it is labor-intensive; hmceval beats the previous two methods in balancing reliability and efficiency. finding a good balance between reliability and efficiency is non-trivial as the two desiderata are often in conflict with each other. usually, reliability is improved at the expense of efficiency @xcite . there are three main modules in human-machine collaborative evaluation (hmceval), namely the model confidence estimation (mce) module, the human effort estimation (hee) module, and the sample assignment execution (sae) module. first, the mce module measures the confidence of predicted evaluation for each dialogue response based sample. our implementation of mce is based on three estimation methods, namely, bert based maximum class probability (mcp), trust score (ts) @xcite , and true class probability (tcp) @xcite . ts and tcp have originally been introduced for images; we add a bert layer to expand it to dialogues. second, the hee module estimates the effort. our implementation is based on annotation time cost prediction by dialogue-related and worker-related features. third, the sae module decides whether a dialogue response sample should be assigned to a human or a machine for evaluation by maximizing the confidence and minimizing the (human) effort. we implement the module by integer linear programming (ilp). we demonstrate the effectiveness of hmceval on dialogue malevolence evaluation @xcite . the main reason we choose this particular task is that dialogue malevolence is highly related to social good @xcite , which is of vital importance for cdss, but it is hard to evaluate because of the need of deep semantic understanding @xcite . we carry out experiments on the recently introduced malevolent dialogue response detection and classifying (mdrdc) dataset @xcite . our results show that the proposed hmceval framework significantly surpasses machine evaluation and human judgement in terms of balancing reliability and effort. hmceval achieves around 99% evaluation accuracy (compared to human evaluation) with as much as half of the human effort saved. the results demonstrate that hmceval can be used for reliable and efficient evaluation of cdss since the accuracy is high and the effort is significantly reduced compared to fully human evaluation. to address the problem of low agreement with human judgments, learning based methods have been developed @xcite . @xcite propose adem to evaluate the appropriateness of responses. @xcite propose ruber, which shows better agreement with human judgments than adem. ruber is designed for relevance and similarity by blending relevance between the generated response with human ground truth and context. several methods utilize pretrained language models such as bert for automatic evaluation. @xcite propose contextualized ruber, which outperforms ruber. similarly, a predictive engagement metric is built by utilizing user engagement score @xcite ; quality is evaluated by transformer based language models without reference response @xcite . the above methods cover more aspects and integrate linguistic features @xcite , thus the agreement with human judgement is higher than most word-overlap based methods. however, for most of the metrics, the model performance still has space to improve, for instance, the accuracy of engagement is 0.76 @xcite . our proposed hmceval framework could be applied to these metrics and improve general evaluation reliability with an acceptable amount of human effort. human judgement is applied in common evaluation aspects including fluency, consistence, relevance, appropriateness, coherence, quality for cdss (finch and choi, 2020). it is reliable, yet expensive and time intensive, especially for large scale evaluation @xcite . in order to guarantee reliability, agreement among different workers is needed, which makes the high effort problem more severe @xcite . unlike the methods listed above, the hmceval framework specifically aims to balance reliability and human effort for the evaluation of cdss. ## conclusion and future work in this work, we have introduced a human-machine collaborative evaluation framework (hmceval) for reliable and efficient cds evaluation. experiments on the task of evaluating malevolence in dialogue responses show that hmceval can achieve around 99% reliability with half human effort spared. a limitation of hmceval is that given 50% samples assigned to a human, 1.1-1.5% samples are evaluated inaccurately. this is due to contexts that consist of a small number of turns, or high confidence for some dialogues where language is used in a non-literal way. although hmceval could be generalized to several evaluation metrics of cds, e.g., bert-based ruber and bert-based engagement, for score-based metrics, suitable confidence estimation is required. in the future, we seek to improve the model confidence and human effort estimation by considering better neural architectures and more factors; we also plan to conduct a comprehensive and reliable analysis of the performance of current state-of-the-art cds models by applying hmceval to various evaluation aspects.