D i QAD : A Benchmark Dataset for Open-domain Dialogue Quality Assessment (2023)

Abstract: Dialogue assessment plays a critical role in the development of open-domain dialogue systems. Existing work are uncapable of providing an end-to-end and human-epistemic assessment dataset, while they only provide sub-metrics like coherence or the dialogues are conversed between annotators far from real user settings. In this paper, we release a large-scale dialogue quality assessment dataset (DiQAD), for automatically assessing open-domain dialogue quality. Specifically, we (1) establish the assessment criteria based on the dimensions conforming to human judgements on dialogue qualities, and (2) annotate large-scale dialogues that conversed between real users based on these annotation criteria, which contains around 100,000 dialogues. We conduct several experiments and report the performances of the baselines as the benchmark on DiQAD. The dataset is openly accessible at https://github.com/yukunZhao/Dataset_Dialogue_quality_evaluation .

## introduction open-domain dialogue system (ods) is quite popular in artificial intelligence @xcite @xcite , especially with the remarkable performance achieved in large language models (llms) @xcite @xcite . dialogue assessment is critical to the development of open-domain dialogue systems @xcite as it guides what constitutes good conversational dialogues. however, open-domain dialogue assessment still remains a challenging task @xcite . on the one hand, open-domain dialogue assessment is complicated and costly. a straightforward solution is the human-based method, i.e., recruiting human evaluators to interact with the dialogue systems and provide their 姚明是什么时候退役的? ## open-domain dialogue quality assessment this paper treats open-domain dialogue quality assessment as a critical task, which differentiates higher quality dialogues based on real users' dialogues, to guide the future dialogue generation towards more satisfying and human-epistemic responses. to this end, the quality assessment criteria are required to reflect higher quality for human cognition as much as possible. the human demands upon dialogue quality are usually regarded as entailing several fine-grained dimensions @xcite . inspired by a fully analyzed human-epistemic evaluation (finch and choi, 2020), we adopt the following 6 dimensions to establish our quality assessment criteria (see examples in based on the above dimensions, we conduct a 3-scale (0, 1, 2) quality assessment. a dialogue is set to at least answer the user's question; thus, the quality is at least a binary category of 1 and 0 to distinguish whether the user's question is resolved or not. furthermore, we aim to identify high-quality dialogues that guide future dialogue generation towards higher quality, like humans. to achieve this, we introduce an additional category of 2 indicating a high-quality rating. 1 the quality assessment includes turn-level and dialogue-level evaluation. we provide dialoguelevel quality evaluation criteria. specifically, we assess the quality of dialogue according to the following instructions (the examples are shown in r: perhaps you remain affiliated with your ex. when you break up with your ex completely, she would not be upset anymore. 1 give only one reason that do not provide complete information q: which products are manufactured by foxconn? r: foxconn manufactures electronic products including the iphone, macbook, dell, hp computers, and other consumer electronics. r: foxconn manufactures electronic products for major american, canadian, chinese, finnish, and japanese companies. it is a multinational electronics contract manufacturer ... 2 provide the all desired information. the user, as shown in the third case in in the next section, we introduce how we build the diqad based on the above criteria. ## diqad: a large-scale dataset for dialogue quality assessment in this section, we describe how we construct the diqad based on the above annotation instructions in detail. we first collect amounts of real dialogues from an online platform. then, we recruit experienced annotators to annotate these dialogues. in the last part of this section, we will show the detailed statistics of diqad. ## experiments in this section, we conduct and illustrate benchmark experiments on diqad, including examining the overall performance of benchmark models ( §5.2), conducting a detailed analysis based on the best benchmark model ( §5.3), comparing to other reference-free evaluation methods ( §5.4), and performing hyperparameter analysis ( §5.5). ## conclusion in this paper, we conduct a human-epistemic dialogue quality assessment on real user conversations. we release a large-scale dataset diqad for opendomain dialogue quality assessment, which contains around 100k dialogues and 0.8 million utterances in 6 domains. we conduct several benchmark experiments on diqad to study the performances of classical, transformer-based models and large language models on the dialogue quality assessment task. we hope the release of this dataset can inspire researchers in the community who lack such real and large-scale datasets, to evaluate real user dialogues to guide further improvement for the language models' generation.