D yna E val: Unifying Turn and Dialogue Level Evaluation (2021)

Abstract: A dialogue is essentially a multi-turn interaction among interlocutors. Effective evaluation metrics should reflect the dynamics of such interaction. Existing automatic metrics are focused very much on the turn-level quality, while ignoring such dynamics. To this end, we propose DynaEval, a unified automatic evaluation framework which is not only capable of performing turn-level evaluation, but also holistically considers the quality of the entire dialogue. In DynaEval, the graph convolutional network (GCN) is adopted to model a dialogue in totality, where the graph nodes denote each individual utterance and the edges represent the dependency between pairs of utterances. A contrastive loss is then applied to distinguish well-formed dialogues from carefully constructed negative samples. Experiments show that DynaEval significantly outperforms the state-of-the-art dialogue coherence model, and correlates strongly with human judgements across multiple dialogue evaluation aspects at both turn and dialogue level.

## introduction modern dialogue systems @xcite @xcite leveraging large-scale language model pre-training @xcite are capable of generating fluent and contextually relevant utterances. yet, they still face difficulties in mimicking human conversations in the sense that they lack certain conversation-level attributes, such as coherence @xcite , consistency @xcite , diversity @xcite and engagement @xcite . one of the main reasons is the dearth of effective dialoguelevel evaluation mechanisms to guide the studies and to monitor progress. commonly used static metrics, such as bleu @xcite , me-teor @xcite ) and rouge @xcite , correlate poorly with human judgements @xcite rendering them unsuitable for dialogue evaluation. while some recent automatic dialogue evaluation metrics @xcite @xcite demonstrate strong correlations with human judgement at the turn-level, they only focus on context-response pairs without explicitly modeling the interaction over an entire dialogue. to perform dialogue-level evaluation, we need to rely on the aggregation of turn-level scores over the dialogue as a proxy for a dialogue-level score. furthermore, a recent study by @xcite found out that even though state-ofthe-art chatbots outperform humans across multiple turn-level evaluation criteria, such as interestingness, engagement and specificity, their dialoguelevel ratings like coherence, likability and diversity are still far below human level. this further reinforces the idea that turn-level quality evaluation may be insufficient to assess the performance of open-domain dialogue systems. in this work, we address the problem of automatic open-domain dialogue evaluation by focusing on the quality of an entire dialogue. this is a departure from the way we frame the problem as a weakly supervised next sentence prediction @xcite or language modeling tasks @xcite for context-response pairs. to this end, we need to answer two important questions: (1) how to effectively represent the entire dialogue? (2) how to incorporate this dialogue-level knowledge into our evaluation framework? we propose dy-naeval to provide meaningful dialogue-level representation with explicit modeling of the interactive dynamics among interlocutors, for a unified turn and dialogue level quality assessment. the main contributions of this work include: (1) the unified turn and dialogue level evaluation represents a departure from turn-level evaluation scheme ; (2) dynaeval is one of the first few metrics where dialogue level dynamics is considered with structured graph representation. (3) empirical results show that dynaeval outperforms the stateof-the-art dialogue coherence model and strongly correlates with human judgements at both turn and dialogue level. ## dynaeval framework dyaneval represents an integration of several ideas. it takes advantage of the structured graph representation of dialogues, useful information on the utterance and speaker level interaction. it is motivated by dialogue coherence modeling. in this paper, we only consider dyadic dialogues, but the formulation can be easily generalized to multi-party conversations. formally, let a and b denote the two speakers participating in the dialogue. a dialogue, d, consists of a sequence of n utterances, [u a 1 , u b 2 , . . . , u a n-1 , u b n ] foot_1 . let d represent the negative dialogue sample obtained via various sampling strategies described in section 3.5. figure 1 illustrates the learning process of dy-naeval in four steps foot_2 : (1) deriving contextualized representation, e i , for utterances within d. (section 3.1). (2) constructing the directed dialogue graph. the nodes are initialized with e i and the edges between node pairs represent the speaker and temporal dependencies (section 3.2). (3) generating utterance-level graph representation, h i , via feature transformation to aggregate useful contextual information from all connected neighbours to the current node (section 3.3). (4) producing a dialogue-level score, which indicates whether d is preferred over d (section 3.4). ## experiments in this work, we consider two experiment settings to assess the effectiveness of dynaeval. the first setting (section 4.2) is similar to the studies on dialogue coherence @xcite where accuracy score is applied to evaluate its discrimination capability in distinguishing original dialogues from negative samples. the second setting (section 4.3) is to evaluate its dialogue-level and turn-level judgement capability via correlation analysis on the human-chatbot conversational datasets. the domain of the evaluation set is different from that of human-human conversation datasets that dyaneval is trained on. ## conclusion & future work dynaeval serves as a unified framework for both turn and dialogue level evaluation in open-domain dialogue. it provides meaningful representations that incorporate information reflecting various important dialogue attributes. its explicit modeling of speaker and utterance level interaction leveraging gcn has been proven beneficial for the evaluation task. lastly, the error analysis in section 4.3 sheds light on how dynaeval can be further improved. dynaeval can also be combined with the specialized turn-level metrics, such as those targeting fluency and engagement, to fully approximate the interactive human evaluation process.