F low E val: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows (2022)

Abstract: Despite recent progress in open-domain dialogue evaluation, how to develop automatic metrics remains an open problem. We explore the potential of dialogue evaluation featuring dialog act information, which was hardly explicitly modeled in previous methods. However, defined at the utterance level in general, dialog act is of coarse granularity, as an utterance can contain multiple segments possessing different functions. Hence, we propose segment act, an extension of dialog act from utterance level to segment level, and crowdsource a large-scale dataset for it. To utilize segment act flows, sequences of segment acts, for evaluation, we develop the first consensus-based dialogue evaluation framework, FlowEval. This framework provides a reference-free approach for dialog evaluation by finding pseudo-references. Extensive experiments against strong baselines on three benchmark datasets demonstrate the effectiveness and other desirable characteristics of our FlowEval, pointing out a potential path for better dialogue evaluation.

## introduction dialogue evaluation plays a crucial role in the recent advancement of dialogue research. while human evaluation is often considered as a universal and reliable method by the community @xcite , automatic dialogue evaluation metrics draw growing attention as they can assess dialogues with faster speed and lower cost @xcite @xcite . traditional word-overlap metrics, like bleu @xcite and meteor @xcite , lose some of their effectiveness in the dialogue setting as reliable references are hard to obtain @xcite . recent works tackle this problem by leveraging more sophisticated architectures @xcite ## actdial: a segment act dataset on open-domain dialogues we propose the new concept of segment act, extracting the core function of each segment in an utterance. we then crowdsource a large-scale opendomain dialogue dataset with our proposed segment act labels, called actdial. ## floweval: a segment-act-flow aware evaluation metric in this section, we describe the details of our proposed dialogue evaluation framework, floweval. floweval is implemented in three stages: segment act harvesting, retrieval, and assessment. ## conclusion in this work, we propose a consensus-based reference-free framework for open-domain dialog evaluation with segment act flows. from extensive experiments against the state-of-the-art baselines, our method can reach the best or comparable correlation with human evaluation. our segmentact-based methods complement well to previous semantic-meaning-focused methods, pushing the ceiling of correlations. moreover, the promise of our consensus-based framework encourages us to step further in the direction of dialog evaluation.