Bipartite-play Dialogue Collection for Practical Automatic Evaluation of Dialogue Systems (2022)

Abstract: Automation of dialogue system evaluation is a driving force for the efficient development of dialogue systems. This paper introduces the bipartite-play method, a dialogue collection method for automating dialogue system evaluation. It addresses the limitations of existing dialogue collection methods: (i) inability to compare with systems that are not publicly available, and (ii) vulnerability to cheating by intentionally selecting systems to be compared. Experimental results show that the automatic evaluation using the bipartite-play method mitigates these two drawbacks and correlates as strongly with human subjectivity as existing methods.

## introduction the performance evaluation of dialogue systems is a crucial and challenging research topic for the dialogue research community. the community recommends human evaluation as the primary evaluation method, which is the gold standard but is timeconsuming and costly. moreover, reproducing the evaluation results is mostly impractical due to the unavailability of maintaining identical evaluators or identical evaluation conditions. human evaluation is therefore unsuitable for evaluating daily updates of developing dialogue systems or comparing systems with non-public ones. thus, constructing a better automatic evaluation method, which is both highly reproducible and low cost, is desirable. in particular, automating interactive evaluation, not static evaluation such as bleu @xcite , is attracting an increasing interest as static evaluation cannot capture diverse aspects of dialogue systems @xcite . an interactive evaluation framework consists of two phases: first, collecting the dialogues in which the systems to be evaluated (hereinafter called evaluation targets) talk to others (hereinafter called dia-*both authors contributed equally to this paper. logue partners), then rating evaluation targets based on the quality of their utterances in the collected dialogues. regarding the collecting (i.e., automating dialogue partners), self-play and all-play-all (figure 1 (a ) and (b)) are the current promising methods; all-play-all collects dialogues among multiple evaluation targets, while self-play collects dialogues with itself. recently, @xcite have reported that all-play-all correlates with human evaluation strongly. however, all-play-all is not perfect and has at least two potential drawbacks: (i) the difficulty of comparison with publicly inaccessible systems and (ii) the vulnerability to cheating by choice of evaluation targets, i.e., with whom the evaluation target will talk (section 3). this paper addresses the above two drawbacks of the all-play-all method while maintaining the all-play-all method's high correlation with human rating. specifically, we propose the bipartite-play method, i.e., fixing and sharing a set of dialogue partners across studies as shown in figure 1 (c ) instead of assigning other evaluation targets as partners as shown in figure 1 (b) (section 4). the bipartite-play method offers (i) a fair comparison with publicly inaccessible systems as long as its developers use our method and (ii) prevention of cheating by an intentional choice of evaluation targets. our experiments show that the bipartiteplay method strongly correlates with humans as the all-play-all method while preventing the potential drawbacks in the all-play-all method. ## limitations of all-play-all method the all-play-all method enables effective dialogue collection for system comparison, as described in section 2. however, we point out that the current all-play-all method cannot handle the following two cases: first, when the group of evaluation targets includes unavailable systems. since allplay-all requires the collection of dialogues with all evaluation targets, it is impossible to compare systems that are not released or that cannot be run by many researchers due to such computational resources. second, when one attempts to boost their system's performance by deploying an unfair evaluation setting. our experiments (section 6) reveal that one can intentionally improve the automatic evaluation results of desired systems by choosing evaluation targets to be compared when using the all-play-all method. if these potential drawbacks can be overcome, existing automated evaluation methods could be enhanced to be more versatile and practical. ## proposed method: bipartite-play we introduce a new automatic dialogue collection method, called bipartite-play method, which updates the two aforementioned limitations. ## bipartite-play based evaluation framework subsequently, we introduce bipartite-play to the interactive dialogue evaluation framework. we combine the dialogue collection based on the bipartite-play method with fed @xcite , which is one of the modern and effective dialogue rating methods. ## experiments: system evaluation we show that the interactive automatic evaluation using the bipartite-play method correlates with humans as strongly as the all-play-all method, which has been reported to be an effective dialogue collection method but requires access to all evaluation targets. we first rank prepared evaluation targets by interactive human evaluation and then measure the correlation with the rankings by interactive automatic evaluations in the three dialogue collection methods: self-play, all-play-all, and bipartite-play. ## discussion the interactive automatic evaluation using the bipartite-play method correlates with humans as strongly as the all-play-all method, as described in section 6, without direct interaction with evaluation targets. additionally, we present another reason to use the bipartite-play method: one can sometimes cheat by choice of evaluation targets when using the all-play-all method. a: i love being in the "flow" -whether it's in speaking to a group of people or working to help facilitate meetings. b: what kind of work do you do? a: i'm a sales manager at a trading company. b: that sounds like a lot of fun. i bet you meet lots of interesting people. do you like it? a: i love it. i get to meet new people all the time. b: that's great! i'm glad you enjoy it. it sounds like you have a great job. a: thanks. i've got a lot to learn. b: it's always good to learn new things. what else do you like to do in your spare time? a: i like to play chess. how about you? b: i love chess as well. my favorite game to play is checkers. have you ever played that? a: no, i haven't . but i'd like to try. b: it's a great game. you should give it a try. if you like chess, you'll love it. ## conclusion in this paper, we proposed the bipartite-play method as a dialogue collection method. the bipartite-play method can address the impossibility of comparison with publicly inaccessible systems and the vulnerability to cheating by intentional choice evaluation targets to improve the all-playall method. for the proposed method, no dialogue with evaluation targets is required, thereby facilitating system comparison across studies and possibly enabling comparison with inaccessible systems. our experiments showed that, compared with the evaluation using the all-play-all method, the automatic evaluation using the bipartite-play method correlates just as strongly with humans. although we formed a set of the bipartite-play method's partner systems for the experiments considering its diversity of architectures and training data, it may still have some vulnerabilities. in future work, we will explore the property of the botbot dialogue further and refine the set of partner systems for the bipartite-play method.