Shades of BLEU , Flavours of Success: The Case of M ulti WOZ (2021)

Abstract: The MultiWOZ dataset (Budzianowski et al.,2018) is frequently used for benchmarkingcontext-to-response abilities of task-orienteddialogue systems. In this work, we identifyinconsistencies in data preprocessing and re-porting of three corpus-based metrics used onthis dataset, i.e., BLEU score and Inform &Success rates. We point out a few problemsof the MultiWOZ benchmark such as unsat-isfactory preprocessing, insufficient or under-specified evaluation metrics, or rigid database. We re-evaluate 7 end-to-end and 6 policy opti-mization models in as-fair-as-possible setups,and we show that their reported scores cannotbe directly compared. To facilitate compari-son of future systems, we release our stand-alone standardized evaluation scripts. We alsogive basic recommendations for corpus-basedbenchmarking in future works.

## introduction while human judgements are irreplaceable in dialogue systems evaluation and using full dialogue evaluation instead of evaluating isolated responses given ground-truth contexts cannot fully measure system performance @xcite , corpus-based evaluation metrics, such as bleu and corpus-based entity match and success rate @xcite , are still very important for model development and are often used to compare models and establish state-of-the-art. we show on the multiwoz benchmark @xcite , one of the most frequently used and most challenging dialogue system datasets today, that these comparisons do not hold if several basic conditions are not met, and that these conditions are not met for most of the recent works using corpusbased evaluation on this dataset. this means the assessment of progress in terms of dialogue modeling is obscured by noise coming from differences in preprocessing or metrics implementation variants. this paper is not a critique of the multiwoz benchmark or of systems evaluated on it. instead, it is a call for consistency and increased rigor in automatic evaluation. in addition to providing the analysis and identifying problems with the benchmark and current state-of-the-art reporting, we include recommendations for consistency in corpus-based score comparisons. in particular, we advocate for: (1) using standardized implementations of metrics; our work can be summarized as follows: ## related work most works on evaluation methods in dialogue response generation @xcite focus on human evaluation @xcite , e.g., choosing the best methodology with respect to quality and consistency @xcite or robustness @xcite . recent surveys in natural language generation reflect on divergence and inconsistency in human evaluation practice @xcite , in a similar spirit to our examination, but on a broader scale. despite the availability of simulator evaluation @xcite @xcite , corpus-based metrics have been the go-to evaluation method in end-to-end neural dialogue systems since the first implementations @xcite and are a defacto standard until today (cf. section 3.3). there are works showing problems of corpus-based metrics: limited correlation with human judgements @xcite and mixed performance depending on human reference texts used @xcite or evaluated systems @xcite . many works aim at creating more reliable metrics @xcite . recent focus is on trained neural metrics @xcite ), but they are not yet in wide use. our work is probably the closest to post (2018)'s assessment of inconsistencies in different implementations of bleu @xcite , see section 3.2), calling for comparability and proposing a standard implementation. to our knowledge, we are the first to evaluate the use of corpus-based metrics in dialogue systems in this fashion. ## benchmark caveats while multiwoz and the associated metrics described in section 3 represent the state-of-the-art in corpus-based dialogue evaluation practice, the benchmark has the following limitations that researchers need to be aware of: (1) delexicalization problems -imprecise delexicalization based on string matching and varying implementations thereof (section 4.1), (2) lack of standardized postprocessing (i.e., lexicalization methods, section 4.2), (3) database problems, i.e., multiple surface forms of database values and no information about booking availability (section 4.3), (4) atypical metric implementations (section 4.4), (5) lack of diversity evaluation (section 4.5), (6) similarity between training and test data (section 4.6). ## experiments in this section, we work with outputs produced by all systems described in section 3.3. we: (1) unify their responses in terms of delexicalization styles, and then compare bleu when different system bleu score inform & success rate delexical. tokenization venue comparison venue updates reduced search domain source damd damd word intersection name, id state change mintl damd sub-word intersection name, id state change ubar damd sub-word intersection name, id state change soloist hdsa sub-word ---slot names augpt augpt sub-word, nltk first end predicted labes damd word intersection name, id state change dots hdsa word sampling name, id slot names marco hdsa word, nltk subset name, id slot names hdsa hdsa word, nltk subset name, id slot names hdno hdsa word sampling name, id slot names sfn hdsa word sampling name, id slot names uniconv uniconv word sampling name, id, ref. slot names lava lava word sampling name, id slot names ## conclusion the multiwoz benchmark is unique for its size and the inclusion of a complete database, making it possible to build end-to-end task-oriented dialogue systems. because of its naturalness and thanks to multiple fixes and revisions of state annotations, it became very popular for dialogue state tracking. however, it still has limitations for contextto-response generation, partially because of lack of standardized preprocessing and postprocessing. since standard, easy-to-use evaluation scripts are not available, researches are motivated to include their own modifications. this may appear unimportant, but as we showed in our analysis of 13 systems' outputs, it results in large differences in scores and makes any comparison or tracking of progress in this area problematic. we contribute to the solution of this problem by releasing evaluation scripts, which allow consistent evaluation of future work. we further include the evaluation of output diversity, which adds an important aspect missing from corpus-based multiwoz evaluation so far. future work should include a manual revision of multiwoz 2.2 span annotation to reduce training noise and to enable fair evaluation on lexicalized outputs. more important, however, is the use of human evaluation and evaluation of full dialogues in addition to corpus-based metrics @xcite , which is still not standard for end-to-end dialogue systems (cf. section 3.3).