GCDF 1: A Goal- and Context- Driven F -Score for Evaluating User Models (2021)

Abstract: The evaluation of dialogue systems in interaction with simulated users has been proposed to improve turn-level, corpus-based metrics which can only evaluate test cases encountered in a corpus and cannot measure system’s ability to sustain multi-turn interactions. Recently, little emphasis was put on automatically assessing the quality of the user model itself, so unless correlations with human studies are measured, the reliability of user model based evaluation is unknown. We propose GCDF1, a simple but effective measure of the quality of semantic-level conversations between a goal-driven user agent and a system agent. In contrast with previous approaches we measure the F-score at dialogue level and consider user and system behaviours to improve recall and precision estimation. We facilitate scores interpretation by providing a rich hierarchical structure with information about conversational patterns present in the test data and tools to efficiently query the conversations generated. We apply our framework to assess the performance and weaknesses of a Convlab2 user model.

## introduction remarkable progress has been achieved in many dialogue systems research disciplines, from dialogue state tracking (dst) @xcite to policy- @xcite and end-to-end modelling @xcite . progress is usually measured component-wise through task-specific metrics and improvements in the overall performance of the systems leveraging advances in component designs are seldom reported @xcite . @xcite empirically show that component-wise evaluation may not correlate well with the overall performance of the system. they recommend evaluating dialogue systems in an end-to-end, interactive, multi-turn setting to capture the effect of 1 code available at https://bit.ly/3hvs55q . error propagation on system performance and approximate the field performance of a system more accurately. @xcite perform extensive user model interactive evaluation for a wide range of dialogue system architectures implemented in the convlab library @xcite . they find that while the simulated user interaction evaluation overestimates the true performance of the systems evaluated, a mild correlation with human performance assessment exists. in this context, this paper seeks to provide a simple and effective tool to measure the predictive power of a user model, arguing that it is important to understand how well current user models perform and how to enhance them to improve system-wise evaluation accuracy. we propose a simple generalisation of the corpus-based, turn-level f1 score proposed by @xcite as a measure of the similarity between the (semantic-level) simulated user response and the response provided by a real user given the same context. we believe this to be necessary since turn-level f1 favours models which are biased to a potentially restricted set of behaviours learned from a corpus whereas an optimal user model should exhibit a wider variety of behaviours. similar to the convlab2 @xcite evaluation, our metric is goal-driven. it evaluates, at dialogue level, the ability of the user model to express all the constraints 2 (i-gcdf1) and request all the information (r-gdcdf1) prescribed by a goal when interacting with an arbitrary agent. in human-human conversation, repetition of constraints occurs due to co-reference, confirmation, emphasis and through other linguistic and conversational processes. information requests may be specified at the same time with the search constraints and later repeated. language understanding errors may see agents stuck in conversational loops where the same question and answer are repeated ad nauseam. failure to account for these repetitions may thus affect f1 scores. consequently, gcdf1 scores are also context-driven: the dialogue span between repetitions of constraints or requests is analysed to determine whether they are erroneous, elicited by a system behaviour, or due to an intrinsic user behaviour. in addition, the mentioning of not-in-goal constraints warranted by the conversational context (e.g., mentions of don't care values or entities) are accounted for. the gcdf1 evaluator outputs a rich hierarchical structure where the interactions evaluated are classified according to the results of the context-driven analysis. additionally, dialogue level score information and other metadata are output. we also developed tools to analyse evaluator output and query the set of interactions to interpret model behaviour. hence, we hope that our implementation will help developers improve their models. in summary, our contributions are: we support these claims by studying the user model employed by @xcite in their study of dialogue system performance. to measure the ability to appropriately respond in a given dialogue situation, the turn-level f1 score @xcite is used. alternatively, data is generated through interaction of the user model to be assessed with a wide range of system models, a protocol known as cross-evaluation @xcite . system-side metrics of task success computed for each system model are then averaged and used as proxies for the user model performance: a good user model is expected to perform well when interacting with a variety of dialogue systems and should attain a high score. ## metric description the following sections present the i-and r-gcdf1 algorithms. our implementation is based on the multiwoz 2.1 @xcite corpus, where the behaviours mentioned in this section were detected. ## conclusion we proposed the gcdf1 framework and used it to conduct a detailed performance analysis of a user model. understanding error states supports model improvement: for example, we identified that the user model analysed does not understand the reference slot. hence, the user nlu model could be finetuned to resolve this error. the template nlg module was also shown to affect dialogue structure and quality. future work could assess other convlab2 user models and extend our approach to larger corpora with more complex dialogue flows such as sgd @xcite .