Towards Fair Evaluation of Dialogue State Tracking by Flexible Incorporation of Turn-level Performances (2022)

Abstract: Dialogue State Tracking (DST) is primarily evaluated using Joint Goal Accuracy (JGA) defined as the fraction of turns where the ground-truth dialogue state exactly matches the prediction. Generally in DST, the dialogue state or belief state for a given turn contain all the intents shown by the user till that turn. Due to this cumulative nature of the belief state, it is difficult to get a correct prediction once a misprediction has occurred. Thus, although being a useful metric, it can be harsh at times and underestimate the true potential of a DST model. Moreover, an improvement in JGA can sometimes decrease the performance of turn-level or non-cumulative belief state prediction due to inconsistency in annotations. So, using JGA as the only metric for model selection may not be ideal for all scenarios. In this work, we discuss various evaluation metrics used for DST along with their shortcomings. To address the existing issues, we propose a new evaluation metric named Flexible Goal Accuracy (FGA). FGA is a generalized version of JGA. But unlike JGA, it tries to give penalized rewards to mispredictions that are locally correct i.e. the root cause of the error is an earlier turn. By doing so, FGA considers the performance of both cumulative and turn-level prediction flexibly and provides a better insight than the existing metrics. We also show that FGA is a better discriminator of DST model performance.

## introduction dialogue state tracking (dst) is at the core of task-oriented dialogue systems. it is responsible for keeping track of the key information exchanged during a conversation. with the growing popularity of task-based conversational agents, it is essential to review the evaluation of dst to appropriately measure the progress in this evolving area. the task of dst is to predict the user intent through dialogue states @xcite . fig. 1 shows an example dst task from multi-woz @xcite dataset. let u t and s t be the user and system utterances respectively at turn t. then a typical conversation can be expressed as @xmath0, (s 1 , u 1 ), ...(s n , u n )}. the commonly used ground-truth dialogue state for dst is the belief state. belief state b t for turn t is defined as the set of (domain, slot, slot-value) triplets that have been extracted till turn t, thereby it is cumulative in nature. the objective of dst is to predict b t given the dialogue history till turn t. the primary metric for evaluating dst is joint goal accuracy (jga). it compares the predicted dialogue states to the ground truth b t at each dialogue turn t @xcite . as the belief state is cumulative, it is very unlikely for a model to get back a correct prediction after a misprediction. this is why it can provide an underestimated performance in certain cases. besides, jga completely ignores the performance of turn-specific local predictions. let t t be the turn-level belief state that contains all the intents or (domain, slot, slot-value) triplets expressed by the user only at turn t. ideally, a model with higher jga should also perform equally well to predict t t . but, we observe that improving jga can sometimes degrade the performance of predicting t t mainly due to the presence of annotation inconsistencies in the available datasets. for example, in fig. 1 , the presence of (hotel, area, centre) and absence of (attraction, name, all saints church) in ground-truth b 2 and b 4 shows such inconsistencies. so, the generalization of the model may get compromised if the model selection is done only using jga. annotation inconsistencies and errors are common in real-world datasets. hence, to provide a fair estimate, it requires not only track the performance of the cumulative belief state but also turn-level belief state as well. in this work, we address these issues of jga by proposing a novel evaluation metric for dst called flexible goal accuracy (fga). the central idea of fga is to partially penalize a misprediction which is locally correct i.e. the source of the misprediction is some earlier turn. the main contributions of our work are as follows 1 : ## flexible goal accuracy from the previous discussion, it is evident that despite a few limitations, joint goal accuracy is superior to the other two metrics. this is why with the objective to obtain a better evaluation metric for dst, we address the shortcomings of jga by proposing a new metric called flexible goal accuracy (fga). the description of fga is presented in the next part of this section, whereas its working is described as a pseudo-code in algo. 1. for a given a turn t, an error in belief state prediction (i.e. b t ̸ = b ′ t ) can occur in two ways: 1) the source of the error is turn t itself i.e. the turn-level prediction is wrong, 2) the turn-level prediction of turn t is correct but the source of the error is some earlier turn t err ≺ t. fga works differently from jga only for type 2 errors. unlike jga, fga does not penalize type 2 errors completely. it assigns a penalized score based on the distance between the error turn (t err ) and the current turn (t) and the penalty is inversely proportional to this distance (t -t err ). the main idea is to forget the mistakes with time in order to attain a fair judgment of a dst model offline. we decide the correctness of a turn-level match using the logic shown in line 10 of algo. 1. a turn @xmath1. in other words, a turn-level or local match indicates that all the intents shown by the user in a particular turn have been correctly detected without any false positives. just comparing t t and t ′ t to check a turn-level or local match can be erroneous because it will not credit the model for error corrections. ## result and analysis in this section, we report the performance of fga along with the other metrics on four different dst models: trade @xcite , hi-dst @xcite , som-dst @xcite , and trippy @xcite . we use the mul-tiwoz 2.1 dataset @xcite as most of the recent progress in dst are showcased on this dataset. the results are reported in from ## conclusion in this work, we analyzed the limitations of existing dst metrics. we argued that joint accuracy can underestimate the power of a dst algorithm, whereas slot and average goal accuracy can overestimate it. we addressed the issues of joint accuracy by introducing flexible goal accuracy (fga) which tries to give partial credit to mispredictions that are locally correct. we justified that fga provides a relatively balanced estimation of dst performance along with better discrimination property. in conclusion, fga is a practical and insightful metric that can be useful to evaluate future dst models.