Mismatch between Multi-turn Dialogue and its Evaluation Metric in Dialogue State Tracking (2022)

Abstract: Dialogue state tracking (DST) aims to extract essential information from multi-turn dialog situations and take appropriate actions. A belief state, one of the core pieces of information, refers to the subject and its specific content, and appears in the form of domain-slot-value. The trained model predicts “accumulated” belief states in every turn, and joint goal accuracy and slot accuracy are mainly used to evaluate the prediction; however, we specify that the current evaluation metrics have a critical limitation when evaluating belief states accumulated as the dialogue proceeds, especially in the most used MultiWOZ dataset. Additionally, we propose relative slot accuracy to complement existing metrics. Relative slot accuracy does not depend on the number of predefined slots, and allows intuitive evaluation by assigning relative scores according to the turn of each dialog. This study also encourages not solely the reporting of joint goal accuracy, but also various complementary metrics in DST tasks for the sake of a realistic evaluation.

## introduction the dialogue state tracking (dst) module structures the belief state that appears during the conversation in the form of domain-slot-value, to provide an appropriate response to the user. recently, multi-turn dst datasets have been constructed using the wizard-of-oz method to reflect more realistic dialogue situations @xcite @xcite . the characteristic of these datasets is that belief states are "accumulated" and recorded every turn. that is, the belief states of the previous turns are included in the current turn. it confirms whether the dst model tracks essential information that has appeared up to the present point. joint goal accuracy and slot accuracy are utilized in most cases to evaluate the prediction of accumu-lated belief states. joint goal accuracy strictly determines whether every predicted state is identical to the gold state, whereas slot accuracy measures the ratio of correct predictions. however, we determined that these two metrics solely focus on "penalizing states that fail to predict," not considering "reward for well-predicted states." accordingly, as also pointed out in @xcite , joint goal accuracy underestimates the model prediction because of its error accumulation attribute, while slot accuracy overestimates it because of its dependency on predefined slots. however, there is a lack of discussion on the metric for evaluating the most used multiwoz dataset, despite a recently published dataset @xcite proposing some metrics. to address the above challenge, we propose reporting the relative slot accuracy along with the existing metrics in multiwoz dataset. while slot accuracy has the challenge of overestimation by always considering all predefined slots in every turn, relative slot accuracy does not depend on predefined slots, and calculates a score that is affected solely by slots that appear in the current dialogue. therefore, relative slot accuracy enables a realistic evaluation by rewarding the model's correct predictions, a complementary approach that joint goal and slot accuracies cannot fully cover. it is expected that the proposed metric can be adopted to evaluate model performance more intuitively. ## relative slot accuracy as can be observed in equation 2 , slot accuracy has the characteristic that the larger the number of predefined slots (t ), the smaller the deviation between the prediction results. the deviation among dst models will be even more minor when constructing datasets with various dialogue situations, because the number of predefined slots will continually in-crease. it is not presumed to be an appropriate metric in terms of scalability. therefore, we propose relative slot accuracy, that is not affected by predefined slots, and is evaluated with adequate rewards and penalties that fit human intuition in every turn. equation 3 expresses how to calculate the relative slot accuracy, and t * denotes the number of unique slots appearing in the predicted and gold states in a particular turn. relative slot accuracy rewards well-predicted belief states by measuring the scores in accumulating turns. further discussions on the relative score will be discussed in section 4.1. ## experiments we measured multiwoz 2.1, an improved version of multiwoz 2.0 @xcite , which has been adopted in several studies, according to table a5 . five domains (i.e., hotel, train, restaurant, attraction, and taxi) are adopted in the experiment, following @xcite , and there are a total of 30 domain-slot pairs. we selected the dst models in table a5 that perform the mul-tiwoz experiment with the original authors' reproducible code foot_1 . additionally, we reported the f1 score, which can be calculated using the current predicted and gold states. ## conclusion this paper points out the challenge that the existing joint goal and slot accuracies cannot fully evaluate the accumulating belief state of each turn in the multiwoz dataset. accordingly, the relative slot accuracy is proposed. this metric is not affected by unseen slots in the current dialogue situation, and compensates for the model's correct predic-tion. when the dst task is scaled up to deal with more diverse conversational situations, a realistic model evaluation will be possible using relative slot accuracy. moreover, we suggest reporting various evaluation metrics to complement the limitations of each metric in future studies, not solely reporting the joint goal accuracy.