BREAK : Breaking the Dialogue State Tracking Barrier with Beam Search and Re-ranking (2023)

Abstract: Despite the recent advances in dialogue state tracking (DST), the joint goal accuracy (JGA) of the existing methods on MultiWOZ 2.1 still remains merely 60%. In our preliminary error analysis, we find that beam search produces a pool of candidates that is likely to include the correct dialogue state. Motivated by this observation, we introduce a novel framework, called BREAK (Beam search and RE-rAnKing), that achieves outstanding performance on DST. BREAK performs DST in two stages: (i) generating k-best dialogue state candidates with beam search and (ii) re-ranking the candidates to select the correct dialogue state. This simple yet powerful framework shows state-of-the-art performance on all versions of MultiWOZ and M2M datasets. Most notably, we push the joint goal accuracy to 80-90% on MultiWOZ 2.1-2.4, which is an improvement of 23.6%, 26.3%, 21.7%, and 10.8% over the previous best-performing models, respectively. The data and code will be available at https://github.com/tony-won/DST-BREAK

## introduction dialogue state tracking (dst) is an essential component of task-oriented dialogue (tod) systems to help users achieve their specific goals, such as booking restaurants or finding attractions @xcite . the task of dst is to understand the meaning of user utterances and keep track of users' intentions throughout the conversation. since the results of dst affects the subsequent tod tasks, i.e., dialogue policy and response generation, the accuracy of dst is crucial without a doubt @xcite . in dst, the dialogue state is typically represented by a set of (slot, value) pairs, e.g., ("hotel-area", "centre"). here, the list of slots is a pre-defined set, and the corresponding values are extracted from the dialogue context. ## preliminaries in this section, we formally describe the problem and generation-based approach for dst. then we report our in-depth analysis of the errors produced by generation-based dst models. ## break: beam search and re-ranking based on the analysis in section 2.3, we propose a novel framework for generation-based dst. our approach, dubbed break, utilizes beam search and re-ranking at the inference phase. specifically, given a trained dst model, the main idea is to generate dialogue state candidates using beam search and then find the correct dialogue state by re-ranking them. ## experimental results unless otherwise noted, all t5-based results are obtained using the form of the cloze-style (cs). this is due to the computational efficiency, and more details are described in section 5.4. ## conclusion we propose a simple yet effective framework for generation-based dst that breaks the performance barrier in dst. we design our framework based on our findings that the probability of ground truth value being generated by dst models is very high in most decoding steps. our method effectively tracks the dialogue state by (i) generating beam search candidates and (ii) re-ranking them via assessing the semantic matching with the dialogue context. by exploring the highly probable dialogue state candidates discovered by beam search, our method significantly reduces errors compared to the decoding process that generates a single definitive dialogue state. in our experiments, we achieve state-of-the-art performance on multiwoz and m2m datasets by a significant margin, regardless of the backbone plms. for future work, we plan to improve the computational efficiency of the current framework to apply in real-world settings.