Dialogue State Tracking with Sparse Local Slot Attention (2023)

Abstract: Dialogue state tracking (DST) is designed to track the dialogue state during the conversations between users and systems, which is the core of task-oriented dialogue systems. Mainstream models predict the values for each slot with fully token-wise slot attention from dialogue history. However, such operations may result in overlooking the neighboring relationship. Moreover, it may lead the model to assign probability mass to irrelevant parts, while these parts contribute little. It becomes severe with the increase in dialogue length. Therefore, we investigate sparse local slot attention for DST in this work. Slot-specific local semantic information is obtained at a sub-sampled temporal resolution capturing local dependencies for each slot. Then these local representations are attended with sparse attention weights to guide the model to pay attention to relevant parts of local information for subsequent state value prediction. The experimental results on MultiWOZ 2.0 and 2.4 datasets show that the proposed approach effectively improves the performance of ontology-based dialogue state tracking, and performs better than token-wise attention for long dialogues.

## introduction task-oriented dialogue systems aim to assist users to complete certain tasks and have drawn great attention in both academia and industry @xcite @xcite . as the core of taskoriented dialogue systems, dialogue state tracking (dst) is designed to track the dialogue states during the conversation between users and systems, which is generally expressed as a list of {(domain, slot, value)} representing user's goal @xcite . the estimated dialogue states are used for subsequent actions. to achieve the dialogue state, value prediction is made for each slot given the dialogue history. at each turn, the model inquires of the dialogue history and predicts the state values accordingly @xcite @xcite @xcite . with it, how to extract appropriate context information in the noisy dialogue history is crucial and challenging @xcite . @xcite make an empirical study about the effect of different contexts on the performance of dst with several manually designed rules. it indicates that the performance of dst models benefits from selecting appropriate context granularity. in recent mainstream models, a fully token-wise slot attention mechanism is widely used to capture slot-specific information with dialogue history. the attention assigns an attention weight to each token, measuring the relationship of each token in dialogue history for the specified slot, and then attends them with these weights. although encouraging results have been achieved, it also brings some risks. first, such operations disperse the distribution of attention, which results in overlooking the neighboring relation @xcite . some entities (e.g., restaurant and attraction names) in spoken dialogue are generally informal, diverse, and local-compact, where the non-semantic tokens may be included. moreover, a limitation of the used softmax computation is that the probability distribution in the outputs always has full support @xcite , i.e., softmax(z) > 0 for every vector z. it may lead a model to assign probability mass to implausible parts of dialogue history. involving noise may make the model difficult to focus on the essential parts, and it may be more severe with the increase in dialogue length @xcite . to tackle this problem, we propose a sparse local slot attention mechanism for this task. in our approach, local semantic information is firstly achieved at a sub-sampled temporal resolution capturing local dependencies for each slot. then, these local information is attended with sparse attention weights generated by sparsemax function (martins and astudillo, 2016), which outputs sparse posterior distributions by assigning zero probability to irrelevant contents in the dialogue history. we conduct experiments to verify our approach on multiwoz 2.0 and 2.4 datasets. the contributions can be addressed as follows: 1) we propose a sparse local slot attention mechanism to lead the model to focus on relevant local parts to the specific slot for the dst task; 2) we demonstrate that the performance of dst benefits from introducing local information with our proposed approach, and make an empirical study that shows that our model performs better in state prediction for name-related slots and long dialogues than the models based on fully token-wise attention. pre-trained models have brought natural language processing to a new era in recent years. many substantial works have shown that the pretrained models can learn universal language representations, which are beneficial for downstream tasks @xcite @xcite @xcite . more recently, very deep pre-trained language models, such as bidirectional encoder representation from the transformer (bert) @xcite and generative pretraining (gpt) @xcite , trained with an increasing number of self-supervised tasks have been proposed to make the models capturing more knowledge from a large scale of corpora, which have shown their abilities to produce promising results in downstream tasks. in view of it, many pieces of research of dst have explored to establish the models on the basis of pre-trained language models (hosseini-asl et al., 2020; @xcite @xcite @xcite @xcite . related to extracting slot-specific information, most of the previous studies rely on dense tokenwise attention @xcite @xcite . however, several pieces of research have indicated that local information may be missing with it @xcite @xcite @xcite . motivated by it, we investigate introducing local modeling in this task. the most rele-vant research is @xcite , which makes a comprehensive study of how different granularities affect dst. however, this research employs simple hand-crafted rules to neglect several utterances in a dialogue history. our proposed approach in this work is data-driven. ## conclusion in his work, we propose a sparse local slot attention for dialogue state tracking to alleviate allocating attention weights to content unrelated to the specific slot of interest. in our approach, local semantic information is firstly achieved at a sub-sampled temporal resolution capturing local dependencies for each slot. then, these local information is attended with sparse attention weights generated by sparsemax function. the experimental results show that, comparing to several existing models based on dense token-wise attention, our approach effectively improves the performance of ontology-based dialogue state tracking in the state prediction for name-related slots and long dialogues.