Span-Selective Linear Attention Transformers for Effective and Robust Schema-Guided Dialogue State Tracking (2023)

Abstract: In schema-guided dialogue state tracking models estimate the current state of a conversation using natural language descriptions of the service schema for generalization to unseen services. Prior generative approaches which decode slot values sequentially do not generalize well to variations in schema, while discriminative approaches separately encode history and schema and fail to account for inter-slot and intent-slot dependencies. We introduce SPLAT, a novel architecture which achieves better generalization and efficiency than prior approaches by constraining outputs to a limited prediction space. At the same time, our model allows for rich attention among descriptions and history while keeping computation costs constrained by incorporating linear-time attention. We demonstrate the effectiveness of our model on the Schema-Guided Dialogue (SGD) and MultiWOZ datasets. Our approach significantly improves upon existing models achieving 85.3 JGA on the SGD dataset. Further, we show increased robustness on the SGD-X benchmark: our model outperforms the more than 30x larger D3ST-XXL model by 5.0 points.

## introduction dialogue state tracking (dst) refers to the task of estimating and tracking the dialogue state consisting of the user's current intent and set of slotvalue pairs throughout the dialogue @xcite . traditional approaches to dst assume a fixed ontology and learn a classifier for each slot @xcite . however, in real-world applications services can be added or removed requiring the model to be re-trained each time the ontology changes. recently more flexible schemaguided approaches which take as input natural language descriptions of all available intents and slots and thus can be applied zero-shot to new services have been gaining popularity @xcite @xcite . [slot] encodes the semantics of the natural language description of "to_location" and is matched with the span representation of "long beach, ca". similarly [utt] encodes the semantics of the current utterance and is matched with the target [intent] encoding. discriminative dst models are based on machine reading comprehension (mrc) methods, meaning they extract and fill in non-categorical slot values directly from the user utterances @xcite @xcite . we use the terms discriminative and extractive interchangeably when referring to these methods. generative dst models leverage seq2seq language models which conditioned on the dialog history and a prompt learn to sequentially generate the appropriate slot values. prior generative methods do not generalize well to variations in schema @xcite @xcite whereas discriminative methods separately encode history and schema and fail to account for inter-slot and intent-slot dependencies. in this work we introduce the span-selective linear attention transformer, short splat, a novel architecture designed to achieve better generalization, robustness and efficiency in dst than existing approaches. splat is fully extractive and, unlike prior generative approaches, constrains the output space to only those values contained in the input sequence. figure 1 shows an example i and intent representations h [intent] k . a span encoder computes span representations h span mn for all spans x m , . . . , x n . the target span is selected by matching the slot query h [slot] q to the target span h span ij . our main contributions are as follows: ## experimental setup we describe our experimental setup including datasets used for pre-training and evaluation, implementation details, baselines and evaluation metrics in detail below. ## evaluation we evaluate the effectiveness of our model through a series of experiments designed to answer the following questions: 1) how effective is the proposed model architecture at dst in general? 2) does the model generalize well to unseen services? 3) is the model robust to changes in schema such as different slot names and descriptions? 4) which parts of the model contribute most to its performance? ## related work extractive dst. following the traditional extractive setting @xcite propose a machine reading comprehension (mrc) approach which decodes slot values turn-by-turn using a different learned classifier for each slot. as a classifier has to be learned for each new slot this approach cannot easily be transferred to new slots. schema-guided approaches address this by explicitly conditioning predictions on a variable schema which describes intents and slots in natural language @xcite . @xcite and @xcite introduce schema-guided models but predict slots independently from one another requiring multiple encoder passes for each turn and failing to model intent-slot and inter-slot dependencies. @xcite use mrc for non-categorical and handcrafted features for categorical slots. ## conclusion in this work we introduced splat, a novel architecture for schema-guided dialogue state tracking which learns to infer slots by learning to select target spans based on natural language descriptions of slot semantics, and further showed how to pretrain splat via a recurrent span selection objective for better span representations and a stronger slot prediction performance. we find that our proposed architecture yields significant improvements over existing models and achieving 85.3 jga on the sgd dataset and 57.4 jga on the multiwoz dataset. in schema-guided dst the ability to generalize to new schemas and robustness to changes in schema descriptions is of particular interest. we demonstrated that our model is much more robust to such changes in experiments on the sgd-x benchmark where splat outperforms the more than 30× larger d3st-xxl model by 5.0 points.