Effective Sequence-to-Sequence Dialogue State Tracking (2021)

Abstract: Sequence-to-sequence models have been applied to a wide variety of NLP tasks, but how to properly use them for dialogue state tracking has not been systematically investigated. In this paper, we study this problem from the perspectives of pre-training objectives as well as the formats of context representations. We demonstrate that the choice of pre-training objective makes a significant difference to the state tracking quality. In particular, we find that masked span prediction is more effective than auto-regressive language modeling. We also explore using Pegasus, a span prediction-based pre-training objective for text summarization, for the state tracking model. We found that pre-training for the seemingly distant summarization task works surprisingly well for dialogue state tracking. In addition, we found that while recurrent state context representation works also reasonably well, the model may have a hard time recovering from earlier mistakes. We conducted experiments on the MultiWOZ 2.1-2.4, WOZ 2.0, and DSTC2 datasets with consistent observations.

## introduction sequence-to-sequence (seq2seq) modeling @xcite is one of the most widely adopted generative framework for a multitude of nlp tasks. while it has also been applied for task-oriented dialogue modeling @xcite @xcite , how to best setup seq2seq models on this task remains an understudied topic. in this paper, we investigate this problem from two perspectives: pre-training objectives and dialogue context representation, and we focus on the dialogue state tracking (dst) task. the flexibility of the seq2seq model allows us to adopt and compare pre-training strategies for other nlp tasks sharing the same architec-ture. specifically, we first experimented with different pre-training setups of t5 @xcite which have been shown to be effective for generic language understanding. additionally, as an exploratory effort, we applied pegasus @xcite , a pre-training procedure designed for text summarization, to the task of dst. additionally, to investigate how different dialogue context representations affect seq2seq performance, we compare two versions of all models: one accepts full conversation history as context, and one that feeds the previously predicted states recurrently as summary of context. we conduct systematic experiments on the multi-woz @xcite benchmark. for fair comparison with existing approaches, we report results on multiwoz 2.1-2.4 @xcite @xcite , all 4 variations of the benchmark proposed to date. in addition, we report results on the woz 2.0 @xcite and dstc2 @xcite datasets. our findings can be summarized as follows: ## related work generative sequential models have been applied for task-oriented dialogue problems in several ways. @xcite @xcite adopted gpt-2, a unidirectional pretrained transformer lm, as backbones for the generation of states, actions and responses. under the framework of seq2seq, perhaps most similar to our work is @xcite , which adopts a transformer encoder-decoder architecture, with the encoder pre-trained with bert which is also used to encode schema. besides, @xcite is an early example that uses encoder outputs as state representation, merged with kb representation for the decoder to generate responses; @xcite proposes a simplistic two stage copynet on top of seq2seq model to enable word copying from input sequences; @xcite proposes a hierarchical seq2seq model for coarse-to-fine dst; @xcite proposes a "flat" encoder-decoder structure which reuses a bert encoder for the function of a decoder with hidden layer states reused. in terms of pre-training, bert and gpt are still the most commonly used techniques @xcite . various pre-training methods developed for dialogue-specific problems have also been developed. @xcite uses dialogue specific datasets for pre-training and fine-tuning. @xcite studies 4 ways of pre-training aiming at better capturing discourselevel dependencies for multi-turn dialogues; @xcite proposed a contrastive pre-training loss to capture important qualities of dialogues; @xcite proposed a curriculum pre-training procedure for response generation, subsuming opendomain, knowledge-grounded, task-oriented dialogue applications. @xcite factorize the generative dialogue model according to the noisy channel model, pre-training each component separately. ## conclusion we studied the problem of how to perform the dst task with seq2seq models effectively from the perspective of pre-training and context representation. we demonstrated that seq2seq pre-training objectives involving masked span prediction are more preferred than auto-regressive predictions for dialogue understanding. this observation further generalizes to the adoption of pegasus, a span prediction objective for summarization, which works surprisingly well on dst tasks. we also find that recurrent state representation for dialogue context can work reasonably well.