Dialogue Summaries as Dialogue States ( DS 2), Template-Guided Summarization for Few-shot Dialogue State Tracking (2022)

Abstract: Annotating task-oriented dialogues is notorious for the expensive and difficult data collection process. Few-shot dialogue state tracking (DST) is a realistic solution to this problem. In this paper, we hypothesize that dialogue summaries are essentially unstructured dialogue states; hence, we propose to reformulate dialogue state tracking as a dialogue summarization problem. To elaborate, we train a text-to-text language model with synthetic template-based dialogue summaries, generated by a set of rules from the dialogue states. Then, the dialogue states can be recovered by inversely applying the summary generation rules. We empirically show that our method DS2 outperforms previous works on few-shot DST in MultiWoZ 2.0 and 2.1, in both cross-domain and multi-domain settings. Our method also exhibits vast speedup during both training and inference as it can generate all states at once. Finally, based on our analysis, we discover that the naturalness of the summary templates plays a key role for successful training.

## introduction task-oriented dialogue systems (tod) have penetrated our daily lives much more than before and they will continue to increase their presence. for example, many of our mobile devices are equipped with such dialogue agents like siri, and we now often encounter customer service or flight reservation bots. dialogue state tracking (dst) is an essential element of such task-oriented dialogue systems @xcite . the main goal of this component is to understand the user's requirements expressed during the conversation under a given schema or ontology. hence, * equal contribution: js proposed the main idea and scaled up the experiments. hy designed and implemented the heuristic state tracking component. hm conducted rapid prototyping, analysis, and ablations. furthermore, in realistic industrial settings, having to expand the existing model and ontology to include new domains and slot-values is a common phenomenon. naturally, there have been many recent works that proposed (zero) few-shot settings to rely on less annotated data. for instance, both starc @xcite and transferqa @xcite achieve great few-shot dst performance on multiwoz 2.0 by prompting large pretrained language models like bert @xcite and t5 @xcite with natural language questions (e.g. "in what area is the user looking for a hotel?"). meanwhile, despite the good performance of the aforementioned works, they still suffer from certain issues. 1) they often require a large amount of expensive labeled training data from other tasks or domains for task-specific pre-training. for example, as shown in addressing the above challenge, we propose to cast dialogue state tracking as a dialogue summarization task; hence the name is dialogue summaries as dialogue states (ds2). the main hypothesis for this reformulation is that dialogue summaries are essentially unstructured dialogue states. in this paper, we explore this reformulation to the limit. we fine-tune large text-to-text pre-trained language models (e.g. t5, bart) with synthetic dialogue summaries. these summaries are created by heuristic rules from the dialogue states, as in figure 1 . hence, as these models already excel in text summarization, the research question we ask is whether we can guide dialogue summarization models to generate dialogue summaries that conform to the templates we provide. then, we can extract the dialogue states by inversely applying the rules we used to create the synthetic summaries. compared to previous approaches, our method has several advantages that come naturally. first, we easily reduce the pre-train & fine-tune discrepancies without any dst-specific engineering by leveraging dialogue summarization datasets. these datasets are an order of magnitude smaller in annotated data size (e.g. samsum @xcite has ∼ 200k utterances). second, we achieve great speedup in both training and inference because we only need to summarize once, and we can extract model # of pre-train data data type tod-bert @xcite ∼1.39m dialogue utterances pptod @xcite ∼2.37m dialogue utterances transfer qa @xcite ## related work dialogue state tracking is a well-known subtask of task oriented dialog systems @xcite . the current state-of-the-art techniques fine-tune pre-trained language models @xcite @xcite @xcite @xcite @xcite @xcite @xcite are often further trained with a large amount of annotated data. few-shot dst is a promising direction for reducing the need of human annotation while achieving quasi-sota performance with a fraction of the training data. different techniques have been proposed @xcite @xcite @xcite @xcite @xcite . we briefly describe and compare ds2 with existing few-shot models in section 4.5. ## some slot-value entities include prepositions. samsum @xcite is a dialogue summarization dataset. we further pre-train t5-large (raffel et al., 2020) more with samsum by using the code from @xcite before we fine-tune for ds2. ## cost of template engineering for multiwoz, we devised templates for all 30 slots in the 5 domains that were used. based on names of the slots, we wrote a state-to-summary function that generates a natural phrase along with the slot value with prefix templates. the summary-to-state parsing functions were written using regular expressions based on the rules we implemented for template generation. overall, this process took approximately one week for one expert to finish. we believe that this is a much lower cost compared to full dst data design and collection efforts. applying to a new domain may take even lower costs by using our code-base. appendix section a.4 describes this process in detail. ## limitations in this section, we discuss several limitations of this work. first, applying our model to a new do-main requires a new summary template. since ds2 performance is sensitive to the quality of the template as shown in the ablation study, considerable amount of knowledge on both domain and nlp is desired. however, following the guide in section a.4 would take less than one week for a researcher, which costs much less than collecting full dst data. second, ds2 is not capable of zero-shot inference because it should learn the template, at least from a few samples. third, regular expression pattern matching may fail during the state extraction. there is no guarantee for the model output to fit in the template. the matching may still fail for a correctly formatted summary if a value entity contains template-like patterns. using a neural network-based converter might easily solve this problem. fourth, there is still room for improvement using dst-specific engineering (span matching or ontology searching as in trippy @xcite ). finally, output summary length is bounded by the plm's maximum sequence length, so ds2 might fail when we have too many slot values. we leave these for future investigation. ## conclusion this work tackles the few-shot dst problem by reformulating it into dialogue summarization. the strategy is to minimize the pre-train and fine-tune discrepancy by adapting a pre-trained language model (plm) to a more familiar task: summarization. hence, instead of forcing the model to learn a completely new task like dst, we provide rulebased summary templates from dialogue states. we guide the summarization to conform to such templates, and utilize heuristic dialogue state extraction from the generated summaries. the experimental results show that our model ds2 outperforms baselines for few-shot dst on multiwoz in both crossdomain and multi-domain settings. in addition, ds2 significantly reduces inference time complexity compared to existing qa-based methods. we also observed that naturalness of the template was very important.