Simple Data Augmentation with the Mask Token Improves Domain Adaptation for Dialog Act Tagging (2020)

Abstract: The concept of Dialogue Act (DA) is universal across different task-oriented dialogue domains - the act of “request” carries the same speaker intention whether it is for restaurant reservation or flight booking. However, DA taggers trained on one domain do not generalize well to other domains, which leaves us with the expensive need for a large amount of annotated data in the target domain. In this work, we investigate how to better adapt DA taggers to desired target domains with only unlabeled data. We propose MaskAugment, a controllable mechanism that augments text input by leveraging the pre-trained Mask token from BERT model. Inspired by consistency regularization, we use MaskAugment to introduce an unsupervised teacher-student learning scheme to examine the domain adaptation of DA taggers. Our extensive experiments on the Simulated Dialogue (GSim) and Schema-Guided Dialogue (SGD) datasets show that MaskAugment is useful in improving the cross-domain generalization for DA tagging.

## introduction dialog act (da) tagging, one of the important nlu components of modern task-oriented dialog systems, aims to capture the speaker's intention behind the utterances at each dialog turn. several different schema and taxonomies have been introduced by several different researchers @xcite @xcite over the years. however, the main focus of the recent work @xcite @xcite on da tagging was on human-human social conversations @xcite , which is less applicable for task-oriented setting. recently, several task-oriented dialogue datasets @xcite @xcite have been released. however, the discrepancy in their annotation schema hinders the progress on building da taggers that can generalize across domains and possibly datasets. to address this issue, @xcite propose a universal schema for das by aligning annotations for multiple existing corpora. in this regard, another useful corpora employed as a testbed in this work is schema-guided dialogues (sgd) @xcite , which covers 20 domains under the same da annotation schema. it is often challenging and costly to obtain a large amount of in-domain dialogues with annotations. however, unlabeled dialogue corpora in target domain can easily be curated from past conversation logs or collected via crowd-sourcing @xcite at a more reasonable cost. the goal of this work is to investigate how to leverage pre-trained masked language models (e.g., bert) to better adapt da taggers to unseen domains with available unlabeled dialogues. pre-trained language models @xcite have been successful for several nlp tasks including dialogue systems @xcite , 2019; @xcite @xcite . however, domain adaptation capabilities of these models remain to be further explored for goal-oriented dialogues. in this paper, we use the pre-trained mask token of bert model to define maskaugment, which stochastically augments text input by randomly replacing its tokens with the mask token. we adopt consistency regularization approach @xcite to introduce an unsupervised teacher-student learning scheme by leveraging maskaugment for generating teacher and student representations retaining different amount of the original content from the unlabeled dialogue example. our extensive experiments on gsim @xcite and sgd @xcite datasets suggest: (i) bert establishes a much stronger baseline compared to previous work @xcite , (ii) the proposed teacher-student learning via maskaug-ment is useful in further improving the target domain f1 score over bert baseline: up to 3% when the full source domain data is used, and up to 10% for the low-resource setting. ## maskaugment in this section, we first discuss the task setup, bert-based da tagging model, and relevant background. we then define the proposed fine-tuning objectives leveraging maskaugment. ## conclusion we study cross-domain generalization of pretrained language models for da tagging. while the fine-tuned bert model performs well on indomain da tagging, its cross-domain generalization is still not satisfactory. to combat this shortcoming, we investigate domain adaptation through the proposed unsupervised teacher-student training that leverages the maskaugment method for data augmentation. our empirical results show that the proposed training scheme leads to significant improvements on domain adaptation for dialog act taggers. in the future, we plan to explore maskaugment for other tasks in nlp domain.