Domain-Adaptive Pretraining Methods for Dialogue Understanding (2021)

Abstract: Language models like BERT and SpanBERT pretrained on open-domain data have obtained impressive gains on various NLP tasks. In this paper, we probe the effectiveness of domain-adaptive pretraining objectives on downstream tasks. In particular, three objectives, including a novel objective focusing on modeling predicate-argument relations, are evaluated on two challenging dialogue understanding tasks. Experimental results demonstrate that domain-adaptive pretraining with proper objectives can significantly improve the performance of a strong baseline on these tasks, achieving the new state-of-the-art performances.

## introduction recent advances in pretraining methods @xcite @xcite have achieved promising results on various natural language processing (nlp) tasks, including natural language understanding, text generation and question anwsering @xcite @xcite . in order to acquire general linguistic and semantic knowledge, these pretraining methods are usually performed on open-domain corpus, like wikipedia and bookscorpus. in light of the success from open-domain pretraining, a further question is naturally raised: whether downstream tasks can also benefit from domain-adaptive pretraining? to answer this question, later work @xcite has demonstrated that continued pretraining on the unlabeled data in the target domain can further contribute to the corresponding downstream task. however, these studies are dependent on additional data that can be unavailable in certain scenarios, and they only evaluated on easy downstream tasks. for instance, @xcite perform continued pretraining with masked language modeling loss on several relevant domains, and they obtain improvements on eight well-studied classification tasks, which are too simple to exhibit the strength of continued domain-adaptive pretraining. besides, it is still unclear which pretraining objective is the most effective for each downstream task. in this work, we give a deeper analysis on how various domain-adaptive pretraining methods can help downstream tasks. specifically, we continuously pretrain a bert model @xcite with three different kinds of unsupervised pretraining objectives on the domain-specific training set of each target task. two of them are masked language model (mlm) @xcite and span boundary objective (sbo) @xcite , both objectives have been explored in previous work. in addition, a novel pretraining objective, namely perturbation masking objective (pmo), is proposed to better learn the correlation between arguments and predicates. after domain-adaptive pretraining, the adapted bert is then tested on dialogue understanding tasks to probe the effectiveness of different pretraining objectives. we evaluate on two challenging tasks that focus on dialogue understanding, i.e. conversational semantic role labeling (csrl) and spoken language understanding (slu). csrl @xcite was recently proposed by extending standard semantic role labeling (srl) @xcite with cross-utterance relations, which otherwise require coreference and anaphora resolution for being recognized. we follow previous work to consider this task as sequence labeling. on the other hand, slu includes intent detection and slot filling. to facilitate domain-adaptive pretraining, we only use the training set of each downstream task. in this way, the usefulness of each pretraining objective can be more accurately examined, as no additional data is used. experimental results show that domain-adaptive pretraining significantly helps both tasks. besides, our novel objective achieves better performances than the existing ones, shedding more lights for future work on pretraining. conversational semantic role labeling. xu et al. (2021) first proposed the csrl task, which extends standard srl by explicitly annotating other cross-turn predicate-argument structures inside a conversation. compared with newswire documents, human conversations tend to have more ellipsis and anaphora situations, causing more problems for standard nlu methods. their motivation is that most dropped or referred components in the latest dialogue turn can actually be found in the dialogue history. as the result, csrl allows arguments to be in different utterances as the predicate, while srl can only work on each single utterance. comparing with standard srl, csrl can be more challenging due to the long-range dependencies. similar to srl, we view csrl as a sequence labeling problem, where the goal is to label each token with a semantic role. spoken language understanding. proposed by @xcite , the slu task consists of two key components, i.e., intent detection and slot filling. given a dialogue utterance, the goal is to predict its intents and to detect pre-defined slots, respectively. we treat them as sentence-level classification and sequence labeling, respectively. ## domain-adaptive pretraining objectives while previous works have shown the benefit of continued pretraining on domain-specific unlabeled data (e.g., lee et al. (2020); gururangan et al. (2020) ), these methods only adopt the masked language model (mlm) objective to train an adaptive language model on a single domain. it is not clear how the benefit of continued pretraining may vary with factors like the objective function. in this paper, we use the dialogue understanding task as a testbed to investigate the impact of three pre-training objectives to the overall performance. in particular, we explore the mlm @xcite and span boundary objective (sbo) @xcite , and introduce a new objective, namely perturbation masking objective (pmo), which is more fit for the dialogue nlu task. ## experiments we evaluate pretraining objectives on three datasets, duconv, newsdialog foot_1 and crosswoz. the former two datasets are annotated by @xcite for the csrl task and the last one is provided by @xcite for the slu task. duconv is a chinese knowledge-driven dialogue dataset, focusing on the domain of movies and stars. newsdialog is a dataset collected in a way that follows the setting for constructing general open-domain dialogues: two participants engage in chitchat, and during the conversation, the topic is allowed to change naturally. @xcite annotates 3k dialogue sessions of duconv to train their csrl parser, and directly test on 200 annotated dialogue sessions of newsdialog. crosswoz is a chinese wizard-of-oz task-oriented dataset, including 6k dialogue sessions and 102k utterances on five domains. since the state-of-the-art models on these tasks are all developed based on bert, we use the same model architectures but just replace the bert base with our domain-adaptive pretrained bert. notice that, we also experiment with other pretrained language models such as roberta and xlnet. we observed similar results but here we only report the results based on bert due to the space limitation. in particular, we perform the domain-adaptive pretraining on csrl task using all dialogue sessions of training set in duconv @xcite and newsdialog @xcite , which includes 26k and 20k sessions, respectively; on the slu task, we use the whole crosswoz training dataset. the hyper-parameters used in our model are listed as follows. the network parameters of our model are initialized using the pretrained language model. the batch size is set to 128. we use adam @xcite with learning rate 5e-5 to update parameters. results and discussion. on the csrl task, we follow @xcite to use the micro-averaged f1 over the (predicate, argument, label) tuples. specifically, we calculate f1 over all arguments (referred as f1 all ) and those in the same and different dialogue turns as predicates (referred as f1 intra and f1 cross ). on the slu task, we report results on f1 intent , f1 slot and f1 all . let us first look at the csrl task. pretraining with mlm objective could slightly improve the performance by 0.4 and 0.12 in terms of f1 all on duconv and newsdialog, respectively. by additionally considering the span boundary objective, the overall performance especially f1 cross could be further improved by at least 0.75 and 2.6, respectively. these results are expected since arguments in the csrl task are usually spans and sbo is better than mlm in learning the span representation. we can also see that our proposed perturbation masking objective boosts the performance by a larger margin than sbo, indicating that learning correlations between arguments and predicates is more crucial to the nlu task. by summing three objectives, the csrl model could achieve the best performance, significantly improving the baseline that without domain-adaptive pretraining by 1.05 and 3.2 f1 all score, respectively. from we also investigate the impact of span masking scheme to the overall performance. recall that, in the span masking, we randomly sample the span length and a start position of the span. @xcite showed that no significant performance gains are observed by using more linguisticallyinformed span masking strategies such as masking named entities or noun phrases. specifically, they use the spacy's foot_2 named entity recognizer and constituency parser to extract named entities and noun phrases, respectively. in this paper, we revisit these span masking scheme. since there is no available constituency parser designed for the dialogue, we use an unsupervised grammar induction method @xcite to extract grammars from the training data. noun phrases from viterbi parse trees from different grammars are tallied without labels, resulting in a posterior distributions of the spans, which are used in our span sampling. as shown in ## conclusion in this paper, we probe the effectiveness of domainadaptive pretraining on dialogue understanding tasks. specifically, we study three domain-adaptive pretraining objectives, including a novel objective: perturbation masking objective on three nlu datasets. experimental results show that domainadaptive pretraining with proper objectives is a sim-ple yet effective way to boost the dialogue understanding performance.