Semantic-based Pre-training for Dialogue Understanding (2022)

Abstract: Pre-trained language models have made great progress on dialogue tasks. However, these models are typically trained on surface dialogue text, thus are proven to be weak in understanding the main semantic meaning of a dialogue context. We investigate Abstract Meaning Representation (AMR) as explicit semantic knowledge for pre-training models to capture the core semantic information in dialogues during pre-training. In particular, we propose a semantic-based pre-training framework that extends the standard pre-training framework (Devlin et al.,2019) by three tasks for learning 1) core semantic units, 2) semantic relations and 3) the overall semantic representation according to AMR graphs. Experiments on the understanding of both chit-chats and task-oriented dialogues show the superiority of our model. To our knowledge, we are the first to leverage a deep semantic representation for dialogue pre-training.

## introduction dialogue systems have attracted increasing attention from both academia and industry researches @xcite @xcite . the tasks can be commonly divided into two categories: task-oriented dialogue systems @xcite @xcite and chit-chat dialogue systems @xcite @xcite @xcite @xcite . the former aims to interact in the context of a specific task, while the latter chats with users without task and domain restrictions. despite differences in goals, a common challenge for both tasks is understanding the semantic information conveyed in a dialogue history. recently, semantic representations from pretrained language models have achieved remarkable * work done as an intern at tencent ai lab. success on a spectrum of dialogue tasks @xcite @xcite @xcite @xcite , where knowledge learned in pre-training over large-scale dialogue corpora can be transferred to downstream applications. current pre-training techniques typically focus on the surface text. however, they do not explicitly consider deep semantic clues beyond text, which leads to some unexpected behavior, such as paying attention to meaningless words @xcite , and suffering from spurious feature associations @xcite and adversarial attacks @xcite . incorporating semantic information into dialogue systems has been shown to be helpful for many downstream tasks, such as dialogue intent prediction @xcite , dialogue state tracking @xcite , and dialogue relation extraction @xcite . these methods first parse dialogue turns into semantic structures, and then incorporate them as extra features into neural systems. however, they 1) only focus on domain-specific benchmark data, leaving the general potentiality of semantic structures unexploited; 2) require either human annotations or an external parser to obtain semantic structures, raising costs or/and causing error propagation for real applications. we present sara, a semantic-graph-based pre-training framework for dialogues, aiming to endow a pre-trained dialogue model with a stronger ability to infer semantic structures from conversations by using explicit semantic structures for more fine-grained supervisions. in particular, we exploit the abstract meaning representation (amr; banarescu et al. 2013), a fine-grained deep structure widely adopted in semantic parsing @xcite @xcite @xcite and generation @xcite @xcite @xcite . as shown in figure 1 , amr represents a sentence using a rooted directed graph, highlighting the core semantic units (e.g., "police", "hum", "boy") in a sentence and connecting them with semantic relations (e.g., ":arg0", ":time"). we explicitly leverage amr graphs for pretraining our dialogue model. as shown in figure 2 , sara consists of three pre-training sub-tasks: 1) semantic-based mask language modeling, which extends the standard mask language modeling task @xcite by paying more attention to core semantic units in a dialogue; 2) semantic relation prediction, which aims to learn semantic relations between words; 3) semantic agreement, which optimizes the overall similarity between a dialogue and its corresponding amr graph. the sara combines strengths of both powerful contextualized representation of pre-trained models and explicit semantic knowledge, while eliminating the requirement of an external semantic parser in downstream applications. we choose bert @xcite and roberta @xcite models as backbone, which are then continual pre-trained on a largescale conversation dataset using our framework. experiments show that our semantic-based framework gives better results than current pre-training methods that use much more training data, achieving new state-of-the-art results on both chitchat understanding (dialogue relation extraction) and task-oriented dialogue understanding tasks (dialoglue benchmark). our method also gives better results than previous semantic-base systems on downstream tasks, without using an external parser. further analysis suggests that semantic information introduced by amr can help our model to better understand semantically complex dialogues. to our knowledge, we are the first to leverage deep semantic representation for dialogue pre-training. our code and the pretrained models are available at https://github. com/goodbai-nlp/sem-plm . wirsching et al. (2012) design a dialogue system which performs database operations based on semantic features. gupta et al. (2018) and aghajanyan et al. (2020) integrate intents and slots into a semantic tree and solve intent classification and slot-filling tasks as semantic parsing. cheng et al. (2020) represent task-oriented dialogue as a semantic graph to perform dialogue state tracking. a most related work is bai et al. (2021), who build dialogue-level amr graphs for both social chat understanding and dialogue response generation. our work is similar could help police house ##wife housewife police possible-01 help-01 :arg1 :arg1 :arg0 the police could help the house ##wife . in showing the effect of semantic knowledge for improving dialogue understanding. however, different from them, we focus on enhancing the language model with semantic knowledge during pre-training, and our model does not require an external amr parser in downstream applications. ## method figure 2 illustrates our semantic-based pre-training framework for dialogues. we take a pre-trained transformer @xcite encoder as the backbone, using amr as explicit semantic knowledge to continuously pre-train the model on dialogues in a multitask setting. in particular, the following three semantic-aware tasks are designed: the former two learn semantic knowledge from amr nodes and amr edges, respectively. the last task regularizes the overall representation of a dialogue using graph-level semantic features. we follow @xcite and construct dialogue-level amr graphs by 2 steps: 1) building utterance-level amr graphs by independently transforming utterances into amr using a pretrained amr parser. 2) connecting utterance-level amr graphs with a root node, where edges are labeled with the corresponding speaker. formally, denote an input dialogue sequence 1 as @xmath0, x 2 , ..., x n ], where n is the number of tokens in the dialogue. the corresponding amr is a directed acyclic graph @xmath1, e⟩, where v denotes a set of nodes (i.e., amr concepts) and e (i.e., amr relations) denotes a set of labeled edges. an edge can be further represented by a triple ⟨v i , r ij , v j ⟩, meaning that the edge is from node v i to v j with label r ij . ## experiments we evaluate the effectiveness of our semantic pretraining model on 8 dialogue tasks and compare the results with the state-of-the-art pre-trained and semantic-enriched models. ## conclusion we investigated the abstract meaning representation as explicit semantic clues for dialogue pre-training, using a semantic-based pre-training framework. experiments on two benchmarks show that the proposed framework is highly effective on both chit-chat understanding and task-oriented dialogue understanding. our method gives the best results on multiple datasets.