Syntactic Graph Convolutional Network for Spoken Language Understanding (2020)

Abstract: Slot filling and intent detection are two major tasks for spoken language understanding. In most existing work, these two tasks are built as joint models with multi-task learning with no consideration of prior linguistic knowledge. In this paper, we propose a novel joint model that applies a graph convolutional network over dependency trees to integrate the syntactic structure for learning slot filling and intent detection jointly. Experimental results show that our proposed model achieves state-of-the-art performance on two public benchmark datasets and outperforms existing work. At last, we apply the BERT model to further improve the performance on both slot filling and intent detection.

## introduction spoken language understanding (slu) plays a vital role in a task-oriented dialogue system. slot filling and intent detection @xcite are two major tasks for slu as shown in figure 1 @xcite . slot filling aims to obtain the semantic structure for the utterance. meanwhile, intent detection annotates the categorical intent of the utterance. in typical pipeline methods, slot filling and intent detection are built separately. slot filling is implemented as a standard sequence labeling task @xcite and intent detection is built as a classification task @xcite , respectively. essentially, slot filling and intent detection impact mutually. therefore, more prior work @xcite @xcite @xcite @xcite implement two aforementioned tasks jointly as multi-task learning and achieve more promising results than those pipeline methods. however, most prior work utilize sequential model, such as recurrent neural network, to accumulate the contextual representation for each word to implement slu with no consideration of prior linguistic knowledge. intuitively, slot filling and intent detection rely on indicative contextual words for disambiguation and suffer from the degradation on wide contexts. syntactic dependency parse tree as shown in figure 1(b) , which provides linguistic dependency relation among words, has been shown generally beneficial in various nlp tasks such as machine reading comprehension @xcite , neural machine translation @xcite and relation extraction @xcite . the major reasons are that the dependency parse tree can capture long-range relations between words and contain implicit clues for disambiguation. to access a better slu, we emphasize that slu model should utilize the dependency representation as prior linguistic knowledge. in this paper, we propose a joint slu model that applies a graph convolutional network (gcn) over dependency trees to integrate the syntactic structure for joint learning slot filling and intent detection, where the gcn can pool information over arbitrary dependency structures efficiently, which has been proven in @xcite . concretely, our proposed model encode the utterance and output a contextual word representation via a bi-directional lstm @xcite , then a gcn over dependency tree take contextual word representation as input to obtain the syntactic structure representation for utterance. in addition, it is worth noting that existing dependency parsers are impossible to parse all the sentences exactly. thus, a multi-head attention is utilized to fuse the syntactic representation with original contextual word representation against the errors caused by incomplete dependency parser, where the multi-head attention can supplement the syntactic representation with contextual word representation as a self-adaption manner. at last, the fused representation is applied to implement slot filling and intent detection jointly. the experiments are conducted on two benchmarks slu datasets: atis @xcite and snips @xcite . the experimental results demonstrate that our proposed model outperforms the existing state-of-the-art approaches. at last, bert model @xcite , as a pre-trained model, is used to our proposed framework. the experimental results also show that our proposed model incorporated with bert model can further improve the performance on both slot filling and intent detection. the main contributions of this work are therefore include as follows: 1)we introduce a model that utilizes a gcn to integrate the syntactic structure for joint learning slot filling and intent detection, which to the best of our knowledge is the first work that syntactic structure and gcn are used to implement above two tasks jointly. 2) we utilize a multi-head attention to fuse the syntactic representation with contextual word representation against the errors caused by incomplete dependency parser. 3) we conduct our experiments on two public datasets, and our proposed model achieves new the state-of-the-art performance in overall accuracy metric. ## methodology in this section, we will describe our syntactic graph convolutional network for slu tasks. the overall architecture of our model is demonstrated in fig 2 . we first use a bilstm encoder to obtain the contextual representation of an utterance. then we perform multi-hop gcn propagation over the dependency tree initialized by the hidden states of the bilstm encoder to capture syntactic representation. subsequently, we integrate the syntactic representation and the contextual hidden states via a feature aggregation layer. finally, we pass the fused representation to the output layer for final predictions. both slot filling and intent detection are optimized simultaneously via a joint learning scheme. ## qualitative analysis in this section, we present a detailed qualitative analysis on each component of our proposed model and provide certain typical cases to show the effectiveness of incorporating syntactic information. we first perform an ablation study to validate the effect of different modules of our model. then we explore more methods of feature aggregation and demonstrate the superiority of our proposed local-to-global multi-head attention mechanism. next, we give some typical cases of our model and baseline to show the effect of syntactic structure. finally, we conduct experiment with bert to verify that our model is more effective with pre-trained model. ## related work slot filling and intent detection are two major tasks for slu. recently, the typical pipeline methods build slot filling and intent detection separately, where slot filling is implemented as a standard sequence labeling task @xcite and intent detection is built as a classification task @xcite , respectively. more recent work @xcite @xcite @xcite @xcite implement slot filling and intent detection as a joint model to eliminate the error propagation without any linguistic knowledge. instead, our work apply the linguistic knowledge (i.e. dependency tree) as a prior to guide the learning slot filling and intent detection jointly. dependency tree, as a important linguistic knowledge, is applied to recent natural language processing tasks. in relation extraction and machine translation, many studies @xcite @xcite have show that the dependency trees can capture long-distance relations effectively. in machine reading comprehension, @xcite use syntax to guide the text modeling by incorporating explicit syntactic constraints into attention mechanism for better linguistically motivated word representations and achieve promising results on both squad 2.0 and race datasets. inspired by @xcite , our work utilize the dependency tree to guide the joint model for slot filling and intent detection. different from @xcite , our work apply the representation over dependency tree as a inner feature instead of syntactic constraints into attention mechanism. graph convolution network (gcn) also have been utilized for many natural language processing tasks. @xcite propose a flexible graph convolution based method for learning word embeddings. @xcite apply a gcn as sentence encoders to produce latent feature representations of words in a sentence for semantic role labeling task. @xcite present a simple and effective approach to incorporate syntactic structure by the way of gcn into the encoderdecoder model for machine translation. @xcite build a single text graph for a corpus based on word co-occurrence and document word relations, then learn a text gcn for the corpus to classify the text. different from above work, our work propose a model that applies a gcn over dependency trees to integrate the syntactic structure for joint learning slot filling and intent detection. ## conclusion in this paper, we propose a novel joint model that applies a graph convolution network over dependency trees to integrate the syntactic structure for learning slot filling and intent detection jointly. in addition, we utilize multi-head attention to fuse syntactic representation with contextual word representation to access complementary representation for slu task. experimental results show that our proposed model outperforms strong baseline models and achieves state-of-the-art performance on both atis and snips datasets in overall accuracy metric. finally, we apply the bert model to our framework and experiments demonstrate that our proposed model integrating bert model can improve the performance on both slot filling and intent detection more obviously.