Enhancing Joint Multiple Intent Detection and Slot Filling with Global Intent-Slot Co-occurrence (2022)

Abstract: Multi-intent detection and slot filling joint model attracts more and more attention since it can handle multi-intent utterances, which is closer to complex real-world scenarios. Most existing joint models rely entirely on the training procedure to obtain the implicit correlation between intents and slots. However, they ignore the fact that leveraging the rich global knowledge in the corpus can determine the intuitive and explicit correlation between intents and slots. In this paper, we aim to make full use of the statistical co-occurrence frequency between intents and slots as prior knowledge to enhance joint multiple intent detection and slot filling. To be specific, an intent-slot co-occurrence graph is constructed based on the entire training corpus to globally discover correlation between intents and slots. Based on the global intent-slot co-occurrence, we propose a novel graph neural network to model the interaction between the two subtasks. Experimental results on two public multi-intent datasets demonstrate that our approach outperforms the state-of-the-art models.

## introduction spoken language understanding (slu) aims to enable the machine to understand user's utterance, which contains two typical subtasks: intent detection and slot filling @xcite . intent detection obtains the user's intent from the input utterance and slot filling recognizes entities carrying detailed information of the intent. @xcite found that complex scenarios of real-world may often include multiple intents in an utterance. take an example in figure 1 , the utterance indicates two intents: "getweather" and "bookrestaurant". @xcite suggested that solving the two subtasks jointly makes better performance because of the correlation between intents and slots (e.g. "bookrestaurant" and b-restaurant_type). therefore, researchers in recent years @xcite @xcite pay more attention to joint multiple intent detection and slot filling. @xcite first tried to explore the joint model for multiple intent detection and slot filling based on attention with a slot-gated mechanism to model dependencies between intents and slots. @xcite proposed an adaptive graph-interactive framework (agif) where the core is an adaptive intent-slot interaction multilayer graph attention network. it makes it possible for each token to capture different relevant intent information, thus enabling fine-grained integration of multiple intents. on the basis of agif, @xcite proposed a global-locally graphinteraction network (gl-gin), which builds a local slot-aware graph and a global intent-slot graph for each utterance to model slot dependency and intentslot interaction. nevertheless, these works rely entirely on model training to obtain implicit intentslot correlation for interaction between the two subtasks. and existing works often construct graphs for each individual utterance, which neglects global statistical knowledge from the entire corpus. the simplified example is illustrated in figure 2(a) . to overcome the above limitations, inspired by the success of leveraging global knowledge from training sets to help model optimization in multilabel classification @xcite @xcite , we decide to explicitly leverage the intent-slot correlation summarized from the 7967 entire training corpus. by counting co-occurrence numbers between intents and slots in utterances and calculating the frequency, we visualize the statistical results on the mixatis (which is a public multi-intent utterance dataset) in figure 2 (c). we can clearly see that there are significant differences in co-occurrence frequency between different intents and slots. based on the observation, we propose to regard the co-occurrence frequency as prior knowledge for guiding the model to mine the correlation between intents and slots. methodologically, we devise an intent-slot graph with intent and slot labels as vertices, and the statistical co-occurrence frequency between intents and slots across the entire training corpus is formulated as weighted edges. figure 2(b) shows the simplified example of our graph. the weight between each intent and each slot can be regarded as the correlation degree of them. then, we apply the graph convolutional network (gcn) to the intent-slot graph to propagate information and update vertices embeddings. in this way, integrated global knowledge from the corpus is capable of helping the model capture explicit correlation between intents and slots, thereby facilitating interactions between the two subtasks to boost performance. moreover, we explore several attention and fusion mechanisms to extract label-related information from each utterance and the global training corpus, for the purpose of incorporating semantic information of tokens that play an important role in judging slots and intents, while reducing noise from irrelevant tokens. we conduct experiments on two public multiintent utterance datasets. experimental results show that our approach significantly outperforms the previous state-of-the-art model in the overall accuracy of utterances. and extensive experimental analyses justify that our approach can make better use of the global intent-slot co-occurrence to help the model capture more accurate correlations between intents and slots, so that enhance joint multiple intent detection and slot filling. the source code for this paper can be obtained from https://github.com/smxiao/gisco . ## methodology problem definition given input utterance @xmath0, u 2 , ..., u n ], where n is the length of the utterance. joint multiple intent detection and slot filling contains two subtasks: (1) multiple intent detection can be seen as a multi-label classification task that predicts the multiple intents o @xmath1, ..., o i m ] of the input utterance, where m indicates the number of output intent labels; (2) slot filling can be defined as a sequence labeling task that predicts the slot label for each token of the input utterance and output a slot label sequence o @xmath2, ..., o s n ]. as shown in figure 3 , our approach is composed of four major components: ( 1 ) an utterance encoder; (2) intent and slot label embedder; (3) intentslot co-occurrence graph network; (4) multi-intent detection and slot filling decoders. and we use a joint training scheme to optimize multiple intent self-attentive encoder intra-utterance attention bilstm intra-corpus attention intra-utterance attention intra-corpus attention fusion bilstm intent decoder slot decoder … … … … fusion … … … … … … : intent labels matrix : slot labels matrix [ ] utterance encoder intent and slot label embedder intent-slot co-occurrence graph intent and slot decoders gcn intent labels representation slot labels representation intent labels updated representaiton slot labels updated representaiton adjacency matrix … … … m i is updated during training m is updated during training detection and slot filling simultaneously. next, we will describe each component in detail. ## related work the study of joint model for intent detection and slot filling has been central to spoken language understanding in recent years. because of strong dependency between intent categories and slot labels, jointly solving the two tasks is not only more effective through guiding each other, but also usually requires only one model to be trained and finetuned, which reduces the impact of cascading error compared with pipeline models to some extent. @xcite proposed the first joint model named triangular chain conditional random fields for slot filling and intent detection, and the model outperforms the most advanced pipeline methods at that time. @xcite applied neural network on utterance parsing tree to solve the joint task. @xcite explored the strategy of integrating explicit alignment information for slot filling, and further applies attention mechanism to a recurrent neural network (rnn). @xcite introduced the slot-gated mechanism which was the first attempt to explicitly guide slot filling with intent information. @xcite adopt a bidirectional interaction attention module, which means that the interaction is not only from intent to slot, but also from slot to intent oppositely. @xcite first attempted to use graph neural network to solve the problem of nonparallelization of rnn and inability to capture the remote dependency. @xcite considered the cross-impact between intent detection and slot filling and proposes a co-interactive transformer for the joint task. however, the above joint models can mainly deal with single intent utterance, and cannot handle complex cases with multiple intents. based on the observation that 52% of utterances in an amazon internal dataset contain more than one intent, @xcite first focused on multiple intent scenario. it utilizes attention with slot-gated mechanism to construct a joint framework for multiple intent detection and slot filling. but their model predicts the slot for each token guided with same intent information. for more fine-grained integration of intent information, @xcite applied multi-layer graph attention network for adaptive interaction from intent to slot. further, aiming at the issue of slow inference speed of existing autoregressive models, @xcite proposed the first non-autoregressive model for multiple intent detection and slot filling, which is a global-local graph framework to model slot dependency locally and propagate information from multiple intents to slots globally. in our work, as far as we know, we are the first to consider leveraging the global intent-slot cooccurrence across the entire training corpus to enhance the joint of the two subtasks. because taking full advantage of the well-informed co-occurrence information between intents and slots from the entire corpus can help model determine more intuitive and accurate correlation between them than constructing a graph for each independent utterance in previous studies. the entire corpus to enhance joint multiple intent detection and slot filling. specifically, we construct a global intent-slot co-occurrence graph to discover correlation between intents and slots. then, we combine it with a gcn-based model to guide the information propagation and achieve interaction for the joint task. besides, we explore several attention mechanisms for dynamically extracting related information from independent utterance context and capturing shared label-specific features among all utterances in the training corpus, then fuse them adaptively to represent intent and slot label. experimental analyses on two public multi-intent datasets justify that our approach is effective and efficient.