MISCA : A Joint Model for Multiple Intent Detection and Slot Filling with Intent-Slot Co-Attention (2023)

Abstract: The research study of detecting multiple intents and filling slots is becoming more popular because of its relevance to complicated real-world situations. Recent advanced approaches, which are joint models based on graphs, might still face two potential issues: (i) the uncertainty introduced by constructing graphs based on preliminary intents and slots, which may transfer intent-slot correlation information to incorrect label node destinations, and (ii) direct incorporation of multiple intent labels for each token w.r.t. token-level intent voting might potentially lead to incorrect slot predictions, thereby hurting the overall performance. To address these two issues, we propose a joint model named MISCA. Our MISCA introduces an intent-slot co-attention mechanism and an underlying layer of label attention mechanism. These mechanisms enable MISCA to effectively capture correlations between intents and slot labels, eliminating the need for graph construction. They also facilitate the transfer of correlation information in both directions: from intents to slots and from slots to intents, through multiple levels of label-specific representations, without relying on token-level intent information. Experimental results show that MISCA outperforms previous models, achieving new state-of-the-art overall accuracy performances on two benchmark datasets MixATIS and MixSNIPS. This highlights the effectiveness of our attention mechanisms.

## introduction spoken language understanding (slu) is a fundamental component in various applications, ranging from virtual assistants to chatbots and intelligent systems. in general, slu involves two tasks: intent detection to classify the intent of user utterances, and slot filling to extract useful semantic concepts @xcite . a common approach to tackling these tasks is through sequence classification for intent detection and sequence labeling for slot filling. recent research on this topic, recognizing the high correlation between intents and slots, shows that a joint model can improve overall performance by leveraging the inherent dependencies between the two tasks @xcite @xcite . a number of joint models have been proposed to exploit the correlations between single-intent detection and slot filling tasks, primarily by incorporating attention mechanisms @xcite @xcite @xcite @xcite . however, in real-world scenarios, users may often express utterances with multiple intents, as illustrated in figure 1 . this poses a challenge for single-intent systems, potentially resulting in poor performance. recognizing this challenge, @xcite is the first to explore the detection of multiple intents in a slu system, followed by @xcite who first propose a joint framework for multiple intent detection and slot filling. @xcite and @xcite have further explored the utilization of graph attention network @xcite to explicitly model the interaction between predicted intents and slot mentions. recent state-of-the-art models co-guiding @xcite and rela-net @xcite further incorporate the guidance from slot information to aid in intent prediction, and design heterogeneous graphs to facilitate more effective interactions between intents and slots. we find that two potential issues might still persist within these existing multi-intent models: (1) the lack of "gold" graphs that accurately capture the underlying relationships and dependencies between intent and slot labels in an utterance. previous graph-based methods construct a graph for each utterance by predicting preliminary intents and slots @xcite . however, utilizing such graphs to update representations of intents and slots might introduce uncertainty as intent-slot correlation information could be transferred to incorrect label node destinations. to overcome the above issues, in this paper, we propose a new joint model with an intent-slot coattention mechanism, which we name misca, for multi-intent detection and slot filling. equivalently, our novel co-attention mechanism serves as an effective replacement for the graph-based interaction module employed in previous works, eliminating the need for explicit graph construction. by enabling seamless intent-to-slot and slot-to-intent information transfer, our co-attention mechanism facilitates the exchange of relevant information between intents and slots. this novel mechanism not only simplifies the model architecture, but also maintains the crucial interactions between intent and slot representations, thereby enhancing the overall performance. in addition, misca also presents a label attention mechanism as an underlying layer for the coattention mechanism. this label attention mechanism operates independently of token-level intent information and is designed specifically to enhance the extraction of slot label-and intent label-specific representations. by capturing the characteristics of each intent/slot label, the label attention mechanism helps misca obtain a deep understanding and fine-grained information about the semantic nuances associated with different intent and slot labels. this, in turn, ultimately helps improve the overall results of intent detection and slot filling. our contributions are summarized as follows: (i) we introduce a novel joint model called misca for multiple intent detection and slot filling tasks, which incorporates label attention and intent-slot co-attention mechanisms. foot_0 (ii) misca effectively captures correlations between intents and slot labels and facilitates the transfer of correlation information in both the intent-to-slot and slot-to-intent directions through multiple levels of label-specific representations. (iii) experimental results show that our misca outperforms previous strong baselines, achieving new state-of-the-art overall accuracies on two benchmark datasets. ## problem definition and related work given an input utterance consisting of n word tokens w 1 , w 2 , ..., w n , the multiple intent detection task is a multi-label classification problem that predicts multiple intents of the input utterance. meanwhile, the slot filling task can be viewed as a sequence labeling problem that predicts a slot label for each token of the input utterance. @xcite show the significance of the multiple intents setting in slu. @xcite then introduce a joint approach for multiple intent detection and slot filling, which models relationships between slots and intents via a slot-gated mechanism. however, this slot-gated mechanism represents multiple intents using only one feature vector, and thus incorporating this feature vector to guide slot filling could lead to incorrect slot predictions. to generate fine-grained intents information for slot label prediction, @xcite introduce an adaptive interaction framework based on graph attention networks. however, the autoregressive nature of the framework restricts its ability to use bidirectional information for slot filling. to overcome this limitation, @xcite proposes a w 2 w 3 w n-1 w n ... bilstm i bilstm s task-shared utterance encoder intent decoder slot decoder b-city b-time o ... task-specific encoder intent-slot co-attention label attention ... ... w 1 ... ... ... ... ... ... o o ... ... ... ... ... label attention global-locally graph interaction network that incorporates both a global graph to model interactions between intents and slots, and a local graph to capture relationships among slots. more recently, @xcite propose two heterogeneous graphs, namely slot-to-intent and intent-to-slot, to establish mutual guidance between the two tasks based on preliminarily predicted intent and slot labels. meanwhile, @xcite propose a heterogeneous label graph that incorporates statistical dependencies and hierarchies among labels to generate label embeddings. they leverage both the label embeddings and the hidden states from a label-aware inter-dependent decoding mechanism to construct decoding processes between the two tasks. see a discussion on potential issues of these models in the introduction. please find more related work in section 4.3. ## our misca model figure 2 illustrates the architecture of our misca, which consists of four main components: (i) taskshared and task-specific utterance encoders, (ii) label attention, (iii) intent-slot co-attention, and (iv) intent and slot decoders. the encoders component aims to generate intentaware and slot-aware task-specific feature vectors for intent detection and slot filling, respectively. the label attention component takes these taskspecific vectors as input and outputs label-specific feature vectors. the intent-slot co-attention component utilizes the label-specific vectors and the slot-aware task-specific vectors to simultaneously learn correlations between intent detection and slot filling through multiple intermediate layers. the output vectors generated by this co-attention component are used to construct input vectors for the intent and slot decoders which predict multiple intents and slot labels, respectively. ## conclusion in this paper, we propose a novel joint model misca for multiple intent detection and slot filling tasks. our misca captures correlations between intents and slot labels and transfers the correlation information in both forward and backward directions through multiple levels of label-specific representations. experimental results on two benchmark datasets demonstrate the effectiveness of misca, which outperforms previous models in both settings: with and without using a pre-trained language model encoder.