Group is better than individual: Exploiting Label Topologies and Label Relations for Joint Multiple Intent Detection and Slot Filling (2022)

Abstract: Recent joint multiple intent detection and slot filling models employ label embeddings to achieve the semantics-label interactions.However, they treat all labels and label embeddings as uncorrelated individuals, ignoring the dependencies among them. Besides, they conduct the decoding for the two tasks independently, without leveraging the correlations between them.Therefore, in this paper, we first construct a Heterogeneous Label Graph (HLG) containing two kinds of topologies: (1) statistical dependencies based on labels’ co-occurrence patterns and hierarchies in slot labels; (2) rich relations among the label nodes.Then we propose a novel model termed ReLa-Net.It can capture beneficial correlations among the labels from HLG.The label correlations are leveraged to enhance semantic-label interactions. Moreover, we also propose the label-aware inter-dependent decoding mechanism to further exploit the label correlations for decoding. Experiment results show that our ReLa-Net significantly outperforms previous models.Remarkably, ReLa-Net surpasses the previous best model by over 20% in terms of overall accuracy on MixATIS dataset.

## introduction intent detection and slot filling @xcite are two fundamental tasks in spoken language understanding (slu) @xcite which is a core component in the dialog systems. in recent years, a bunch of joint models @xcite @xcite @xcite @xcite have been proposed to tackle single intent detection and slot filling at once via modeling their correlations. in real-world scenarios, a single utterance usually expresses multiple intents. to handle this, multi-intent slu @xcite is explored and @xcite first propose to jointly model the multiple intent detection and slot filling in a multi-task framework. qin to overcome the above challenges, in this paper, we first construct a heterogeneous label graph (hlg) including both the global statistical dependencies and slot label hierarchies, based on which we define a bunch of edge types to represent the topological structures and rich relations among the labels. then we propose a novel model termed recurrent heterogeneous label matching network (rela-net) to capture the beneficial label correlative information from hlg and sufficient leverage them for tackling the joint task. to capture the label correlative information from hlg, we design a heterogeneous label graph transformations (hlgt) module, which conducts relationspecific information aggregation among the label nodes. we design a recurrent dual-task interacting module to leverage label correlations for semanticslabel interactions. at each step, it takes both tasks' semantic hidden states and label knowledge generated at the previous step as input. then sequencereasoning bilstm @xcite and gats @xcite are utilized for interactions. as for decoding, we design a novel label-aware inter-dependent decoding mechanism that takes both the hidden states and label embeddings as input and measures their correlation scores in the joint label embedding space. by this means, the joint label embedding space serves as a bridge to connect the two tasks' decoding processes which then can be guided by the dual-task inter-dependencies conveyed in the learned label embeddings. we evaluate our rela-net foot_0 on benchmark datasets and the results show that rela-net achieves new state-of-the-art performance. further analysis proves that our method can capture nontrivial correlations among the two tasks' labels and effectively leverage them to tackle the joint task. ## problem definition given a utterance u, the task of joint multiple intent detection and slot filling aims to output a intent label set o @xmath0, ..., o i m } and a slot label sequence o @xmath1, ..., o s n }, where n is the length of u and m is the number of intents expressed in u. ## heterogeneous label graph mathematically, the hlg can be denoted as @xmath2, e, a, r), where v is the set of nodes, e is the set of edges, a is the set of node types and r is the set of relations or edge types. as shown in fig. 3 , there are three types of nodes: intent nodes, slot nodes, and pseudo nodes, which correspond to the intent labels, slot labels, and pseudo slot labels. in hlg, there are two kinds of topologies, which correspond to statistical and hierarchical dependencies, respectively. to represent and capture these dependencies, we define 12 relations on hlg: r 1 ∼ r 8 for statistical dependencies and r 9 ∼ r 12 for hierarchical dependencies, whose definitions are shown in b-depart_date. date_relative b-fromloc. city_name i-fromloc. city_name depart_date b-flight_time atis_flight_time fromloc r1: i2i_stat_dep r2: i2s_stat_dep r3: s2s_stat_dep r4: s2i_stat_dep r5: i2i_stat_strong_dep r6: i2s_stat_strong_dep r7: s2s_stat_strong_dep r8: s2i_stat_strong_dep r9: b2i_hierarchy r10: i2b_hierarchy r11: parent2child_hierarchy r12: child2parent_hierarchy r6 r7 r7 r4 r3 r9 r10 r12 r11 r12 r11 : pseudo node : intent node : slot node ϕ node j with the corresponding relation. note that in each sample, most slot labels are o which denotes the word has no meaningful slot. considering the slot label o cannot provide valuable clues, although it has high conditional probabilities (always 1.0) with other labels. therefore, the slot node o is set as an isolated node in hlg. r 1 ∼ r 8 are based on the hierarchies we discovered in slot labels. a b-slot node can connect and interact with many nodes, including intent nodes, other b-slot nodes, its parent pseudo node, and its i-slot node. differently, an i-slot node only connects with its b-slot nodes because there is a natural affiliation relation between a pair of b-and i-slot labels: an i-slot label can only appear after its b-slot label. as for the pseudo labels we defined, there is a parent-child relationship between the parent pseudo labels and their child b-slot labels. in hlg, a pseudo label works like an information station, which allows its children (b-slot nodes) to share their semantics and features. self-attentive semantics encoder intent bilstm global-gat intent matching decoder slot bilstm local-gat slot matching decoder heterogeneous label graph transformations dynamically-masked heterogeneous label graph transformations estimated intent labels estimated slot labels intent label knowledge projection slot label knowledge projection tokenlevel slot label knowledge sentencelevel slot label knowledge t x predicted intents predicted slots joint set of intent and slot labels input utterance sentencelevel intent label knowledge recurrent dual-task interacting 4 rela-net overview. the architecture of our rela-net is shown in fig. 4 . firstly, the initial label embeddings containing beneficial label correlations are generated by the proposed heterogeneous label graph transformations (hlgt) module, and the initial semantic hidden states are generated by the self-attentive semantics encoder. then we allow the two tasks to interact recurrently. at each time step, for each task, the semantic hidden states and the label knowledge of both tasks are fused, and their correlations are learned in a bilstm and then a gat. then the designed label-aware inter-dependent decoding mechanism produces the estimated labels via measuring the correlations between the hidden state and the label embeddings. the obtained labels are fed to the designed dynamically-masked heterogeneous label graph transformations (dm-hlgt) module, which dynamically derives the sample-specific label knowledge via operating on a dm subgraph of hlg. then the obtained label knowledge is used at the next time step. finally, after t step, the last produced labels are taken as the final predictions. next, we introduce the details of each module. ## related work to capture and leverage the dual-task correlations, a group of models @xcite @xcite @xcite @xcite @xcite @xcite have been proposed for joint intent detection and slot filling. however, they only consider the single-intent scenario. unlike them, we tackle the joint task of multiple intent detection and slot filling, which is more challenging and practical in the real-world scenario. recently, multi-intent slu has received increasing attention since it can handle complex utterances expressing multiple intents. @xcite propose a two-stage model to tackle the multiple intent detection task. and @xcite propose the first joint model that employs a multi-task framework including a slot-gate mechanism to tackle the joint task of multiple intent detection and slot filling. with the wide utilization of graph neural networks (gnns) in various nlp tasks @xcite @xcite , state-of-the-art multi-intent slu systems also leverage gnns to model the cross-task interactions. @xcite utilize gats to introduce the fine-grained information of the predicted multiple intents into slot filling in an adaptive way. further, @xcite propose a gat-based model in which slot filling is conducted in a nonautoregressive manner. previous models ignore the correlations among the two tasks' labels, treating their embeddings as separated parameters to be learned. our method is significantly different from previous ones. this is the first work to discover, capture and leverage the topologies and relations among the labels for joint multiple intent detection and slot filling. ## conclusion in this work, we improve joint multiple intent detection and slot filling from a new perspective: exploiting label typologies and relations. specifically, we first design a heterogeneous label graph to represent the statistical dependencies and hierarchies in rich relations. then we propose a recurrent heterogeneous label matching network to end-to-end capture and leverage the beneficial label correlations. experiments demonstrate that our method significantly outperforms previous models, achieving new state-of-the-art. further studies verify the advantages of our methods and show that our model can generate better hidden states and learns nontrivial label embeddings. in the future, we will apply our method in other multi-task scenarios.