A Label-Aware BERT Attention Network for Zero-Shot Multi-Intent Detection in Spoken Language Understanding (2021)

Abstract: With the early success of query-answer assistants such as Alexa and Siri, research attempts to expand system capabilities of handling service automation are now abundant. However, preliminary systems have quickly found the inadequacy in relying on simple classification techniques to effectively accomplish the automation task. The main challenge is that the dialogue often involves complexity in user’s intents (or purposes) which are multiproned, subject to spontaneous change, and difficult to track. Furthermore, public datasets have not considered these complications and the general semantic annotations are lacking which may result in zero-shot problem. Motivated by the above, we propose a Label-Aware BERT Attention Network (LABAN) for zero-shot multi-intent detection. We first encode input utterances with BERT and construct a label embedded space by considering embedded semantics in intent labels. An input utterance is then classified based on its projection weights on each intent embedding in this embedded space. We show that it successfully extends to few/zero-shot setting where part of intent labels are unseen in training data, by also taking account of semantics in these unseen intent labels. Experimental results show that our approach is capable of detecting many unseen intent labels correctly. It also achieves the state-of-the-art performance on five multi-intent datasets in normal cases.

## introduction in spoken language understanding (slu) of taskoriented dialog systems, each utterance is often interpreted as a kind of action being performed by the speaker, which we call speech or dialog acts @xcite . these acts may commit speakers to some course of actions, like asking or acknowledging, along with a series of distinctive semantic notions involved in a task. usually the system forms the semantic frames by identifying intents and slots to express dialog acts. for instance, given a sample utterance, "are there any accidents on my route to work at 10 ?", the intent detection task will first identify intents, i.e., 'get info traffic', 'get location work' and then the slot-filling task will predict a slot such as (time:10). in such case, an 'intent label' for an utterance is defined as a purpose or a goal that clearly states user's act. dominant slu systems have adopted several techniques to predict single intents by treating it as a multi-class classification problem @xcite @xcite . however, in real world scenario, many utterances may have multiple intents @xcite like the above example. multi-intent slu often requires more sophisticated reasoning on given utterances to disambiguate different intent natures. @xcite first explored the joint multi-intent and slotfilling task by treating multi-intents as a single context vector, but not scalable to a large number of intents. @xcite further proposed a stateof-the-art model to consider each intent-slot interaction via adaptive graph attention. however, these approaches cannot successfully tackle more complex multi-intent scenarios when sentences may not have explicit conjunctions. the second challenge in slu intent detection is intent fluidity variation, which we refer to the extent of naturalness when a dialogue progresses. in less stylized conversations, they usually contain a less bounded set of intents which may change with dialog context/states. thus, usually some utterances' intents may not be seen during training and this problem deteriorates in the multi-intent scenario @xcite . second, there is no rigorous definition of an intent annotation format or how many intents should be defined. therefore, conventional models trained on one dataset with a fixed set of intent labels may possibly fail to detect a new in-domain intent. we refer it to the zero-shot problem. @xcite suggests a two-stage process to first classify if a query is in-scope; then to assign intents. however, it cannot scale easily to unseen intents in multi-intent scenario. to tackle the above two challenges, we found that leveraging embedded semantics in intent labels may be useful. in conventional intent classification, these systems usually classify an utterance to a label which is represented by an indexed id like 0 (i.e. one-hot encoding). however, representing intents with indexed ids fails to consider embedded semantics in the labels too. for instance, we can use words 'get' and 'direction' in an intent label 'get direction' to help with identifying semantically equivalent words in an utterance, i.e., i 'want direction' to sf. for a given set of intent labels within one domain, we can compare the semantic similarity between words in an utterance and words in these intents. similarly in the zero-shot setting, even if some intents may not be visible during training, we could still compare the word semantics in these intents with a new utterance. in this paper, we propose our new framework: label-aware bert attention network (laban) in figure . 1. we first introduce bert to capture the multi-intent natures when utterances do not have explicit conjunctions. then, instead of treating intent labels only for indexed ids, we use words in each intent label in training data to construct a label embedding space. after encoding an utterance and all intents in a given training set for embeddings separately, a label-aware layer will generate scores of how likely this utterance belongs to each intent. to accommodate the zeroshot case, we could additionally introduce unseen intents' embeddings too to jointly construct the embedding space. in contrast with prior works' limited predictability only on seen intents, our model unfreezes the constraint by considering semantics in intent labels to deal with new unseen labels. the code and resources are released in https://github.com/waynewu6250/laban . the paper has the following contributions: multi-intent detection intent detection mainly aims to classify a given utterance with its intents from user inputs. different approaches such as convolutional-lstm and capsule network have been proposed to solve the problem @xcite @xcite . considering intents highly associated with slot-filling, many joint models @xcite @xcite @xcite utilize intent information like gradients or crossimpact networks to further reinforce the slot-filling prediction. however these methods do not consider multiple intent cases. therefore, @xcite first adopted hierarchical structures to identify multiple user intents. @xcite and @xcite further exploited interactive relations between intents and slots. @xcite leveraged the dialog context to better harness the joint tasks. our model follows these models' paradigm and focuses on more complex cases: 1) multi intents no longer exist in separate parts of the sentence which our bert introduction can be beneficial and 2) some testing intents are not available during training. zero-shot learning zero-shot learning (zsl) aims to recognize objects whose instances may not be seen during training @xcite . early works usually focused in the fields of computer vision @xcite @xcite . they adopted a two-stage approach to first identify object's attributes and estimated class posteriors based on similarity, which often suffered from domain shift between intermediate and target tasks. recent advances in zsl directly learned a mapping between feature and semantic spaces @xcite @xcite or built a common intermediate space @xcite . similar treatment could be applied in natural language. @xcite proposed cdssm to consider cosine similarity of deep semantics from utterances and intents. @xcite and @xcite extended zsl in user intent detection with capsule neural networks. si training utterances will i make it to the beach by noon if i leave now? i need to know if there's a lot of traffic on my way home? … how long will it take me to drive to fort collins? test utterances intent labels 𝑌 et al. (2021) proposed disentangled intent representations for multi-task training. we follow these works and extend to multi-intent detection cases with intent semantics and pretrained models. ## problem formulation in this section, we formally state the multi-intent detection problem in the normal and zero-shot case. ## conclusion in this paper, we propose the extension of finetuning bert and label-aware semantic interactions into the multi-intent detection task in slu. it successfully provides the solution to zero/few-shot setting where there are unseen labels in new utterances. by considering the label semantics, we can generate scores of how likely new utterances belong to these unseen intents. we compare the performance of our approach with previous methods and obtain significant improvements over baselines. it sheds the light that constructing a label semantic space could help the model to distinguish seen and unseen intents in utterances better. it provides the guidance in the work of improving slu zero-shot multi-intent detection by considering dialogue contexts and external knowledge learning, or a more challenging task of detecting out-of-domain (ood) detection where unseen intents are not available.