PCMID : Multi-Intent Detection through Supervised Prototypical Contrastive Learning (2023)

Abstract: Intent detection is a major task in Natural Language Understanding (NLU) and is the component of dialogue systems for interpreting users’ intentions based on their utterances. Many works have explored detecting intents by assuming that each utterance represents only a single intent. Such systems have achieved very good results; however, intent detection is a far more challenging task in typical real-world scenarios, where each user utterance can be highly complex and express multiple intents. Therefore, in this paper, we propose PCMID, a novel Multi-Intent Detection framework enabled by Prototypical Contrastive Learning under a supervised setting. The PCMID model can learn multiple semantic representations of a given user utterance under the context of different intent labels in an optimized semantic space. Our experiments show that PCMID achieves the current state-of-the-art performance on both multiple public benchmark datasets and a private real-world dataset for the multi-intent detection task.

## introduction natural language understanding (nlu) is a key component of dialogue systems, particularly for the task of intent detection @xcite to interpret users' utterances throughout conversations. intent detection can essentially be regarded as a semantic classification problem. given an utterance like "show me the type of aircraft that delta uses," intent detection can be implemented as a classifier that predicts an intent label for the utterance such as "aircraft type." this type of intent detection is often called single-intent detection which has been amply explored in previous works. many of these works adapt joint models to also capture the relationship between intent detection and slot filling @xcite * equally contributed. @xcite . these models currently achieve very promising results. despite the promising results on single-intent detection, the assumption that user utterances each represent only a single intent does not always hold in real-world scenarios where an utterance can represent multiple intents @xcite . for example, consider a natural expansion of the previous sample utterance: "show me the type of aircraft that delta uses and what time zone is denver in?". in this case, the classifier should predict multiple intent labels including aircraft type and city instead of just one. in scenarios such as these, failing to consider multiple intent predictions will result in incomplete natural language understanding and limit the performance of a dialogue system. in order to deal with this issue, gangadharaiah and narayanaswamy (2019) explore the joint multiintent detection and slot filling task using a multitask framework with a slot-gated mechanism @xcite . this work incorporates multi-intent information within a single context vector but does not integrate fine-grained intent information for token-level slots, resulting in ambiguous relationships between filled slots and intents. following this work, @xcite introduce an intentslot graph interaction layer to model correlations between slots and intents, but the solution does not scale to cases where conjunctions are implicitly defined in utterances. most recently, @xcite propose a labelaware bert attention network, which achieves the state of the art performance on multi-intent detection. this work first applies two bert encoders, one to capture the semantics of utterances without explicit conjunctions and the other for the words in the intent labels. then, a label-aware layer is constructed which iteratively projects intent label representations to utterance representations in order to compute a projection weight that represents the similarity between utterance and intent labels for the purpose of multi-intent detection. although this intuitive approach works well for detecting multiple intents, it fails to take the high complexity of the intent label embedding space into account and does not consider the semantics between individual utterance tokens and the intent label words. this results in ambiguity in matching intent labels to utterances such that the related intent information for each utterance token cannot be captured. furthermore, because two bert encoders are applied, model size and computational efficiency become issues for both the training and inference phases. therefore, in this paper, we propose our framework pcmid foot_0 : multi-intent detection through supervised prototypical contrastive learning, with the following main steps and associated contributions: we conduct experiments on multiple public benchmark datasets and a private real-world dataset and show that our approach achieves the current state-of-the-art performance on the multi-intent detection task. ## approach in this section, we describe in detail our proposed model architecture, pcmid, as illustrated in figure 1 . the architecture consists of an utterancelabel encoder module that uses a single bert encoder, a multi-head attention block module, and a multi-label supervised contrastive learning module. ## experiment datasets. we conduct experiments on three popular public datasets including mixatis foot_1 , mixs-nips foot_2 and facebook semantic parsing system (fsps) foot_3 , which contain 18, 7, and 25 unique intent labels, and 14749, 44173, and 44783 utterance samples, respectively @xcite @xcite . we also use a real-world private commercial dataset in the credit repair domain, referred to as credit16 foot_4 , which contains 16 unique intent labels and 4384 utterance samples. this dataset reflects the real-world challenges regarding the multi-intent detection task. we pre-process the mixatis, mixsnips and fsps datasets following the data processing pipeline from the previous work @xcite . also, we pre-process the credit16 dataset by lower-casing and then removing punctuation characters and emojis from the utterances and intent labels. next, we remove new-line characters and repeated white space characters from each utterance sample. we split the credit16 dataset into train, validation and test sets using the ratio of [0.8, 0.1, 0.1], respectively. the detailed dataset characteristics are shown in baseline experiments. we conduct experiments comparing our approach to previous works including: stack-prop @xcite , which proposes a shared self-attentive encoder and two decoders for joint intent and slot filling tasks separately. joint-mid-sf (gangadharaiah and narayanaswamy, 2019), which uses bi-lstms with a slot-gated mechanism for slot filling and intent detection tasks. agif @xcite , which adapts graph bi-lstm interactive architectures for both slot filling and intent detection tasks simultaneously. laban @xcite , which encodes both utterances and intent labels through two different encoders, along with an adaptive label-aware attentive layer for the multi intent detection task specifically. experimental setup. in our work, pcmid, we conduct experiments based on the pre-trained bert-base-uncased language model. we apply a pre-trained sentence embedding, the bert-base-nli-stsb-mean-tokens 6 from sentence transformer @xcite , to further improve our model's semantic understanding capability in our later experiments. additionally, we have also conducted experiments using pre-trained bert-large-uncased model in our pcimid framework. we use bert default pooling (pcmidpooler), mean pooling (pcmid-mean) and selfattentive pooling (s-pcmid-pooler) in our experiments. our experiments are conducted using an nvidia titan rtx with 24gb gpu memory. during our training we use the adamw optimizer as our parameter optimizer, apply steplr as 6 https://huggingface.co/sentence-transformers/ bert-base-nli-stsb-mean-tokens our learning rate scheduler, in which it adjusts the learning every 4 training steps with a multiplicative factor of learning rate decay equal to 0.9. we set the training epochs, initial learning rate, and batch size hyperparameters to 50, 1e -5 , and 128, respectively in our experiments on the mixatis, mixsnips, and fsps datasets. however, in our credit16 experiment, we failed to converge within the 50 training epochs, so we increased the number of training epochs to 100 for the credit16 dataset. we set the threshold for multi-label classification to 0.5 and evaluate the results based on the f1-score and accuracy metrics. we run the training experiments 10 times per dataset and obtain the trained model parameters based on the best validation accuracy epoch for computing the testing results. we report results of the final model by averaging 5 testing results per dataset. ## results multi-intent results. our main experimental results for the multi-intent detection task are shown in we find that all pcmid models achieve close results on the mixsnips datasets. s-pcmid base pooler, which is pcmid with bert default pooling using pre-trained sentence embeddings, is slightly better than other models. this fact suggests that the model can capture better contex- samples labels predictions 1 give me the fares from miami to cleveland next sunday and which airline is us atis_airfare, atis_airline atis_airfare, atis_airline 2 list la and how much is a limousine service in la guardia atis_city, atis_ground_fare atis_ground_fare 3 understand that i already have safety alert center and it's too expensive for what i already_client, money_too_expensive already_client, money_too_expensive 4 1 i don't have debt with you 2 i ignore your calls emails and messages because i'm no longer interested don't_owe_anything stop_calling, don't_owe_anything our work, pcmid, outperforms previous baseline results on every public benchmark dataset with only a single bert encoder. regarding our private credit16 dataset, we compare our work with la-ban and bert and show that our work still gets the best performance. however, we do not compare our work with the other baseline models (stack-prop, joint-mid-sf, agif) on the credit16 dataset since credit16 is not designed for the joint multi-intent detection and slot filling task, and is therefore not comparable to these works. single-intent results. in order to show the robustness of our work, we also evaluate our approach in the single-intent setting using atis 7 and 7 https://github.com/zjc664656505/pcmid/tree/ snips 8 datasets and compare the experiment results of our work with previous works. as can be seen in samples analysis as a motivational example, consider cases 1 and 2 in the it indicates that the model can be further trained with some domain specific terms like location, company, etc. because the credit16 dataset contains real world user conversations, the utterances in the dataset contain some noise. when faced with the imperfect case 3 (incomplete sentence) from the credit16, pcmid is able to make the correct prediction. even if human labels are corrupted, as in case 4, pcmid makes a correct classification ignoring the noise. in summary, pcmid is a robust model, but it can be further improved by learning the representation relationships between specific terminologies and labels. in terms of model size, laban's @xcite parameters are about twice that of our pcmid due to laban's use of a separate bert encoder for the label. but their performance is worse than our model. additionally, bert finetuning classification can yield a decent performance gain on some tasks. by simply building a cross-attention layer of target and utterances on top of bert, the model can be further boosted following the pcl strategies. with small additional costs, the model can gain consistent improvements, especially in the multi-label tasks. ## conclusion in our paper, we propose a multi-label classification model pcmid with a single bert encoder to handle the multi-intent detection task with smaller model size and higher computational efficiency but still achieve higher performance compared with previous state-of-art work. our approach maps complex utterance states to an utterance-label representation with respect to a joint label representation, and adapts prototypical contrastive learning to optimize both the utterance encoder and the representation. experiments and analysis demonstrate the effectiveness of our proposed model, which can further alleviate the discrepancies between utterance and label representations and improve the quality of label representations and the performance of multi-label classification. in the future, we will explore how to further optimize the label representations in crossdomains and study the effectiveness of the model on domain transfer tasks.