A Pointer Network-based Approach for Joint Extraction and Detection of Multi-Label Multi-Class Intents (2024)

Abstract: In task-oriented dialogue systems, intent detection is crucial for interpreting user queries and providing appropriate responses. Existing research primarily addresses simple queries with a single intent, lacking effective systems for handling complex queries with multiple intents and extracting different intent spans. Additionally, there is a notable absence of multilingual, multi-intent datasets. This study addresses three critical tasks: extracting multiple intent spans from queries, detecting multiple intents, and developing a multilingual multi-label intent dataset. We introduce a novel multi-label multi-class intent detection dataset (MLMCID-dataset) curated from existing benchmark datasets. We also propose a pointer network-based architecture (MLMCID) to extract intent spans and detect multiple intents with coarse and fine-grained labels in the form of sextuplets. Comprehensive analysis demonstrates the superiority of our pointer network based system over baseline approaches in terms of accuracy and F1-score across various datasets.

## introduction task-oriented dialogue systems have become a major field of study in recent years, significantly advancing the capabilities of natural language understanding (nlu). these systems execute command-based tasks, demonstrating versatility in handling diverse user queries through a set of predefined skills, known as intents. users interact with dialogue systems to fulfill their needs, and intent detection plays a pivotal role in comprehending user queries and generating appropriate responses in task-oriented conversations, thereby maintaining user engagement. the task of intent detection involves identifying the intent(s) within a given statement or query, which represents the underlying meaning conveyed by the user. for example, the query "how is the weather today?" would be associated with the getw eather intent. dialogue systems rely on detecting these intents to understand user queries and provide suitable answers. however, in real-world conversation, a query or a statement often contain multiple different intents. for instance, as shown in fig. 1 , for the query (from facebook english dataset): "remind me to pick up contact lenses tomorrow, set the alarm for 5 mins and 30 seconds", contains two distinct intent categories with following spans: 'remind me to pick up contact lenses tomorrow' ('set reminder' intent) and 'set the alarm for 5 mins and 30 seconds' ('set alarm' intent). both of these are fine intent categories. multiple similar fine intents can be merged to create one coarse intent as explained in over the past few years, researchers concentrate on intent identification across different domains. flexible and adaptive intent class detection models have been developed for dynamic and evolving realworld applications. @xcite figure 1 : examples of multi-label multi intent datasets (snips, @xcite et @xcite @xcite focus on streaming data to identify evolving new classes using incremental learning. @xcite , ifstc @xcite , senc-mas @xcite , sencforest @xcite , ecsminer @xcite aim at senc (streaming emerging new class) problem on intents on streams. @xcite work on emergence and disappearance of intents. @xcite uses high dimensional data for streaming classification. @xcite identifies multiple novel intents using a clustering framework. @xcite @xcite @xcite detect new intents in the form of outlier detection. unlike the previous single-intent detection models, which can easily utilize the utterance's sole intent to guide slot prediction, multi-intent slu (spoken language understanding) encounters the challenge of multiple intents, presenting a unique and worthwhile area of research. @xcite @xcite explore intent detection in different directions. agif @xcite , gl-gin @xcite , @xcite , @xcite work on multiple intent identification problem but these approaches do not detect the sentence spans related to different intents and also do not distinguish the primary and non-primary intents. based on convert @xcite backed framework, @xcite extract spans for different slots but does not extract and identify multiple intents. @xcite @xcite focus on entity extraction in different forms. previous research also includes both pipeline-based approaches @xcite and end-to-end methods @xcite @xcite . however, our work is different from the fact that we identify multiple intent spans along with their corresponding fine and coarse labels. our work differs from the fact that, we extract multiple intent spans from a given sentence and detect its coarse and fine intent labels. in this paper, we seek to address the following research ques-tions in the field of multi-label multi-class intent detection with span extraction: 1. we introduce a novel multi-label multi-class intent detection dataset (mlmcid-dataset) utilizing a diverse set of existing datasets with various intent sizes in multilingual settings (english and non-english languages), including coarse and finegrained intent labeling along with primary and nonprimary intent marking. 2. we thereafter, build a pointer network based encoder-decoder framework to extract multiple intent spans from the given query. empirical findings on various mlmcid datasets demonstrate that our pointer network based roberta model surpasses other baselines methods including llms, achieving a higher accuracy with an improvement in macro-f1. ## dataset we conduct different experiments to evaluate our framework on various datasets -all of which are benchmark datasets in nlu domain. we consider three different sizes of the datasets (as per intent class count -mentioned within bracket) -(i) small: a) snips (10 intents) @xcite , b) atis (21 intents) @xcite , c) facebook multi-lingual (12 intents) @xcite (consisting of the comparable corpus of english, spanish and thai data), abbreviated as fb. intents of similar domains which convey a similar broader meaning and are manually grouped together to make coarse-grained labels from original fine-grained labels foot_0 . we use the facebook data from mlmciddataset comprising 1000 text instances and corresponding intent labels are annotated for its 3 variations -english, spanish and thai. the text instances of english, spanish and thai languages are termed as facebook (english), facebook (spanish) and facebook (thai) dataset respectively. ## problem definition to formally describe the multi-label, multi-class intent detection (mlmcid) problem setting, let there be an input sentence s @xmath0, w 2 , ..., w n } contains n words. the model aims to extract multiple intent spans along with their coarse and fine classes in the form of a sextuple, ## solution approach for the task of multi-label, multi-class intent detection (mlmcid), our goal is to jointly extract the intent spans along with detecting multiple coarse and fine intents. our mlmcid output representation is a sextuple format. we employ pointer network based architecture for joint extraction of the sextuple. following are the different components of solution framework approach: ## experiments to validate our proposed framework, we compare the pointer network model (pnm) of mlmcid while taking various embeddings as input: bert, roberta, distilbert, and electra on all datasets. we also explore different large language models (llama2-7b, gpt-3.5 and gpt-4) to check how effectively they can extract multiple intent spans and detect different intents. after that, we experiment with different variations of overall best performing roberta model -varying the training data findings 1: for coarse label intent detection, as shown in findings 3: for all english datasets, bert, roberta, distilbert and electra performs almost similar with decent accuracy and f1-score which signifies the utility of pointer network model based mlmcid architecture. findings 4: we observe that the llms (llama-2-7b, gpt-3.5, gpt-4) fall behind in performance from pointer network based approaches with different encoders, even though they are much larger than our proposed framework, thus strengthening the need for such a specialized mlmcid architecture. llama2-7b performs poorly among three llms -this may be due to the fact of less contextual understanding in this specific task. more details in appendix a. findings 5: roberta with pnm in mlmcid performs better than any other models for overall average accuracy and f1-score across all english datasets for both primary and average course and findings 6: for non-english languages like spanish @xcite and thai (facebook) datasets , we observe that for both fine and coarse grained intent labels, xlm-r and mbert both produce good results but xlm-r outperforms mbert in all aspects across all datasets and overall for both primary intent detection and overall average intent detection with intent span extraction. findings 7: to check the effectivity of span extraction by pointer network, we vary the similarity (extracted intent span vs actual intent span) threshold utilise that extracted span to check the overall accuracy. we check for 50% -90% similarity threshold range and overall framework (roberta with pnm) accuracies (for both primary and average intent) across all datasets for coarse and fine intent labels are shown in ## loss function we calculate loss of different intent classes across all samples for primary, non-primary intents and their respective primary and non primary spans as shown in equation 5, 6 and 7 respectively. for training our model, we minimize the sum of negative log-likelihood loss for classifying the intent and the four pointer locations corresponding to the primary and non primary intent spans as shown in equation 8. here, c is the number of intent classes and (y 1 ) ∈ {in c1 , in f 1 } and (y 2 ) ∈ {in c2 , in f 2 }. (y 1 )ij and (y 2 )ij are the one-hot ground truth labels for sample i and class j for the primary and non-primary intents respectively, and p ij is the predicted probability for sample i and class j. n represents the n th training instance with n being the batch size, j represents the j th decoding time step with j being the length of the longest target sequence among all instances in the current batch. st p , e p ; @xmath1, p 2 } respectively represent the softmax scores corresponding to the true start and end positions of the primary and non primary spans. fig 3 shows the