Intent Discovery with Frame-guided Semantic Regularization and Augmentation (2023)

Abstract: Most existing intent discovery methods leverage representation learning and clustering to transfer the prior knowledge of known intents to unknown ones. The learned representations are limited to the syntactic forms of sentences, therefore, fall short of recognizing adequate variations under the same meaning of unknown intents. This paper proposes an approach utilizing frame knowledge as conceptual semantic guidance to bridge the gap between known intents representation learning and unknown intents clustering. Specifically, we employ semantic regularization to minimize the bidirectional KL divergence between model predictions for frame-based and sentence-based samples. Moreover, we construct a frame-guided data augmenter to capture intent-friendly semantic information and implement contrastive clustering learning for unsupervised sentence embedding. Extensive experiments on two benchmark datasets show that our method achieves substantial improvements in accuracy (5%+) compared to solid baselines.

## introduction dialogue systems such as chatbots and virtual assistants increasingly rely on their ability to understand and identify user intents, which directly impacts the system's performance. @xcite . real-world dialogue systems have to deal with evolving user needs, resulting in continuously increasing intents. to address these challenges, intent discovery methods that aim to detect unknown user intents from a large number of unlabeled utterances have been developed in recent years @xcite . previous intent discovery work focuses on the unsupervised setting with clustering algorithms @xcite . @xcite proposed alternating-view k-means for joint multi-view representation learning and clustering u1: i am not seeing recent cash withdrawal on my account. pending cash withdrawal u2: i have a cash withdrawal that i don't recognize. cash withdrawal not recognized u3: why cash withdrawal on my account that i don't remember? similarities calculated by different models analysis. however, the unsupervised approaches often produce unsatisfactory performance because of their low degree of utilization of prior knowledge of known intents. therefore, recent works @xcite @xcite invariantly pretrain an intention detection model using limited known intents data, followed by leveraging clustering techniques to infer unknown intention. @xcite propose an iterative representation learning and clustering method, deepaligned, for intention detection and discovery. learning intention-friendly semantic representation is vital for semi-supervised methods to improve the performance of intent detection and discovery. however, it's insufficient to merely rely on the labeled data. methods pretrained on known intents are limited to the syntactic forms of sentences, therefore, fall short of recognizing adequate variations under the same meaning of unknown intents. figure 1 displays the semantic similarities between the sentences with the overlapping phrase "cash withdrawal on my account" but a different intention. red and green polylines represent the sentence similarity produced by the bert @xcite and deepaligned. it can be observed that both bert and deepaligned models ill-group u1 and u3 into the same class because the similarity between u1 and u3 is higher than the decision threshold (i.e., 86% illustrated cls] frame1 [sep] frame2 [cls] sentence 𝒙 𝒊 [cls] aug-sentence 𝒙 𝒊 𝒂 ℒ 𝐶𝐸 𝑓 sentence bert frame bert frame 𝒙 𝒊 𝒇 frame 1: <@xmath0: i have a pending payment ? frame 2: <@xmath1: a pending payment sentence 𝒙 𝒊 why is it saying i have a pending payment? aug-sentence 𝒙 𝒊 𝒂 why's it talk i have got a pending payment? 𝜇 𝜎 𝑧 ҧ 𝜇 ത 𝜎 ҧ 𝑧 transformer block 𝑙 + 1 transformer block 𝑙 + 1 transformer block 𝑙 + 1 transformer block 𝑙 + 1 [𝐂𝐋𝐒] [𝐂𝐋𝐒] ℒ 𝐾𝐿 ℒ 𝐶𝐸 ℒ 𝑆𝐶𝐿 as a black dash line in figure 1 ). but ideally, the model should group u2 and u3 into the same intent class (depicted as the blue line in figure 1 ). it is important to enhance the semantic representation learning ability of intent discovery methods. in this paper, we propose an approach utilizing frame knowledge as conceptual semantic guidance to bridge the gap between known intents representation learning and unknown intents clustering. frame knowledge is defined as the conceptual structure based on framenet @xcite , capturing the background knowledge necessary to understand a situation. specifically, we introduce a novel frame-guided semantic regulation and augmentation method to enhance intention discovery. semantic regularization minimizes the bidirectional kl divergence between model predictions made on the frame and sentencebased samples to learn conceptual semantic information. in the meantime, we construct a frameguided data augmenter to capture intent-friendly semantic information and implement contrastive learning for semi-supervised sentence embedding learning. finally, we perform an iterative representation learning and clustering method for intention detection and discovery. extensive experiments on two benchmark datasets show that our method achieves substantial improvements, over 5% in accuracy compared to solid baselines. in our work, we exploit kaf-spa @xcite to extract the frame annotation x f i , as shown in the figure 2 and produce augmented sentence x a i for the utterance x i . ## conclusion and future work this paper introduces frame knowledge as conceptual semantic guidance and proposes a frameguided semantic regularization and augmentation method. experimental results on two benchmarks demonstrate that our method outperforms strong baselines by a wide margin. in future work, we will explore practical ways to reason latent intentions based on the frame ontology knowledge.