Slot Induction via Pre-trained Language Model Probing and Multi-level Contrastive Learning (2023)

Abstract: Recent advanced methods in Natural Language Understanding for Task-oriented Dialogue (TOD) Systems (e.g., intent detection and slot filling) require a large amount of annotated data to achieve competitive performance. In reality, token-level annotations (slot labels) are time-consuming and difficult to acquire. In this work, we study the Slot Induction (SI) task whose objective is to induce slot boundaries without explicit knowledge of token-level slot annotations. We propose leveraging Unsupervised Pre-trained Language Model (PLM) Probing and Contrastive Learning mechanism to exploit (1) unsupervised semantic knowledge extracted from PLM, and (2) additional sentence-level intent label signals available from TOD. Our approach is shown to be effective in SI task and capable of bridging the gaps with token-level supervised models on two NLU benchmark datasets. When generalized to emerging intents, our SI objectives also provide enhanced slot label representations, leading to improved performance on the Slot Filling tasks.

## introduction natural language understanding (nlu) has become a crucial component of the task-oriented dialogue (tod) systems. the goal of nlu is to extract and capture semantics from users' utterances foot_1 . there are two major tasks in nlu framework, including intent detection (id) and slot filling (sf) @xcite . while the former focuses on identifying overall users' intents, the latter extracts semantic concepts from natural language sentences. in nlu tasks, intents denote sentence-level annotations while slot types represent token-level labels. despite recent advances, state-of-the-art nlu methods @xcite require a large amount of annotated data to achieve competitive performance. however, the fact that annotations, especially token-level labels, are expensive and time-consuming to acquire severely inhibits the generalization capability of traditional nlu models in an open-world setting @xcite . recent works attempt at tackling the problems in low-resource settings on both intent level @xcite @xcite and slot level @xcite . however, most approaches remain restricted to closed-world settings where there exist pre-defined sets of seen and emerging sets of classes. some approaches even require additional knowledge from related tokenlevel tasks that might not be readily available. additionally, with increasing exposure to the ever-growing number of intents and slots, tod systems are expected to acquire task-oriented adaptation capability by leveraging both inherent semantic language understanding and task-specific knowledge to identify the crucial emerging concepts in the users' utterances. this ability can be referred to as slot induction in tod systems. recently, pre-trained contextualized language models (plm) such as bert @xcite have shown promising capability of capturing semantic and syntactic structure without explicit linguistic pre-training objectives @xcite @xcite . despite imperfections, the captured semantics from plm via unsupervised probing mechanisms could be leveraged to induce important semantic phrases covering token-level slot labels. additionally, as an effective unsupervised representation learning mechanism @xcite , contrastive learning (cl) is capable of refining the imperfect plm semantic phrases in a self-supervised manner to mitigate biases existent in the plm. in specific, given a sample phrase in the same area corresponding to spatial_relation slot type, as a presumed structural knowledge, plm tends to split the preposition and determiner from the noun phrase during segmentation, resulting in in the and same area. despite its structural correctness, the identified segments fail to align with ground truth slots due to the lack of knowledge from the overall utterance semantics. on the other hand, cl can also be leveraged on a sentence level when intent labels are available. in fact, there exist strong connections between slot and intent labels @xcite . for instance, utterances with book_restaurant intent tend to contain location slots than those from rate_book intent. therefore, as intent labels are less expensive to acquire, they could provide additional signals for cl to induce slot labels more effectively when available. in this work, we propose leveraging plm probing together with cl objectives for slot induction (si) task. despite imperfections, plm-derived segmentations could produce substantial guidance for si when slot labels are not readily available. we introduce cl to further refine plm segmentations via (1) segment-level supervision from unsupervised plm itself, and (2) sentence-level supervision from intent labels to exploit the semantic connections between slots and intents. our refined bert from si objectives can produce effective slot representations, leading to improved performance in slot-related tasks when generalized towards emerging intents. our contributions can be summarized as follows: • we propose leveraging semantic segments derived from unsupervised plm probing (upl) to induce phrases covering token-level slot labels. we name the task as slot induction. contrastive learning contrastive learning (cl) has been widely leveraged as an effective representation learning mechanism @xcite . the goal of cl is to learn the discriminative features of instances via different augmentation methods. in natural language processing (nlp), cl has been adopted in various contexts ranging from text classification @xcite , embedding representation learning @xcite to question answering @xcite . cl has also been integrated with plm as a more effective fine-tuning strategy for downstream tasks @xcite . in our work, we propose an integration of cl with plm probing techniques to further refine imperfect plm-derived segments via (1) unsupervised signals from plm itself, and (2) less expensive sentence-level intent label supervision for improved si performance. ## problem formulation slot induction we introduce the task of slot induction (si) whose objective is to identify phrases containing token-level slot labels. unlike traditional sf and previously proposed aisi framework @xcite , in our si task, both slot boundaries and slot types are unknown during training. the task is also related to phrasal segmentation/ tagging (ps) methods @xcite . however, there are three key distinc- specifically, given an utterance with the length of t tokens @xmath0, x 2 ..., x t ], si task aims to make decisions at t -1 positions whether to (1) tie the current token with the previous one to extend the current phrase foot_2 , or (2) break away from the previous token/ phrase to form a new phrase. evaluation metric we adopt the break-tie (b-t) schema @xcite to evaluate si task. the metric allows for direct comparison between supervised sequential labeling and unsupervised ps methods. in si setting, tie represents the connection between tokens of the same slot type while break denotes the separation between (1) tokens from different slot types, and (2) tokens from a slot type and non-slot tokens. as the objective of si is on slot tokens, consecutive non-slot tokens should not contribute to the overall performance. therefore, additional na labels are introduced to guarantee that evaluations are only conducted on slot tokens and their adjacent tokens. figure 1 depicts the connections of sf and ps labels with b-t schema. for ps, break denotes the separation of two consecutive phrases. if no phrase is identified by ps methods, every token is considered as tie to one another. in the figure 1 example, as "south carolina" is the only identified phrase, the given sentence is simply split into two phrases where break denotes their junction. precision, recall and f-1 metrics are reported for individual labels, namely b-p,b-r,b-f1 for break and t-p,t-r,t-f1 for tie. given an utterance, an optimal si model makes correct decisions to either break and tie at every token index. therefore, h-mean, denoting the harmonic mean between f-1 scores of tie and break label predictions, is considered the golden criteria for si model comparison. ## proposed framework in this section, we introduce our proposed multilevel contrastive learning framework for si task with 2 major components: segment-level contrastive learning (segcl) and sentence-level contrastive learning (sentcl) as depicted in figure 2 . we first introduce the backbone unsupervised plm probing (upl) for both components. ## conclusion in our work, we propose the study of token-level slot induction (si) via an unsupervised pre-trained language modeling (plm) probing in conjunction with contrastive learning (cl) objectives. by leveraging both unsupervised signals from plm and sentence-level signals from intent labels via cl objectives, our proposed framework not only achieves competitive performance in comparison with other unsupervised phrasal segmentation baselines but also bridges the gap in performance with upper bound methods that require additional token-level labels on two nlu benchmark datasets. we also demonstrate that our proposed si training is capable of refining the original plm, resulting in more effective slot representations and benefiting downstream sf tasks when generalized towards emerging intents. further studies of better exploitation of full-depth segmentation trees, enhanced segment augmentation mechanisms and better semantic alignment extraction between slots and intents are promising directions for our future work. we also seek to extend the current si studies beyond english and towards multilingual nlu systems. @xcite @xcite