Semi-supervised New Slot Discovery with Incremental Clustering (2022)

Abstract: Discovering new slots is critical to the success of dialogue systems. Most existing methods rely on automatic slot induction in unsupervised fashion or perform domain adaptation across zero or few-shot scenarios. They have difficulties in providing high-quality supervised signals to learn clustering-friendly features, and are limited in effectively transferring the prior knowledge from known slots to new slots. In this work, we propose a Semi-supervised Incremental Clustering method (SIC), to discover new slots with the aid of existing linguistic annotation models and limited known slot data. Specifically, we harvest slot value candidates with NLP model cues and innovatively formulate the slot discovery task under an incremental clustering framework. The model gradually calibrate slot representations under the supervision of generated pseudo-labels, and automatically learns to terminate when no more salient slot remains. Our thorough evaluation on five public datasets demonstrates that it significantly outperforms state-of-the-art models.

## introduction slot filling identifies contiguous word spans in an utterance based on slots to represent the meaning of user @xcite . it is essential for the performance of dialogue systems @xcite @xcite . traditional supervised methods have shown remarkable performance in this task @xcite @xcite . however, such approaches can only recognize pre-defined entity types from a limited slot set and require a significant amount of labeled training data, which is laborious and expensive to obtain. in practical settings, new unseen slots (as shown in figure 1 ) may emerge after the deployment of the dialogue system, rendering these supervised models ineffective. hence, there are works trying to do automatic slot induction without human labels. such methods often work in two steps: extract slot candidates and value first, then obtain slots via ranking. for example, @xcite combines semantic frame parsing with word embeddings for slot induction. @xcite further construct lexical knowledge graphs and perform a random walk to get slots. @xcite extend the ranking into an iterative process and build a slot tagger based on obtained slots for higher recall. nonetheless, the ranking process needs deliberate human intervention and largely affects the final results @xcite . moreover, instead of entirely without labeled data, we often have access to a small or partial amount of that in real practice. therefore, another line of efforts resort to zeroshot (or few-shot) cross-domain adaptation, whose goal is to identify unseen slots in the target domain by leveraging evidence from labeled data in the source domain. these methods can be organized into two types. in one type, slot description or even example values are directly interacted with user utterances to conduct one-stage slot filling individually for each slot @xcite @xcite @xcite . in the other, the slot filling task is decomposed into two or more stages @xcite @xcite . first, they identify all slot values from utterances by the coarse-grained binary sequence labeling. then these values are mapped to the representations of different slots in semantic space for slot assignment. however, there are several drawbacks in these methods. firstly, existing domain adaption efforts pay inadequate attention to slot value identification. they either ignore this process completely as in @xcite or oversimplify it as a coarse-grained sequence tagging @xcite . as evidenced in @xcite , identifying proper slot values is of critical importance for discovering new slots. secondly, existing methods tend to assume that new slot names, descriptions or even value examples are available, which is hardly true for slot discovery. overemphasizing these auxiliary description information inhibits the model to learn from existing known slots. also, they fail to provide proper guidance for the model to learn clustering-friendly features. in this work, we thus propose a semi-supervised incremental clustering method (sic) to discover new slots. instead of sequence labeling or span extraction, we design an iterative clustering and updating framework to gradually solicit evidence from both labeled and unlabeled data. as illustrated from figure 2 , relying on slot value candidates extracted via linguistic annotation models, we firstly pre-train a feature extractor with the limited labeled data under the supervision of the softmax loss. then, we iteratively perform clustering and feature extractor training. the former provides high-quality self-supervised signals to guide the latter training stage, while the latter yields clusteringfriendly features for the former. for each of the clustering stage, we estimate the cluster numbers and gradually expand for stable slot discovery. we evaluate the proposed model on five public datasets. it achieves new state-of-the-art performance across these datasets in various evaluation metrics. the contributions are summarized as follows: our work is closely related to automatic slot induction works, cross-domain slot filling and semisupervised spoken language understanding (slu) works. we briefly discuss their connections and analyze their differences. ## method in this section, we formally define the semisupervised new slot discovery task first. then we break into subsections to elaborate our proposed method as shown in figure 2 . starting from the candidate value extraction and feature extractor pre-training, we gradually extend to the iterative clustering and updating scheme. in the slot filling task, given an utterance @xmath0, ..., w n ⟩ with n tokens, the target is to predict a corresponding tag sequence @xmath1, ..., o n ⟩ in bio format. each tag o i can take three types of values: b-slot type, i-slot type and o, where band i-indicate the start and inside token of one slot type, and o means the token does not belong to any slot type. here, we work differently. suppose there is a candidate value @xmath2, • • • , w i+k ⟩ of length |x| = k + 1 identified from the utterance u , our setting is that in the whole dataset d with n candidate values, a limited amount are labeled we have y l y u which indicates that we have a set of new slots to discover and the number of these unknown slots is not given. in our work, the candidate values are extracted via existing nlp models. we also experiment on ground truth values to compare the difference. ## conclusion we presented a novel approach for semi-supervised new slot discovery in dialogue systems that detects new slots without any prior knowledge such as slot name, description or new slot number. our method removes the heavy reliance on large-scale annotated data and shows great potential in handling unseen situations for robust system deployment. it leverages off-the-shelf linguistic annotation models to extract candidate values, then builds an incremental clustering scheme to gradually solicit evidence from both labeled and unlabeled data to discover slot structures from the corpus. experiments on five datasets in four domains mark significant improvements over various groups of baselines on different evaluation metrics.