APP : Adaptive Prototypical Pseudo-Labeling for Few-shot OOD Detection (2023)

Abstract: Detecting out-of-domain (OOD) intents from user queries is essential for a task-oriented dialogue system. Previous OOD detection studies generally work on the assumption that plenty of labeled IND intents exist. In this paper, we focus on a more practical few-shot OOD setting where there are only a few labeled IND data and massive unlabeled mixed data that may belong to IND or OOD. The new scenario carries two key challenges: learning discriminative representations using limited IND data and leveraging unlabeled mixed data. Therefore, we propose an adaptive prototypical pseudo-labeling(APP) method for few-shot OOD detection, including a prototypical OOD detection framework (ProtoOOD) to facilitate low-resourceOOD detection using limited IND data, and an adaptive pseudo-labeling method to produce high-quality pseudo OOD and IND labels. Extensive experiments and analysis demonstrate the effectiveness of our method for few-shot OOD detection.

## introduction out-of-domain (ood) intent detection learns whether a user query falls outside the range of pre-defined supported intents. it helps to reject abnormal queries and provide potential directions of future development in a task-oriented dialogue system @xcite @xcite @xcite @xcite . since ood data is hard to label, we need to rely on labeled in-domain samples to facilitate detecting ood intents. previous ood detection studies generally work on the assumption that plenty of labeled ind intents exist. they require labeled in-domain data to learn intent representations and then use scoring functions to estimate the confidence score of a test query belonging to ood. for example, @xcite proposes maximum softmax probability (msp) to use maximum softmax probability as the confidence score and regard a query as ood if the score is below a fixed threshold. @xcite further introduces another distance-based method, gaussian discriminant analysis (gda), which uses the maximum mahalanobis distance @xcite to all in-domain classes centroids as the confidence score. although these models achieve satisfying performance, they all rely on sufficient labeled ind data, which limits their ability to practical scenarios. in this paper, we focus on a more practical setting of ood detection: there are only a few labeled ind data and massive unlabeled mixed data that may belong to ind or ood. we call it as few-shot ood detection. note that the mixed data don't have labeled ind or ood annotations. considering that unlabeled data is easily accessible, we believe this setting is more valuable to be explored. however, few-shot ood detection carries two key challenges. (1) learning discriminative representations using limited ind data: ood detection requires discriminative intent representations to separate ind&ind intents and ind&ood intents. however, traditional models based on crossentropy or supervised contrastive learning @xcite fail to distinguish intent types under the few-shot setting. limited labeled ind data makes it hard to learn discriminative intent representations. (2) leveraging unlabeled mixed data: unlabeled data contains ind and ood intents which both benefit in-domain intent recognition and ood detection. but it's nontrivial to leverage the mixed data because we don't know prior knowledge of ood data. to solve the issues, we propose an adaptive prototypical pseudo-labeling (app) for few-shot ood detection. to learn discriminative representations, we propose a prototypical ood detection framework (protoood) using limited ind data. inspired by the idea of pcl @xcite , we introduce an instance-instance loss to pull together samples of the same class and an instanceprototype loss to enforce the prototypes to be center points of classes. after training a prototypical in-domain classifier using few-shot ind data, we compute the maximum cosine distance of an input query to all in-domain prototypes as the confidence score. if the score is above a fixed threshold, we believe it's an ood intent. compared to existing ood detection methods @xcite @xcite @xcite , our prototypical ood detection framework models rich class-level semantics expressed implicitly by training instances. empirical experiments in section 3.3 demonstrate our framework achieves superior performance both on ind and ood metrics. to leverage unlabeled mixed data, we propose an adaptive pseudo-labeling method to iteratively label mixed data and update the prototypical ind classifier. we find typical pseudo-labeling methods @xcite @xcite work poorly because the model can't produce highquality pseudo ind&ood labels, and get even worse performance. therefore, we introduce two instance-prototype margin objectives to adaptively pull together pseudo ind samples and prototypes and push apart pseudo ood samples. we aim to distinguish the confidence score distributions of ind and ood data by adjusting distances between ind or ood samples and prototypes (see section 4.2). our contributions are: (1) we propose an adaptive prototypical pseudo-labeling (app) method for few-shot ood detection. (2) we introduce a prototypical ood detection framework (protoood) to learn discriminative representations and facilitate low-resource ood detection using limited ind data, and an adaptive pseudo-labeling method to produce high-quality pseudo ood&ind labels to leverage unlabeled mixed data. (3) experiments and analysis demonstrate the effectiveness of our method for few-shot ood detection. ## conclusion in this paper, we establish a practical ood detection scenario: there are only a few labeled ind data and massive unlabeled mixed data that may belong to ind or ood. we find existing ood work can't effectively recognize ood queries using limited ind data. therefore, we propose an adaptive prototypical pseudo-labeling (app) method for few-shot ood detection. two key components are the prototypical ood detection framework (protoood) and adaptive pseudo-labeling strategy. we perform comprehensive experiments and analysis to show the effectiveness of app. we hope to provide new insight of ood detection and explore more selfsupervised learning methods for future work. 6fruh 'hqvlw\ ,1' 22' (a) @xmath0,1' 22' (b) @xmath1,1' 22' (c) @xmath2,1' 22' (d) @xmath3: change of score distribution curve of ind and ood data during pseudo-labeling with l pcl + l ood .