Open World Classification with Adaptive Negative Samples (2022)

Abstract: Open world classification is a task in natural language processing with key practical relevance and impact.Since the open or unknown category data only manifests in the inference phase, finding a model with a suitable decision boundary accommodating for the identification of known classes and discrimination of the open category is challenging.The performance of existing models is limited by the lack of effective open category data during the training stage or the lack of a good mechanism to learn appropriate decision boundaries.We propose an approach based on Adaptive Negative Samples (ANS) designed to generate effective synthetic open category samples in the training stage and without requiring any prior knowledge or external datasets.Empirically, we find a significant advantage in using auxiliary one-versus-rest binary classifiers, which effectively utilize the generated negative samples and avoid the complex threshold-seeking stage in previous works.Extensive experiments on three benchmark datasets show that ANS achieves significant improvements over state-of-the-art methods.

## introduction standard supervised classification assumes that all categories expected in the testing phase have been fully observed while training, i.e., every sample is assigned to one known category as illustrated in figure 1 @xcite . this may not always be satisfied in practical applications, such as dialogue intention classification, where new intents are expected to emerge. consequently, it is desirable to have a classifier capable of discriminating whether a given sample belongs to a known or an unknown category, e.g., the red samples in figure 1 @xcite . this problem can be understood as a (c + 1) classification problem, where c is the number of known categories and the additional category is reserved for open (unknown) samples. this scenario is also known as multi-class open set recognition @xcite . to discriminate the known from the open samples during inference, it is necessary to create a clear classification boundary that separates the known from the open category. however, the lack of open category samples during training makes this problem challenging. current research in this setting mainly focuses on two directions. the first direction mainly estimates a tighter decision boundary between known classes to allow for the possibility of the open category. existing works in this direction include the local outlier factor (lof) @xcite @xcite , deep open classification (doc) @xcite and adaptive decision boundary (adb) @xcite . lof and adb calibrate the decision boundary in the feature space while doc does it in the probability space. the second direction deals with learning a better feature representation to make the boundaryseeking problem easier. in this direction, deep-unk @xcite and seg @xcite added constraints to the feature space, scl @xcite and @xcite fine-tuned the feature extractor backbone with contrastive learning. @xcite considered introducing open samples from other datasets as negative, and @xcite generated samples with contradictory meanings using a large pretrained model. the latter two deliver large performance gains. these improvements demonstrate the significance of negative samples in determining the boundary between the known and open categories. to accomplish the same in the absence of additional datasets or knowledge, we propose a novel negative-sample constraint and employ a gradientbased method to generate pseudo open category samples. as shown in figure 1 (d), negative samples are adaptively generated for each category to closely bound each category. given the generated negative samples, we then empirically find that using auxiliary one-versus-rest binary classifiers can better capture the boundary between the known and the open category, relative to a (c + 1)-way classifier @xcite , where all the open categories, possibly distributed in multiple modes or arbitrarily scattered over the feature space, are categorized into one class. specifically, we first learn a c-category classifier on known category data. then for each known category, we learn an auxiliary binary classifier, treating this category as positive and others as negative. during inference, one sample is recognized as open if all the binary classifiers predict it as negative, thus not belonging to any known category our main contributions are summarized below: ## methodology problem definition suppose we are given a training dataset @xmath0, y 1 ), (x 2 , y 2 ), . . . , (x n , y n )} consisting of n examples, where x i is an input text sequence and y i its corresponding label, which belongs to a predefined set @xmath1, l 2 , . . . , l c } with c categories, thus y @xmath2, ∀@xmath3, where [n ] := [1, . . . , n ]. in this paper, we use [•] to represent index sequence. in an open world classification setting, the goal is to learn a model which categorizes a test instance to either one of the predefined c categories or as an open category. in practice, the open category is denoted as a unique category l 0 . ## experiments we conduct experiments on three datasets: banking, clinc and stackoverflow. details and examples of the datasets are found in appendix a.4. ## conclusions we have introduced ans, a pseudo open category sample generation approach for open-world classification problems. the generation process is free from extra datasets or prior knowledge. the synthesized samples are effective for improving existing methods. combined with one-versus-rest framework, significant improvements are observed on three benchmarks. the gradient-based negative sample generation proposed can be applied to other nlp tasks such as out-of-scope discovery, which we leave as future work.