ODIST : Open World Classification via Distributionally Shifted Instances (2021)

Abstract: In this work, we address the open-world classification problem with a method called ODIST, open world classification via distributionally shifted instances. This novel and straightforward method can create out-of-domain instances from the in-domain training instances with the help of a pre-trained generative language model. Experimental results show that ODIST performs better than state-of-the-art decision boundary finding method.

## introduction in the supervised learning setting, it is generally assumed that test set data points will be organized along the same classes observed during training. this assumption, however, proves unreliable in many applications, especially in dynamic and open environments. for instance, @xcite show that an intent classifier performs rather poorly in a dialogue system when the user expresses intents unobserved in the training dialogues. in an open environment, the ideal classifier should classify incoming data to the correct existing classes that appeared in training and detect those examples that do not belong to any existing classes. such classifier is thus described as open set recognition @xcite or open world classification @xcite . the existing research to achieve this capability in natural language processing (nlp) and computer vision (cv) mainly focuses on decision boundary finding. @xcite ; @xcite ; @xcite use svm to detect the negative classified examples. @xcite introduce the concept of open-space risk in cv. @xcite ; @xcite propose a series of weibull-calibrated svm to reduce the open space risk further. recent research shows that it is also possible to use deep neural networks to capture advanced features from the data @xcite . in cv, @xcite train a multi-class classifier and take the outputs of the penultimate layer to fit weibull distribution. @xcite reject the low confidence samples with the threshold based on the probability of softmax distribution. @xcite add a temperature scaling on the softmax function to get a calibrated softmax score.in nlp, @xcite adopt the sigmoid function to learn the one-vs-all classifier and calculate the confidence threshold by fitting training data to gaussian statistics. @xcite propose to learn the adaptive decision boundary (adb). adb performs best among all the above methods on the open text classification. besides adjusting the decision boundary on the feature space learned from in-domain data, a good feature space representing both in-domain and novel out-of-domain (ood) examples is also essential for novelty detection, namely: open representation learning. we can illustrate this approach with the following nlp example: let us assume that we have only learned features for "it is red" (for cherries) and "it is yellow" (for bananas) for a fruit classification task. the problem we are trying to overcome manifests when the model is exposed to a blueberry during testing. since it has not seen the class during training, it does not possess a proper method to extract features for "blue". ideally, we want a representation learning approach that can compute such a representation instead of using the representation of "red" or "yellow". the straightforward solution is to explore some examples with "blue" during model training, although a blueberry does not exist as a class for in-domain training. however, in real-world applications, we do not foresee the ood examples that would come in the future. similarly, for cv, recent work @xcite augment distributionally shifted images by rotating/flipping the original image and pretrain an image representation space for novelty detection. inspired by the work, we propose a novel and sim-ple distributionally shifted data creation method for nlp. and then, we train a classifier on in-domain training examples and distributionally shifted examples. such a classifier can work with existing decision boundary finding methods for further open space risk reduction. related works: besides the works @xcite in open-world learning, our work is also related to data augmentation. in cv, @xcite propose a simple image pretraining method based on data augmentation. in nlp, @xcite al. (2020); lewis et al. (2020a) pretrain language model by contrastive learning on augmented data. wu et al. (2020) propose word/span deletion, word reorder, word replacement and lewis et al. (2020a) use paraphrasing method to augment examples. differently from what we explore in this paper, these works focus on similar instances instead of ood examples. in this work, we take advantage of the recent success of pretrained language models. we use the sequence-to-sequence language model bart @xcite for distributionally shifted example creation. bart can fill the masked sentences by generation. furthermore, we use the finetuned bart foot_0 on mnli @xcite for predicting the relationship between the original text and augmented examples for filtering. in our setting, we create distributionally shifted instances d @xmath0, l m+1 ), . . . , (x a k , l m+1 )} by augmenting the training set d of seen classes into a new augmented class l m+1 . we learn a model f (x) using both in-domain training examples in d and the ood examples in d a . during prediction, a data point is classified to either: one of the m seen classes from d; or l 0 either because it is classified as l m+1 (from d a ) or because all m seen classes reject it. therefore, our method is a (m + 2)-class classifier f (x) with the classes @xmath1, l 2 , . . . , l m , l 0 , l m+1 }. this section introduces the creation process of distributionally-shifted instances, model training, and testing procedure. ## conclusion in this paper, we study the open-world classification problem. differently from existing research, we propose to learn an open representation. to achieve that goal, we propose a novel and simple method to create distributionally shifted instances from the training examples. the experimental results show that the method is effective and improves over sota results on three classification datasets.