Closed Boundary Learning for Classification Tasks with the Universum Class (2023)

Abstract: The Universum class, often known as the *other* class or the*miscellaneous* class, is defined as a collection of samples that do not belong to any class of interest. It is a typical class that exists in many classification-based tasks in NLP, such as relation extraction, named entity recognition, sentiment analysis, etc. The Universum class exhibits very different properties, namely heterogeneity and lack of representativeness in training data; however, existing methods often treat the Universum class equally with the classes of interest, leading to problems such as overfitting, misclassification, and diminished model robustness. In this work, we propose a closed boundary learning method that applies closed decision boundaries to classes of interest and designates the area outside all closed boundaries in the feature space as the space of the Universum class. Specifically, we formulate closed boundaries as arbitrary shapes, propose the inter-class rule-based probability estimation for the Universum class to cater to its unique properties, and propose a boundary learning loss to adjust decision boundaries based on the balance of misclassified samples inside and outside the boundary. In adherence to the natural properties of the Universum class, our method enhances both accuracy and robustness of classification models, demonstrated by improvements on six state-of-the-art works across three different tasks. Our code is available at https://github.com/hzzhou01/Closed-Boundary-Learning.

## introduction in classification-based tasks, quite often we encounter a class named as other class, miscellaneous class, neutral class or outside (o) class. such a class is a collection of samples that do not belong to any class of interest, such as samples of no relation class in relation extraction task. we adopt the terminology in @xcite ) to * corresponding author. designate all such classes as the universum class (u). universum class exits in various classificationbased problems in nlp, such as relation extraction (re) @xcite , named entity recognition (ner) @xcite , sentiment analysis (sa) @xcite , and natural language inference (nli) @xcite . to distinguish the universum class and the rest of the classes, we call the classes of interest as target classes (t). the set of all classes (a) in the data can be expressed as @xmath0 the sample compositions of the universum class and target classes are usually very different. figure 1 (a) provides some samples of a target class (entitydestination) and the universum class (other) in relation extraction. intuitively, we can observe that the entity-destination samples adhere to an intraclass pattern: an entity goes somewhere. however, the three examples of the other relation type are vastly dissimilar and do not exhibit any intra-class pattern. in fact, the universum samples are labeled according to an inter-class pattern: they do not belong to any of the predefined target classes. we further highlight the differences between the universum class and target classes in two properties. although such heterogeneous samples are easily mapped into a compact cluster for the training set, it is problematic for the test set. this is because the inherent predictive rule of the universum class follows an unique inter-class pattern: a sample is labeled as universum if it does not belong to any target classes. this sharply contrasts with the conventional intra-class patterns seen in target classes. considering human annotation practices, an entity is labeled as location when it aligns with established patterns of location entities. in contrast, a sample is labeled as others not due to intra-class patterns specific to the others class, but because it fails to conform to the patterns of location, person, or organization. consequently, when current classification models treat the universum class and target classes in the same manner, they tend to overfit the noise in the universum class by memorizing various peculiarities of the heterogeneous samples rather than recognizing the general predictive rule. given the variations in data distributions between the test and training sets, only memorizing various peculiarities can easily lead to overfitting, causing a decline in accuracy. furthermore, this inabil-ity to discern the genuine predictive rule for the universum class can also compromise the model's robustness. the universum class is the complementary set of predefined target classes in the task. therefore, it contains all possible implicit classes, i.e., classes not explicitly defined in the task but may appear in the real world. in this case, universum samples in the training data are unable to sufficiently represent all possible patterns of the genuine distribution of the universum class. as depicted in figure 1(c ), gray samples represent universum samples in the test set that are not represented by the training data. classifiers with open boundaries are prone to misclassifying unseen samples in the test set that is not represented by the training data. additionally, we provide a quantitative comparison of the average compactness between the universum class and the target classes within the test data for the ner task @xcite , as depicted in figure 1 @xcite . notably, even though the universum class is the class with the most samples, it exhibits significantly poorer compactness in its learned representations. this empirical observation aligns with our earlier theoretical analysis. both the inability to discern the genuine inter-class predictive rule of the universum class and the lack of representativeness in the training data contribute to this compromised compactness for the universum class. experiment details are in appendix a. despite the substantial difference between the target classes and the universum class, this issue has long been neglected by the research community. the majority of works @xcite @xcite @xcite treat the universum class and target classes equally. typically, a linear layer and a softmax function are applied at the end of the model to generate open decision boundaries, which we believe are inappropriate for tasks containing the universum class. how can we account for the distinct properties of the universum class and target classes to derive better representations and classifiers? we think the key lies in conforming to the inherent properties of the universum class. in this work, we propose a closed boundary learning method for classification-based tasks with the universum class. traditional methods often employ open boundary classifiers and constrain the representations of universum samples to be distributed into a compact cluster. however, the open decision boundaries can easily misclassify universum samples, as illustrated in figure 1(d ). in addition, the restriction on compact space violates the inherent inter-class pattern of the universum class. therefore, we propose to use closed boundary classifiers as shown in figure 1(e ). we constrain the space of target classes to be closed spaces and designate the area outside all closed boundaries in the feature space as the space of the universum class. the treatment perfectly fits the nature of the universum class: a sample is marked as the universum if it does not belong to any target class during labeling. the main contributions of this work are summarized as follows: (1) we address an understudied problem in this paper. the universum class widely exists in many nlp tasks and general machine learning tasks, but hasn't received significant attention in these contexts. ## conclusion in this work, we highlight an understudied problem in classification-based tasks that the universum class is treated equally with target classes despite their significant differences. as a solution, we propose a closed boundary learning method coolu, which conforms the natural properties of the universum samples. specifically, we generate closed boundaries with arbitrary shapes, develop an inter-class rule-based strategy to estimate the probability of the universum class, and propose a boundary learning loss to adjust decision boundaries. coolu offers easy integration with most classification model, given that it operates on representations of the final layer of classification models. our method not only boosts the accuracy of sota models but also significantly enhances their robustness.