One-class Text Classification with Multi-modal Deep Support Vector Data Description (2021)

Abstract: This work presents multi-modal deep SVDD (mSVDD) for one-class text classification. By extending the uni-modal SVDD to a multiple modal one, we build mSVDD with multiple hyperspheres, that enable us to build a much better description for target one-class data. Additionally, the end-to-end architecture of mSVDD can jointly handle neural feature learning and one-class text learning. We also introduce a mechanism for incorporating negative supervision in the absence of real negative data, which can be beneficial to the mSVDD model. We conduct experiments on Reuters and 20 Newsgroup datasets, and the experimental results demonstrate that mSVDD outperforms uni-modal SVDD and mSVDD can get further improvements when negative supervision is incorporated.

## introduction one-class classification (occ), a special classification problem, aims to learn a model on the basis of training samples only from one class. the learned model is expected to make an accurate description of the class (so called target or normal) and then to distinguish the target from samples for negative classes during testing @xcite . the one-class classification problem has arisen in many real-world applications, including anomaly or novelty detection @xcite @xcite , bioinformatics @xcite , and especially computer vision @xcite . one-class text classification would be beneficial to the scenario where anomalous text contents (e.g., web pages, spam emails) @xcite need to be detected, and only a positive training corpus is available. one of the early work on oneclass text classification is @xcite , who implemented versions of one-class support vector machines (oc-svm) @xcite and showed good performances over the reuters dataset @xcite . oc-svm and support vector data description (svdd) @xcite are boundary-based methods @xcite . both try to describe the target data using a boundary. svdd learns an optimal hypersphere with the minimum radius to include the most target data, while oc-svm builds a hyperplane to maximally separate the data points from the origin where outlier examples lie around. reconstruction-based approaches, including au-toencoder @xcite and principal component analysis (pca) @xcite , which aim to learn a more compact representation for the description of target data. the compact representation could be a set of prototypes or subspaces obtained by optimizing a reconstruction error on the target training data. regarding the features for representing text in occ, document-to-word co-occurrence matrices or hand-crafted features have been commonly used in most of the previous work @xcite @xcite . pretrained vectors have been popular for many nlp tasks @xcite . the recent context vector data description (cvdd), proposed by @xcite , fully uses word embedding knowledge and a neural network structure to process one-class classification problems. @xcite introduced deep support vector data description (deep svdd), a fully unsupervised method for deep one-class classification for image data. deep svdd learns to extract the common factors of target training samples with a neural network to minimize the radius of a hypersphere that encloses the network representations of the data. the learned hypersphere, with a center c and a neural feature transformer φ(x), can be an end-to-end feature learning and one-class classification model. target data samples may have distinctive distributions that are located in different regions. therefore, uni-modal deep svdd with one hypersphere may not be enough to describe the target samples. in this work, we extend deep svdd to multiple modes, where each mode describes the target samples from a distinctive aspect. given our multimodal deep svdd, msvdd in short, we can create an ensemble set of hyperspheres with different centers to build a better one-class model. @xcite proposed deep multi-sphere svdd (dmsvdd), a similar but different work from ours. we will also discuss the relationship between the two and compare them in the experiments. in one-class classification, only samples from the target class are available for training, while the model needs to discriminate between the target class and other classes in testing. due to the unavailability of training samples from negative classes, it is hard for the one-class models to learn effective discrimination information, especially for msvdd with a multi-layer neural structure. in this study, we also propose an architecture for improving the discrimination ability of msvdd by incorporating negative supervision. specifically, we define two kinds of losses, contrastive and triplet, for joint training with the objective function of msvdd, which is expected to enhance the discriminative power of msvdd. in summary, the main contributions of this work are as follows. 1) we propose a general one-class neural learning framework, called msvdd, to extend the uni-modal deep svdd to end-to-end multimodal. 2) we also prove that three one-class models, deep svdd, dmsvdd, and cvdd, are all special cases of the msvdd model. 3) we propose two approaches for effectively incorporating negative supervision information to improve the performance of the proposed msvdd. ## preliminaries before describing our msvdd, we first introduce svdd @xcite and its extension, deep svdd @xcite . ## multi-modal deep svdd in this section, we present our msvdd, a method for deep one-class classification. unlike a unimodal model with a hypersphere, msvdd uses a set of hyperspheres to describe target class data and to reject samples from negative classes. figure 1 shows the general idea of msvdd with two modes. consider that we have m modes, each of which is described by a hypersphere m j with center c j and radius r j ; msvdd uses each m j to describe a distinctive aspect of the target class and then ensemble them. this ensembled deep msvdd model could provide better descriptions for the target data. as with deep svdd, given target training samples @xmath0, ..., x n }, msvdd first transforms instance x into a data point of the output feature space with φ, where φ is a deep neural network of @xmath1, ..., w l }. in contrast to deep svdd, msvdd uses m hyperspheres to include almost all of the target data with the minimum radii, i.e., 1 m m @xmath2. as in kernel svdd and deep svdd, it should also punish points lying outside the sphere, i.e., if the distance of x to the center c, φ(x; w) -c 2 2 , is greater than radius r. since we have a set of hyperspheres @xmath3, ..., m m }, one choice would be to punish x with respect to each hypersphere by adding j max 0, φ (x; w) -c j 2 2 -r 2 j to the loss function. however, the above penalty term is very hard, where one sample should satisfy each j-th constraint corresponding to m j . therefore, we loosen the constraint. given non-negative attention weight α ij for x i to each m j , the penalty term can be computed as the weighted average over m constraints. now, only one ensembled constraint is required, i.e., the sum of radii is greater than the sum distance to the center. formally, we can define our msvdd objective as follows: although the ν-property, mentioned in section 2.2, does not hold true for our multi-modal case as it is in general, it is still true when the attention weight α ij is constant for different hyperspheres. this will give us an intuition on the role of ν. proposition 1. 1 the ν-property holds if we set equal attention weight to each hypersphere: i.) ν is an upper bound for the fraction of outlier samples and ii.) ν is a lower bound for the fraction of training samples being rejected or on the optimal boundary. ## multi-modal deep svdd with negative supervision in this section, we incorporate negative supervision into the training of msvdd. the svdd-related models are usually trained with only positive samples from the target one-class, while, if negative samples (samples which should be rejected) are available, the models can be extended to train with them to improve the description @xcite . note that these samples are not necessarily required to be from "real" negative class. in our experiment, we use some external data as the pseudo-negative samples. given a set of extended training samples @xmath4, y 1 ), ..., (x n , y n )}, where the first n samples are labeled y @xmath5, denoting positive, whereas the others are labeled y @xmath6, which denotes negative samples that should be rejected by msvdd. our msvdd is represented with m hyperspheres and is formulated as @xmath7, r 1 ), ......, m m (c m , r m )}. it is required that the positive samples should be inside the m hyperspheres, while the negative samples should lie outside. given training samples composed of positive and a negative samples, we can first get their corresponding distances to each center c j . the goal of optimization should be to pull the positive samples closer the center and to push the negative ones away. formally, we define the distance between one sample x i and one center c j as there are usually two types of losses to obtain the discriminative loss. contrastive type: the contrastive-type loss directly optimizes the distance by encouraging the distance between a positive sample and a center to be smaller, while it forces the larger distance to a negative sample: triplet type: the triplet-type loss is defined for a pair of positive sample x i and negative sample x i . if we consider center c j as an anchor representative of target data, the triplet loss punishes only when d ij , the distance from x i to c j , is greater than d i j , the distance from x i to c j , with a margin τ > 0: for clarity, eqs. 10 and 11 show only the two types of losses for one hypersphere. multi-modal version can be obtained by sum operation over @xmath8, ..., m}. the triplet loss forces only positive samples to be closer to the center than negative samples, and the contrastive loss requires only keeping the distances for negative samples above the radius. these two types of objectives are easy to achieve, especially when we assume that negative samples are "not real." this can result in failing to make a full use of negative supervision. therefore, we will reformulate both l t ri d and l con d . ## conclusion in this work, we proposed msvdd, a new generic one-class text classification framework that uses multi-modal deep svdd. rather than the unimodal deep svdd, msvdd can enhance the description ability to the target one-class data with multiple hyperspheres. we also proved that this generic framework can include three variants, deep svdd, dmsvdd, and cvdd under certain conditions. in addition, in the absence of "real" negative training data, we also proposed approaches for effectively adding negative supervision to further improve the performance of msvdd. the experiments validated that the proposed msvdd provides better performance compared to uni-modal svdd. the experiments also showed the further improvements in most cases when negative supervision was used for msvdd and cvdd. for future work of this study, we will use some sampling strategies to improve the current work.