Conical Classification For Efficient One-Class Topic Determination (2021)

Abstract: As the Internet grows in size, so does the amount of text based information that exists. For many application spaces it is paramount to isolate and identify texts that relate to a particular topic. While one-class classification would be ideal for such analysis, there is a relative lack of research regarding efficient approaches with high predictive power. By noting that the range of documents we wish to identify can be represented as positive linear combinations of the Vector Space Model representing our text, we propose Conical classification, an approach that allows us to identify if a document is of a particular topic in a computationally efficient manner. We also propose Normal Exclusion, a modified version of Bi-Normal Separation that makes it more suitable within the one-class classification context. We show in our analysis that our approach not only has higher predictive power on our datasets, but is also faster to compute.

## introduction in the era of the rapid development of computers and the internet, information on a wide range of topics is pervasive. the amount of text based data is ever increasing in size, magnitude, and variety. whether it is for e-commerce @xcite , clinical diagnosis determination @xcite , or fake news detection @xcite it is vital to have efficient mechanisms for topic classification in order to effectively parse and process text based media. most of the research on topic classification uses these implementations within a binary classification or multi-class classification context @xcite @xcite @xcite . comparatively, there is a relative dearth of content variety discussing and proposing different algorithms that can identify text on a particular subject from a variety of subjects in a one-vs-all configuration, espe-cially regarding how to use vector representations of documents with low computational costs. this is unfortunate, as one class classification of text enables us to identify text of a particular form from a potentially non-exhaustible set of potential topics. in such a setting, it would be arduous to identify all potential topics we may come across and extremely time-consuming to label enough data to train a model for multi-class classification. in practice, the lack of research into one class topic determination has lead to subpar implementations for the sake of speed. one of the best examples of the ramifications of this lack of research focus is insider threat detection systems. despite insider threat detection primarily working with log and textual information, the vast majority of published work on the subject do not utilize natural language processing in their implementations @xcite @xcite @xcite . many that do simply sum over tf-idf vectors before feeding the result as a feature into detection models @xcite . we aim to tackle these issues head on. our contributions are as follows: ## related work with the intention of assessing the predictive power of one-class based text classification methods, joffe et al. has compared one-class support vector machines (ocsvm) to binary support vector machines (svm) to identify specific phenotypes in breast cancer. they found that ocsvm performed comparably to svm in balanced dataset problem spaces and outperformed svm in highly imbalanced datasets @xcite . @xcite , citing the improved performance of switching from a svm to ocsvm approach for minority class classification. they use a general framework which first uses the minority class for training in the one-class classification stage, then incorporate data from the majority class to improve the generalization performance of the constructed classifier @xcite . it turns out that ocsvms have a wide adoption rate for one-class text classification problems. additional examples include manevitz et al. using them for document classification @xcite , and seo utilizing a ocsvm to help classify images in a database using color and text content for content-based image retrieval @xcite . ensemble based methodologies have been used in practice as well. hempstalk et al. has utilized an ensemble-based approach using c4.5 decision trees with laplace smoothing to isolate real target values from those of an artificial class @xcite , validating performance on various uci datasets as well as a custom typist dataset. anderka et al. utilized a similar approach to detect text quality flaws, using a random forest as the base classifier instead @xcite . unfortunately, despite their higher memory and computation requirements, such approaches have little performance benefits compared to the ocsvm; hempstalk et al.'s results indicated that their ensemble approach was not demonstratively superior to the ocsvm approach. while not traditional one-class classification algorithms, there are a set of classifiers that co-train using a set of positive labeled data as well as a set of unlabeled data for evaluation. denis et al. has developed the positive naive bayes (pnb) classifier that works under this setting, using it successfully to classify documents in the 20-newsgroup dataset @xcite . one-class topic determination is a problem space where it is paramount to be computationally fast with low resources in order to process large numbers of documents in a short amount of time. this has traditionally excluded recent advancements in natural language processing such as embeddings from the discussion, as these take significant amounts of computation time on the modest hardware such application spaces necessitate. this has resulted in very few publications dedicated to assessing their application to the space. @xcite propose context vector data description (cvdd) @xcite , a textual anomaly detection algorithm that builds upon word embedding models to learn multiple sentence representations that capture multiple semantic contexts via the self-attention mechanism. hu et al. extended uni-modal support vector data description (svdd) to a multiple modal one, building multi-modal deep support vector data description (msvdd) with multiple hyperspheres, enabling them to build better descriptions for target one-class data @xcite . the methodology used to create the vector representations of documents can be just as important as the detection algorithm used. one main approach that has come about as a result is term frequency (tf) -inverse document frequency (idf). tf-idf is the product of two statistics: tf and idf. tf, as its name suggests, refers to the normalized frequency f of a word w j that appears in the given document d. originally coined as term specificity by jones @xcite , idf provides a measure of how much information a word provides depending on how common the word is in a given corpus. tf-idf has been successfully used for topic classification in a variety of scenarios, ranging from social media @xcite , research analysis @xcite , and news discovery @xcite . as a result, much research has been done on modifications to improve performance. martineau et. al. has proposed delta tf-idf which scales weights using word scores before classification and boasts a higher accuracy than standard tf-idf @xcite . forman studies replacing tf-idf with bi-normal separation (bns), eliminating the need for fine-tuned feature selection and performs exceptionally well on short length documents @xcite . domeniconi et. al. used a supervised variant to prevent the idf term from affecting documents within the category under analysis, so that terms frequently appearing in said category are not penalized @xcite . more recently, vector representations have been developed that use embeddings, such as bert @xcite and glove @xcite . such embeddings allow for words with similar meanings to have a similar representation which has allowed for the impressive performance of deep learning methods on complex and intricate natural language processing problem spaces. ## normal exclusion bns, which is the measure of how much the probability of occurrence of a given word in the positive class differs from the probability of occurrence of a given word in the negative class, has a couple of key benefits as a vsm metric: it is excellent at ranking words for automated feature selection filtering, it has the best performance in single metric vsm analyses, and is consistently a member of the optimal pairs of vsm metrics @xcite . thus, being able to utilize bns within a one-class context would be ideal. the formula used to calculate bns is given in equation 1. here, tpr is the true positive rate p (word|positiveclass) as determined via tp pos , where tp is the number of positive training cases containing the word and pos is the number of positive training cases. likewise, f pr refers to the false positive rate p (word|negativeclass) as determined via f p neg , where f p is the number of negative training cases containing word and neg is the number of negative training cases. f -1 is the inverse normal cumulative distribution function. is a number with small magnitude added to avoid the undefined scenario of f -1 (0); for the purposes of our analysis, we set to 0.0005, or half a count out of 1000. a naive translation to a one-class regimen would be to merely remove bns's dependence on the f pr term. thus, each word would be scaled in relation to its frequency of occurrence within our positive training set. this leads to issues, as words with a naturally high occurrence in language such as the, be, to, of, a, etc. will have predominantly high scaled values. one may try to work around these effects by removing stopwords and unrelated words from our corpus, but this can require significant hand-tuning by an expert in the field while increasing overhead computation costs. we propose an alternative solution that takes advantage of the nature of one-class classification, recalling that we wish to be able to identify text of a particular topic from any assortment of topics possible from the language. we simply need to estimate the f pr of the word with the frequency of the word in our given language. for english, there are large corpuses from which we can extract this information, for example the oxford english corpus (oec) is a dataset that presents all types of english, from blogs to newspaper articles to literary novels and even social media, sourcing from englishes from the united kingdom, the united states, ireland, australia, new zealand, the caribbean, canada, india, singapore, and south africa. for our purposes, we compiled the frequencies of the top 1 3 million words in the human language using tatman's english word count dataset @xcite and stored them within a dictionary for rapid lookup. we can safely set the frequencies of words that do not appear in our dictionary to 0, as these include words that rarely appear in standard language; such words include abaptiston, abaxile, grithbreach, gurhofite, zarnich, and zeagonite. indeed, according to oxford's compiled statistics, the combined frequency of occurence for all such words is approximately a percent of the entire lexicon of the english language, easily within the margin of error for our analysis @xcite . we coin our tweaked formula normal exclusion (ne), as it excludes, or reduces, the weightage of words that are inconsequential to determining the topic of text without requiring a negative corpus to be present. the formula for ne is shown in equation 2. here, dict[word] represents the frequency value for the given word as found within our dictionary. we will scale ne by tf for our model developing the ne-tf vsm. our representation of a word in a model will thus be determined by how frequently a word occurs in our corpus, scaled by the statistical significance of the word within the evaluated text. higher magnitude values give a strong indication that the vector is about our target topic, while lower values would lead to a lower confidence that such a conclusion is correct. ## results performance metrics can be found in ## conclusion we show that conical classification is a computationally efficient method of one-class topic classification that aims to identify whether a vector is within the conical span of the training corpus for a given topic. when combined with normal exclusion, conical classification showcases the optimal combination of predictive power, consistently great results, and fast computation times.