Unsupervised Anomaly Detection in Multi-Topic Short-Text Corpora (2023)

Abstract: Unsupervised anomaly detection seeks to identify deviant data samples in a dataset without using labels and constitutes a challenging task, particularly when the majority class is heterogeneous. This paper addresses this topic for textual data and aims to determine whether a text sample is an outlier within a potentially multi-topic corpus. To this end, it is crucial to grasp the semantic aspects of words, particularly when dealing with short texts, since it is difficult to syntactically discriminate data samples based only on a few words. Thereby we make use of word embeddings to represent each sample by a dense vector, efficiently capturing the underlying semantics. Then, we rely on the Mixture Model approach to detect which samples deviate the most from the underlying distributions of the corpus. Experiments carried out on real datasets show the effectiveness of the proposed approach in comparison to state-of-the-art techniques both in terms of performance and time efficiency, especially when more than one topic is present in the corpus.

## introduction anomaly detection (ad) is a task that can address various objectives such as mining frauds @xcite , diseases @xcite and intrusions @xcite . ad takes several forms: supervised, unsupervised or semi-supervised. unsupervised ad implies that no prior information about the dataset is provided. in this case, the solution usually consists of identifying samples that deviate in a certain way from the others among the same dataset; anomalies being, by definition, rare phenomena. particularly, anomalies in a textual dataset can be defined as samples having an atypical vocabulary (lexical anomaly) or a deviating global meaning (semantic anomaly). identifying abnormalities in textual data can be very useful in many industrial use-cases. a good example is the detection of non-eligible and/or fraudulent course contents in the public french plateform moncompteformation foot_0 where millions of course sessions are available with no possibility of controlling training organizations in a supervised fashion (using labeled data). hence, to assess the effectiveness of our approach, we rely on an external labeled dataset that closely relates to course contents and that is dedicated to course certifications. the dataset is described in appendix a. in addition to the difficulty of mining anomalies in short-text corpora of varying sizes, we also have an important computational cost constraint that is also addressed by the proposed solution. capturing the semantics of a given text is usually performed using word embeddings, which consist in representing a word or a piece of text by a fixed-size vector, supposed to detain its meaning. several word embedding techniques are available such as word2vec @xcite , glove @xcite , fasttext @xcite , bert @xcite , distil-bert @xcite , etc. each of the mentioned works provides ready-to-use models that are pre-trained on very large corpora and intended to be general for a given language and suitable for several nlp downstream tasks. indeed, relying on such pre-trained models has proved efficient in several tasks @xcite and is particularly useful when dealing with small corpora @xcite . if we consider the small corpus given in depending on the data type and the assumptions that can be made, the definition of an outlier may differ, and the choice of the model is crucial, especially in an unsupervised context @xcite . in this paper, we propose a probabilistic ad approach based on mixture models, that effectively identifies the most deviating samples in short-text datasets, even in the case where several topics are present in the inlier class. we also show the effectiveness of using the knowledge learned by word embedding models in capturing the underlying semantics of short texts and efficiently identifying outliers. the main contributions of this paper are: ## related work anomaly detection is an active research area, and a large number of approaches are proposed in several application domains. specifically, our work relates to unsupervised ad for text and clustering-based ad. unsupervised ad is gaining more and more interest in research due to the constant growth of data volumes while labeling data samples is not getting any cheaper. one of the most important family of methods contains reconstruction-based approaches that assume that a well-generalizing model would struggle at compressing rare anomalous samples. this kind of approaches include linear models such as robust pca @xcite and deep autoencoder models based on convolutional networks @xcite , recurrent networks @xcite , etc. ## gaussian mixture models given a corpus d of n short texts (d 1 , . . . , d n ), we represent each sample by a fixed size vector, thus obtaining a matrix @xmath0, . . . , x n ) of size n × m. to tackle the ad problem, we postulate that the samples follow a mixture of distributions, from which the anomalous samples deviate. admittedly, the family of t-distributions provides a heavy-tailed alternative to the gaussian family for anomaly detection. however, as pointed out by @xcite , although useful from modeling perspective, the practical use of multivariate t-distribution is often limited by the difficulty in parameter estimation, particularly so for high dimensional data. note that, in our proposal, the consideration of the time consumption is important. therefore, considering a t mixture model leads to estimate a supplementary parameter (in addition to the estimation of vector means and covariance matrices) that is the degree of freedom of each component. moreover, since we suggest to consider an ensemble method allowing to combine results by varying the number of components (cf. section 3.1), we would therefore increase yet the computation time for estimation of the parameters. however, gaussian mixture model (gmm)-based approaches, more parsimonious than t-mixture model, have shown their effectiveness in anomaly detection, such as dagmm @xcite . for these reasons we retain gmm to address our purpose. in a finite gmm, the data (x 1 , . . . , x n ) are taken to constitute a sample of n independent instances of a random variable x in r m . density can be expressed as: the maximization is commonly performed by expectation-maximization @xcite ; an iterative algorithm based on the maximization of the conditional expectation of the complete data log-likelihood given θ ′ : in real terms, the algorithm is broken down into two steps (e-m steps) and the unknown parameters of θ are updated thanks to the previously computed probabilities. for each component k, we have the procedure used to identify anomalies is described in algorithm 1. it takes as input a set of short texts and returns the ones that are the most likely to constitute an anomaly. the maximum density as normality score denotes the confidence algorithm 1: ad with gmm input: @xmath1, . . . d n }, g the number of components, m an embedding module, α the desired number of output samples; in the evaluation section, we evaluate the ad performance with every possible value of α using the auroc score. ## experimental study to assess the effectiveness of our approach and compare it to state-of-the-art, we conduct a set of experiments on real datasets and discuss the results in this section. ## conclusion this paper addresses semantic anomaly detection in short texts with an additional constraint in time efficiency. in addition to the classical framework where one class is used as the inlier class, we also consider the scenario where several underlying subgroups are present in the normal class. we see anomaly detection as a probabilistic clustering problem, in which we learn a gaussian mixture model and consider the low posterior probability samples as belonging to none of the modeled clusters and more likely to constitute outliers. this uncertainty score proved effective with different numbers of subgroups. in the multi-class setting, we propose gmme, an ensemble approach that improves the performance of gmm when several topics are present in the inlier class. the two approaches outperform state-of-the-art anomaly detection techniques in both scenarios, with an impressively low computation time. in our proposal, we rely on the gaussian mixture model for its flexibility. this choice is motivated by the presence of the proportions π k of each cluster and the spectral decomposition of the covariance matrix σ k taking into account the volume, shape, and orientation of each cluster (as depicted in figure 1 ). the characteristics of the clusters should not be overlooked when tackling the problem of anomaly detection through a clustering approach. furthermore, note that our approach can be extended to latent block models, devoted to coclustering, which may constitute an interesting and promising future path of research.