Using Locally Learnt Word Representations for better Textual Anomaly Detection (2024)

Abstract: The literature on general purpose textual Anomaly Detection is quite sparse, as most textual anomaly detection methods are implemented as out of domain detection in the context of pre-established classification tasks. Notably, in a field where pre-trained representations and models are of common use, the impact of the pre-training data on a task that lacks supervision has not been studied. In this paper, we use the simple setting of k-classes out anomaly detection and search for the best pairing of representation and classifier. We show that well-chosen embeddings allow a simple anomaly detection baseline such as OC-SVM to achieve similar results and even outperform deep state-of-the-art models.

## introduction anomaly detection (ad) consists in detecting observations that deviate from normality: what is normal is defined by available data and assumed to be bounded @xcite , while anomalies (which can be called outliers, or novelty depending on the application) are outside this bound. the most obvious hurdle with ad is that it is usually not possible to characterize anomalies: models are mostly not designed to target a specific type of outlier, and the assumptions made on data are rarely stated. in this context, supervision usually comes from normal data. however, most nlp models employ pre-trained representations: the impact that this kind of prior knowledge may have on ad is difficult to appreciate, and overlooked. a first attempt to characterize outliers in natural language data was made by @xcite , classifying them as coming from either background shifts (coming from a shift in domain) or semantic shifts (coming from a shift in content), and bringing insights on which detection method might better work on each. @xcite showed that background shifts are well detected by language models, which are able to estimate the density of normal data; there is furthermore an abundant literature on adapting pre-trained language model to new normal data @xcite . we hence focus on semantic shifts, which are shown to be well detected by calibration methods. however, this assumes access to a classification model trained on relevant categories; we however prefer to not assume access to any labels, and adopt a simple but convenient way of evaluating ad: repurposing classification datasets by declaring one class to be normal and the others as anomalies, in what is called k-classes-out. in this setting, existing approaches are fewer. some are inspired by topic modeling: they learn topic models optimized to reconstruct normal data well, aiming to detect anomalies by failing to accurately reconstruct them. for example, cvdd @xcite learns a limited number of topic-centroid vectors by applying attention upon pre-trained word-embeddings. a second direction is to train deep self-supervised models to recognize anomalies that are simulated, for example through random perturbation of data, as for date @xcite . while both these models were previously compared on common datasets, cvdd uses pre-trained representations and date is only trained on the data available for the ad task. in this paper, our goal is to investigate the impact of the pre-training data on anomaly detection performance in the k-classes-out setting; we experiment with static and contextual representations, off-the-shelf or obtained strictly on the ad training data, on three datasets. our results show that the most simple configuration -a simple non-neural classification model, when equipped with textual representations obtained from the ad training data, can beat state-of-the-art models on our ad task. ## conclusion in this paper, we implement a fair comparison between existing textual anomaly detection methods in a k-classes-out setting and show that training the models on only the data available for the ad task can lead to better results. this allows methods regarded as baselines, such as oc-svm models, to achieve impressive results, challenging state-ofthe-art models based on deep neural architectures, with only the data available at hand. we intend to extend this line of work towards more challenging textual ad tasks. we also believe our results are indicative of the potential of model adaptation methods for semantic anomaly detection, which is a direction that has only been seldom explored @xcite . in the future, we also intend to extend our investigation to larger, more recent language models for obtaining representations.