DATE : Detecting Anomalies in Text via Self-Supervision of Transformers (2021)

Abstract: Leveraging deep learning models for Anomaly Detection (AD) has seen widespread use in recent years due to superior performances over traditional methods. Recent deep methods for anomalies in images learn better features of normality in an end-to-end self-supervised setting. These methods train a model to discriminate between different transformations applied to visual data and then use the output to compute an anomaly score. We use this approach for AD in text, by introducing a novel pretext task on text sequences. We learn our DATE model end-to-end, enforcing two independent and complementary self-supervision signals, one at the token-level and one at the sequence-level. Under this new task formulation, we show strong quantitative and qualitative results on the 20Newsgroups and AG News datasets. In the semi-supervised setting, we outperform state-of-the-art results by +13.5% and +6.9%, respectively (AUROC). In the unsupervised configuration, DATE surpasses all other methods even when 10% of its training data is contaminated with outliers (compared with 0% for the others).

## introduction anomaly detection (ad) can be intuitively defined as the task of identifying examples that deviate from the other ones to a degree that arouses suspicion @xcite . research into ad spans several decades @xcite and has proved fruitful in several real-world problems, such as intrusion detection systems @xcite , credit card fraud detection @xcite , and manufacturing @xcite . our date method is applicable in the semisupervised ad setting, in which we only train on clean, labeled normal examples, as well as the unsupervised ad setting, where both unlabeled normal and abnormal data are used for training. typical deep learning approaches in ad involve learning features of normality using autoencoders @xcite @xcite or generative adversarial networks @xcite . under this setup, anomalous examples lead to a higher reconstruction error or differ significantly compared with generated samples. recent deep ad methods for images learn more effective features of visual normality through selfsupervision, by training a deep neural network to discriminate between different transformations applied to the input images @xcite . an anomaly score is then computed by aggregating model predictions over several transformed input samples. we adapt those self-supervised classification methods for ad from vision to learn anomaly scores indicative of text normality. electra @xcite proposes an efficient language representation learner, which solves the replaced token detection (rtd) task. here the input tokens are plausibly corrupted with a bertbased @xcite generator, and then a discriminator predicts for each token if it is real or replaced by the generator. in a similar manner, we introduce a complementary sequence-level pretext task called replaced mask detection (rmd), where we enforce the discriminator to predict the predefined mask pattern used when choosing what tokens to replace. for instance, given the input text 'they were ready to go' and the mask pattern [0, 0, 1, 0, 1], the corrupted text could be 'they were prepared to advance'. the rmd multi-class classification task asks which mask pattern (out of k such patterns) was used to corrupt the original text, based on the corrupted text. our generatordiscriminator model solves both the rmd and the rtd task and then computes the anomaly scores based on the output probabilities, as visually explained in detail fig. 1 2 . we notably simplify the computation of the secondly, the discriminator receives supervision signals from two tasks: rmd (which mask pattern was applied to the input sequence) and rtd (the per-token status: original or replaced). pseudo label (pl) anomaly score @xcite by removing the dependency on running over multiple transformations and enabling it to work with token-level predictions. this significantly speeds up the pl score evaluation. to our knowledge, date is the first end-to-end deep ad method on text that uses self-supervised classification models to produce normality scores. our contributions are summarized below: our work relates to self-supervision for language representation as well as self-supervision for learning features of normality in ad. ## our approach our method is called date for 'detecting anomalies in text using electra'. we propose an endto-end ad approach for the discrete text domain that combines our novel self-supervised task (replaced mask detection), a powerful representation learner for text (electra), and an ad score tailored for sequential data. we present next the components of our model and a visual representation for the training and testing pipeline in fig. 1 2 . ## experimental analysis in this section, we detail the empirical validation of our method by presenting: the semi-supervised and unsupervised experimental setup, a comprehensive ablation study on date, and the comparison with state-of-the-art on the semi-supervised and unsupervised ad tasks. date does not use any form of pre-training or knowledge transfer (from other datasets or tasks), learning all the embeddings from scratch. using pre-training would introduce unwanted prior knowledge about the outliers, making our model considering them known (normal). ## conclusion we propose date, a model for tackling anomaly detection in text, and formulate an innovative selfsupervised task, based on masking parts of the initial input and predicting which mask pattern was used. after masking, a generator reconstructs the initially masked tokens and the discriminator predicts which mask was used. we optimize a loss composed of both token and sequence-level parts, taking advantage of powerful supervision, coming from two independent pathways, which stabilizes learning and improves ad performance. for computing the anomaly score, we alleviate the burden of aggregating predictions from multiple transformations by introducing an efficient variant of the pseudo label score, which is applied per token, only on the original input. we show that this score separates very well the abnormal entries from normal ones, leading date to outperform state-of-theart results on all ad splits from 20newsgroups and ag news datasets, by a large margin, both in the semi-supervised and unsupervised ad settings.