A NOMALY T RANSFORMER : T IME S ERIES A NOMALY D ETECTION WITH A SSOCIATION D ISCREPANCY
A BSTRACT
Unsupervised detection of anomaly points in time series is a challenging problem, which requires the model to derive a distinguishable criterion.
Previous methods tackle the problem mainly through learning pointwise representation or pairwise association, however, neither is sufﬁcient to reason about the intricate dynamics.
Recently, Transformers have shown great power in uniﬁed modeling of pointwise representation and pairwise association, and we ﬁnd that the self-attention weight distribution of each time point can embody rich association with the whole series.
Our key observation is that due to the rarity of anomalies, it is extremely difﬁcult to build nontrivial associations from abnormal points to the whole series, thereby, the anomalies’ associations shall mainly concentrate on their adjacent time points.
This adjacent-concentration bias implies an association-based criterion inherently distinguishable between normal and abnormal points, which we highlight through the Association Discrepancy .
Technically, we propose the Anomaly Transformer with a new Anomaly-Attention mechanism to compute the association discrepancy.
A minimax strategy is devised to amplify the normal-abnormal distinguishability of the association discrepancy.
The Anomaly Transformer achieves state-of-theart results on six unsupervised time series anomaly detection benchmarks of three applications: service monitoring, space & earth exploration, and water treatment.
I NTRODUCTION
Real-world systems always work in a continuous way, which can generate several successive measurements monitored by multi-sensors, such as industrial equipment, space probe, etc.
Discovering the malfunctions from large-scale system monitoring data can be reduced to detecting the abnormal time points from time series, which is quite meaningful for ensuring security and avoiding ﬁnancial loss.
But anomalies are usually rare and hidden by vast normal points, making the data labeling hard and expensive.
Thus, we focus on time series anomaly detection under the unsupervised setting.
Unsupervised time series anomaly detection is extremely challenging in practice.
The model should learn informative representations from complex temporal dynamics through unsupervised tasks.
Still, it should also derive a distinguishable criterion that can detect the rare anomalies from plenty of normal time points.
Various classic anomaly detection methods have provided many unsupervised paradigms, such as the density-estimation methods proposed in local outlier factor (LOF, Breunig et al.
(2000)), clustering-based methods presented in one-class SVM (OC-SVM, Sch¨olkopf et al.
and SVDD (Tax & Duin, 2004).
These classic methods do not consider the temporal information and are difﬁcult to generalize to unseen real scenarios.
Beneﬁting from the representation learning capability of neural networks, recent deep models (Su et al., 2019;
Shen et al., 2020; Li et al., 2021) have achieved superior performance.
A major category of methods focus on learning pointwise representations through well-designed recurrent networks and are self-supervised by the reconstruction or autoregressive task.
Here, a natural and practical anomaly criterion is the pointwise reconstruction or prediction error.
However, due to the rarity of anomalies, the pointwise representation is less informative for complex temporal patterns and can be dominated by normal time points, making anomalies less distinguishable.
Also, the reconstruction or prediction error is calculated point by point, which cannot provide a comprehensive description of the temporal context.
Another major category of methods detect anomalies based on explicit association modeling.
The vector autoregression and state space models fall into this category.
The graph was also used to capture the association explicitly, through representing time series with different time points as vertices and detecting anomalies by random walk (Cheng et al., 2008; 2009).
In general, it is hard for these classic methods to learn informative representations and model ﬁne-grained associations.
Recently, graph neural network (GNN) has been applied to learn the dynamic graph among multiple variables in multivariate time series (Zhao et al., 2020; Deng & Hooi, 2021).
While being more expressive, the learned graph is still limited to a single time point, which is insufﬁcient for complex temporal patterns.
Besides, subsequence-based methods detect anomalies by calculating the similarity among subsequences (Boniol & Palpanas, 2020).
While exploring wider temporal context, these methods cannot capture the ﬁne-grained temporal association between each time point and the whole series.
In this paper, we adapt Transfomers (Vaswani et al., 2017) to time series anomaly detection in the unsupervised regime.
Transformers have achieved great progress in various areas, including natural language processing (Brown et al., 2020), machine vision (Liu et al., 2021) and time series (Zhou et al., 2021).
This success is attributed to its great power in uniﬁed modeling of global representation and long-range relation.
Applying Transformers to time series, we ﬁnd that the temporal association of each time point can be obtained from the self-attention map, which presents as a distribution of its association weights to all the time points along the temporal dimension.
The association distribution of each time point can provide a more informative description for the temporal context, indicating dynamic patterns, such as the period or trend of time series.
We name the above association distribution as the series-association , which can be discovered from the raw series by Transformers
Further, we observe that due to the rarity of anomalies and the dominance of normal patterns, it is harder for anomalies to build strong associations with the whole series.
The associations of anomalies shall concentrate on the adjacent time points that are more likely to contain similar abnormal patterns due to the continuity.
Such an adjacent-concentration inductive bias is referred to as the prior-association .
In contrast, the dominating normal time points can discover informative associations with the whole series, not limiting to the adjacent area.
Based on this observation, we try to utilize the inherent normal-abnormal distinguishability of the association distribution.
This leads to a new anomaly criterion for each time point, quantiﬁed by the distance between each time point’s prior-association and its series-association, named as Association Discrepancy .
As aforementioned, because the associations of anomalies are more likely to be adjacent-concentrating, anomalies will present a smaller association discrepancy than normal time points.
Go beyond previous methods, we introduce Transformers to the unsupervised time series anomaly detection and propose the Anomaly Transformer for association learning.
To compute the Association Discrepancy, we renovate the self-attention mechanism to the Anomaly-Attention , which contains a two-branch structure to model the prior-association and series-association of each time point respectively.
The prior-association employs the learnable Gaussian kernel to present the adjacentconcentration inductive bias of each time point, while the series-association corresponds to the selfattention weights learned from raw series.
Besides, a minimax strategy is applied between the two branches, which can amplify the normal-abnormal distinguishability of the Association Discrepancy and further derive a new association-based criterion.
Anomaly Transformer achieves strong results on six benchmarks, covering three real applications.
The contributions are summarized as follows:
• Based on the key observation of Association Discrepancy, we propose the Anomaly Transformer with an Anomaly-Attention mechanism, which can model the prior-association and series-association simultaneously to embody the Association Discrepancy.
• We propose a minimax strategy to amplify the normal-abnormal distinguishability of the Association Discrepancy and further derive a new association-based detection criterion.
• Anomaly Transformer achieves the state-of-the-art anomaly detection results on six benchmarks for three real applications.
Extensive ablations and insightful case studies are given.
R ELATED W ORK 2. U NSUPERVISED T IME S ERIES A NOMALY D ETECTION
As a vital real-world problem, unsupervised time series anomaly detection has been widely explored.
Categorizing by anomaly determination criterion, the paradigms roughly include the densityestimation, clustering-based, reconstruction-based and autoregression-based methods.
As for the density-estimation methods, the classic methods local outlier factor (LOF, Breunig et al.
and connectivity outlier factor (COF, Tang et al.
(2002)) calculates the local density and local connectivity for outlier determination respectively.
DAGMM (Zong et al., 2018) and MPPCACD (Yairi et al., 2017) integrate the Gaussian Mixture Model to estimate the density of representations.
In clustering-based methods, the anomaly score is always formalized as the distance to cluster center.
SVDD (Tax & Duin, 2004) and Deep SVDD (Ruff et al., 2018) gather the representations from normal data to a compact cluster.
THOC
(Shen et al., 2020) fuses the multi-scale temporal features from intermediate layers by a hierarchical clustering mechanism and detects the anomalies by the multi-layer distances.
ITAD (Shin et al., 2020) conducts the clustering on decomposed tensors.
The reconstruction-based models attempt to detect the anomalies by the reconstruction error.
Park et al.
(2018) presented the LSTM-VAE model that employed the LSTM backbone for temporal modeling and the Variational AutoEncoder (VAE) for reconstruction.
OmniAnomaly proposed by
Su et al.
(2019) further extends the LSTM-VAE model with a normalizing ﬂow and uses the reconstruction probabilities for detection.
InterFusion from Li et al.
(2021) renovates the backbone to a hierarchical VAE to model the inter- and intra- dependency among multiple series simultaneously.
GANs (Goodfellow et al., 2014) are also used for reconstruction-based anomaly detection (Schlegl et al., 2019;
Li et al., 2019a;
Zhou et al., 2019) and perform as an adversarial regularization.
The autoregression-based models detect the anomalies by the prediction error.
VAR extends ARIMA (Anderson & Kendall, 1976) and predicts the future based on the lag-dependent covariance.
The autoregressive model can also be replaced by LSTMs (Hundman et al., 2018; Tariq et al., 2019).
This paper is characterized by a new association-based criterion.
Different from the random walk and subsequence-based methods (Cheng et al., 2008; Boniol & Palpanas, 2020), our criterion is embodied by a co-design of the temporal models for learning more informative time-point associations.
2. T RANSFORMERS FOR T IME S ERIES A NALYSIS
Recently, Transformers (Vaswani et al., 2017) have shown great power in sequential data processing, such as natural language processing (Devlin et al., 2019; Brown et al., 2020), audio processing (Huang et al., 2019) and computer vision (Dosovitskiy et al., 2021; Liu et al., 2021).
For time series analysis, beneﬁting from the advantage of the self-attention mechanism, Transformers are used to discover the reliable long-range temporal dependencies (Kitaev et al., 2020;
Li et al., 2019b; Zhou et al., 2021; Wu et al., 2021).
Especially for time series anomaly detection, GTA proposed by Chen et al.
(2021) employs the graph structure to learn the relationship among multiple IoT sensors, as well as the Transformer for temporal modeling and the reconstruction criterion for anomaly detection.
Unlike the previous usage of Transformers, Anomaly Transformer renovates the self-attention mechanism to the Anomaly-Attention based on the key observation of association discrepancy.
M ETHOD
Suppose monitoring a successive system of d measurements and recording the equally spaced observations over time.
The observed time series X is denoted by a set of time points { x 1 , x 2 , · · · , x N } , where x t ∈ R d represents the observation of time t .
The unsupervised time series anomaly detection problem is to determine whether x t is anomalous or not without labels.
As aforementioned, we highlight the key to unsupervised time series anomaly detection as learning informative representations and ﬁnding distinguishable criterion.
We propose the Anomaly Transformer to discover more informative associations and tackle this problem by learning the Association Discrepancy , which is inherently normal-abnormal distinguishable.
Technically, we propose the
Anomaly-Attention to embody the prior-association and series-associations, along with a minimax optimization strategy to obtain a more distinguishable association discrepancy.
Co-designed with the architecture, we derive an association-based criterion based on the learned association discrepancy.
3. A NOMALY T RANSFORMER
Given the limitation of Transformers (Vaswani et al., 2017) for anomaly detection, we renovate the vanilla architecture to the Anomaly Transformer (Figure 1) with an Anomaly-Attention mechanism.
Overall Architecture Anomaly Transformer is characterized by stacking the Anomaly-Attention blocks and feed-forward layers alternately.
This stacking structure is conducive to learning underlying associations from deep multi-level features.
Suppose the model contains L layers with length- N
[Table]
N (1 , 21 ) N (2 , 22 ) N ( N, 2 N )
[Table]
N (1 , 21 ) N (2 , 2 2 ) N ( N, 2 N )
[Table]
N (1 , 21 ) N (2 , 22 ) N ( N, 2 N )
[Table]
[Table] [Table]
a
[Table]
S
Series
[Table]
Association
[Table] [Table]
X
[Table]
[Table] [Table]
N (1 , 21 ) N (2 , 22 ) N ( N, 2 N )
[Table]
N (1 , 21 ) N (2 , 22 ) N ( N, 2 N )
[Table]
N (1 , 21 ) N (2 , 22 ) N ( N, 2 N ) …
Sca l e grad
[Table] [Table]
作业要求
[Table]
(cid:56) 作业要求
[Table]
本次作业为单人作业（需独立完成）。
[Table] [Table]
[Figure]
input time series X ∈ R N × d .
The overall equations of the l -th layer are formalized as:
[Equation]
where X l ∈ R N × d model , l ∈ { 1 , · · · , L } denotes the output of the l -th layer with d model channels.
The initial input
[Equation]
Embedding( X ) represents the embedded raw series.
Z l ∈ R N × d model is the l -th layer’s hidden representation.
Anomaly - Attention( · ) is to compute the association discrepancy.
Anomaly-Attention Note that the single-branch self-attention mechanism (Vaswani et al., 2017) cannot model the prior-association and series-association simultaneously.
We propose the AnomalyAttention with a two-branch structure (Figure 1).
For the prior-association , we adopt a learnable
Gaussian kernel to calculate the prior with respect to the relative temporal distance.
Beneﬁting from the unimodal property of the Gaussian kernel, this design can pay more attention to the adjacent horizon constitutionally.
We also use a learnable scale parameter σ for the Gaussian kernel, making the prior-associations adapt to the various time series patterns, such as different lengths of anomaly segments.
The series-association branch is to learn the associations from raw series, which can ﬁnd the most effective associations adaptively.
Note that these two forms maintain the temporal dependencies of each time point, which are more informative than point-wise representation.
They also reﬂect the adjacent-concentration prior and the learned real associations respectively, whose discrepancy shall be normal-abnormal distinguishable.
The Anomaly-Attention in the l -th layer is:
Initialization
[Equation]
Prior-Association:
[Equation]
Series-
[Equation] [Equation]
where Q , K , V ∈ R N × d model , σ ∈ R N × 1 represent the query, key, value of self-attention and the learned scale respectively.
W l Q , W l K , W l V ∈ R d model × d model , W lσ ∈ R d model × 1 represent the parameter matrices for Q , K , V , σ in the l -th layer respectively.
Prior-association P l ∈ R N × N is generated based on the learned scale σ ∈ R N × 1 and the i -th element σ i corresponds to the i -th time point.
Concretely, for the i -th time point, its association weight to the j -th point is calculate by the Gaussian kernel G ( | j − i | ; σ i ) =
1 √ 2 πσ i exp( − | j − i | 2 2 σ 2 i ) w.r.t.
the distance | j − i | .
Further, we use Rescale( · ) to transform the association weights to discrete distributions P l by dividing the row sum.
S l ∈ R N × N denotes the series-associations.
Softmax( · ) normalizes the attention map along the last dimension.
AssociationDiscrepancy b.
AssociationDiscrepancy d. Min KL Max KL a.
Min KL Max KL c. Association Space Adjacet-seriesAssociation
Global-seriesAssociation Gaussian Family Reconstruction Loss Constrained Space
Normal Time Points Abnormal Time Points Maximize Phase Minimize Phase AssociationDiscrepancy
Association Space Adjacet-series Association
Global-series Association Gaussian Family
Reconstruction Loss Constrained Space Adjacet-seriesAssociation Global-seriesAssociation Constrained by Reconstruction Global-seriesAssociation Adjacet-seriesAssociation Within the Gaussian Family Time series Time series Optimization Direction
Adjacet-seriesAssociation
Global-seriesAssociation Constrained by Reconstruction
Global-seriesAssociation
Adjacet-seriesAssociation
Time series
Time series Global-seriesAssociation
Adjacet-seriesAssociation
Maximize Phase Minimize Phase
AssociationDiscrepancy Prior-Association
Series-Association Gaussian Family
Reconstruction Loss Constrained Space Prior-Association Series-Association Constrained by Reconstruction Series-Association Prior-Association
Within the Gaussian Family Time series
Time series OptimizationDirection AssociationDiscrepancy
[Figure]
In that denote the query, key and value of the m -th head respectively.
The block concatenates the outputs
[Equation]
[Table]
and gets
[Table]
[Equation]
Association Discrepancy We formalize the Association Discrepancy as the symmetrized KL divergence between prior- and series- associations, which represents the information gain between these two distributions (Neal, 2007).
We average the association discrepancy from multiple layers to combine the associations from multi-level features into a more informative measure as:
[Equation]
where KL( ·(cid:107)· ) is the KL divergence computed between two discrete distributions corresponding to every row of P l and S l .
AssDis( P , S ; X
) ∈ R N × 1 is the point-wise association discrepancy of X with respect to prior-association
P and series-association S from multiple layers.
The i -th element of results corresponds to the i -th time point of
X .
From previous observation, anomalies will present smaller AssDis( P , S ; X ) than normal time points, which makes AssDis inherently distinguishable.
3. M INIMAX A SSOCIATION L EARNING
As an unsupervised task, we employ the reconstruction loss for optimizing our model.
The reconstruction loss will guide the series-association to ﬁnd the most informative associations.
To further amplify the difference between normal and abnormal time points, we also use an additional loss to enlarge the association discrepancy.
Due to the unimodal property of the prior-association, the discrepancy loss will guide the series-association to pay more attention to the non-adjacent area, which makes the reconstruction of anomalies harder and makes anomalies more identiﬁable.
The loss function for input series X ∈ R N × d is formalized as:
[Equation]
where (cid:98) X ∈ R N ×
d denotes the reconstruction of X .
(cid:107)·(cid:107) F , (cid:107)·(cid:107) k indicate the Frobenius and k -norm.
λ is to trade off the loss terms.
When λ > 0 , the optimization is to enlarge the association discrepancy.
A minimax strategy is proposed to make the association discrepancy more distinguishable.
Minimax Strategy Note that directly maximizing the association discrepancy will extremely reduce the scale parameter of the Gaussian kernel (Neal, 2007), making the prior-association meaningless.
Towards a better control of association learning, we propose a minimax strategy (Figure 2).
Concretely, for the minimize phase , we drive the prior-association P l to approximate the seriesassociation S l that is learned from raw series.
This process will make the prior-association adapt to various temporal patterns.
For the maximize phase , we optimize the series-association to enlarge the association discrepancy.
This process forces the series-association to pay more attention to the non-adjacent horizon.
Thus, integrating the reconstruction loss, the loss functions of two phases are:
[Equation]
where λ > 0 and ∗ detach means to stop the gradient backpropagation of the association (Figure 1).
As P approximates S detach in the minimize phase, the maximize phase will conduct a stronger constraint to the series-association, forcing the time points to pay more attention to the non-adjacent area.
Under the reconstruction loss, this is much harder for anomalies to achieve than normal time points, thereby amplifying the normal-abnormal distinguishability of the association discrepancy.
Association-based Anomaly Criterion We incorporate the normalized association discrepancy to the reconstruction criterion, which will take the beneﬁts of both temporal representation and the distinguishable association discrepancy.
The ﬁnal anomaly score of X ∈ R N × d is shown as follows:
[Equation]
where (cid:12) is the element-wise multiplication.
AnomalyScore( X ) ∈ R N × 1 denotes the point-wise anomaly criterion of X .
Towards a better reconstruction, anomalies usually decrease the association discrepancy, which will still derive a higher anomaly score.
Thus, this design can make the reconstruction error and the association discrepancy collaborate to improve detection performance.
E XPERIMENTS
We extensively evaluate Anomaly Transformer on six benchmarks for three practical applications.
Datasets Here is a description of the six experiment datasets: (1) SMD (Server Machine Dataset,
Su et al.
(2019)) is a 5-week-long dataset that is collected from a large Internet company with 38 dimensions.
(2) PSM (Pooled Server Metrics, Abdulaal et al.
(2021)) is collected internally from multiple application server nodes at eBay with 26 dimensions.
(3) Both MSL (Mars Science Laboratory rover) and SMAP (Soil Moisture Active Passive satellite) are public datasets from NASA (Hundman et al., 2018) with 55 and 25 dimensions respectively, which contain the telemetry anomaly data derived from the Incident Surprise Anomaly (ISA) reports of spacecraft monitoring systems.
(4) SWaT (Secure Water Treatment, Mathur & Tippenhauer (2016)) is obtained from 51 sensors of the critical infrastructure system under continuous operations.
(5) NeurIPS-TS (NeurIPS 2021 Time Series
Benchmark) is a dataset proposed by Lai et al.
and includes ﬁve time series anomaly scenarios categorized by behavior-driven taxonomy as point-global, pattern-contextual, pattern-shapelet, pattern-seasonal and pattern-trend.
The statistical details are summarized in Table 13 of Appendix.
Implementation details Following the well-established protocol in Shen et al.
(2020), we adopt a non-overlapped sliding window to obtain a set of sub-series.
The sliding window is with a ﬁxed size of 100 for all datasets.
We label the time points as anomalies if their anomaly scores (Equation 6) are larger than a certain threshold δ .
The threshold δ is determined to make r proportion data of the validation dataset labeled as anomalies.
For the main results, we set r = 0 .
1% for SWaT, 0.5% for
SMD and 1% for other datasets.
We adopt the widely-used adjustment strategy (Xu et al., 2018;
Su et al., 2019;
Shen et al., 2020): if a time point in a certain successive abnormal segment is detected, all anomalies in this abnormal segment are viewed to be correctly detected.
This strategy is justiﬁed from the observation that an abnormal time point will cause an alert and further make the whole segment noticed in real-world applications.
Anomaly Transformer contains 3 layers.
We set the channel number of hidden states d model as 512 and the number of heads h as 8.
The hyperparameter λ (Equation 4) is set as 3 for all datasets to trade-off two parts of the loss function.
We use the
ADAM (Kingma & Ba, 2015) optimizer with an initial learning rate of 10 − 4 .
The training process is early stopped within 10 epochs with the batch size of 32.
All the experiments are implemented in
Pytorch (Paszke et al., 2019) with a single NVIDIA TITAN RTX 24GB GPU.
Baselines We extensively compare our model with 18 baselines, including the reconstructionbased models: InterFusion (2021), BeatGAN (2019), OmniAnomaly (2019), LSTM-VAE (2018); the density-estimation models: DAGMM (2018), MPPCACD (2017), LOF (2000); the clusteringbased methods: ITAD (2020), THOC (2020), Deep-SVDD (2018); the autoregression-based models: CL-MPPCA (2019), LSTM (2018), VAR (1976); the classic methods: OC-SVM (2004), IsolationForest (2008).
Another 3 baselines from change point detection and time series segmentation are deferred to Appendix I. InterFusion (2021) and THOC (2020) are the state-of-the-art deep models.
[Table]
[Figure]
[Table]
4. M AIN R ESULTS
Real-world datasets We extensively evaluate our model on ﬁve real-world datasets with ten competitive baselines.
As shown in Table 1, Anomaly Transformer achieves the consistent state-of-theart on all benchmarks.
We observe that deep models that consider the temporal information outperform the general anomaly detection model, such as Deep-SVDD (Ruff et al., 2018) and DAGMM (Zong et al., 2018), which veriﬁes the effectiveness of temporal modeling.
Our proposed Anomaly
Transformer goes beyond the point-wise representation learned by RNNs and models the more informative associations.
The results in Table 1 are persuasive for the advantage of association learning in time series anomaly detection.
In addition, we plot the ROC curve in Figure 3 for a complete comparison.
Anomaly Transformer has the highest AUC values on all ﬁve datasets.
It means that our model performs well in the false-positive and true-positive rates under various pre-selected thresholds, which is important for real-world applications
[Figure]
NeurIPS-TS benchmark This benchmark is generated from well-designed rules proposed by Lai et al.
(2021), which completely includes all types of anomalies, covering both the pointwise and pattern-wise anomalies.
As shown in Figure 4, Anomaly
Transformer can still achieve state-of-the-art performance.
This veriﬁes the effectiveness of our model on various anomalies.
Ablation study As shown in Table 2, we further investigate the effect of each part in our model.
Our association-based criterion outperforms the widely-used reconstruction criterion consistently.
Speciﬁcally, the association-based criterion brings a remarkable 18.76% (76.20 → 94.96) averaged absolute F1-score promotion.
Also, directly taking the association discrepancy as the criterion still achieves a good performance (F1-score: 91.55%) and surpasses the previous state-of-the-art model
THOC (F1-score: 88.01% calculated from Table 1).
Besides, the learnable prior-association (corresponding to σ in Equation 2) and the minimax strategy can further improve our model and get 8.43% (79.05 → 87.48) and 7.48% (87.48 → 94.96) averaged absolute promotions respectively.
Finally, our proposed Anomaly Transformer surpasses the pure Transformer by 18.34% (76.62 → 94.96) absolute improvement.
These verify that each module of our design is effective and necessary.
More ablations of association discrepancy can be found in Appendix D.
[Table]
4. M ODEL A NALYSIS
To explain how our model works intuitively
[Figure]
we provide the visualization and statistical results for our three key designs: anomaly criterion, learnable prior-association and optimization strategy
[Figure]
i n a l L o
[Figure]
[Table]
[Figure]
I npu t
T i m e
Se r i e s
R e c on s t r u c t i on
C r i t e r i on
A ss o c
i a t
[Table]
[Figure]
[Table]
[Figure]
[Table]
(cid:57)
Anomaly criterion visualization To get more intuitive cases about how association-based criterion works, we provide some visualization in Figure 5 and explore the criterion performance under different types of anomalies, where the taxonomy is from Lai et al.
We can ﬁnd that our proposed association-based criterion is more distinguishable in general.
Concretely, the associationbased criterion can obtain the consistent smaller values for the normal part, which is quite contrasting in point-contextual and pattern-seasonal cases (Figure 5).
In contrast, the jitter curves of the reconstruction criterion make the detection process confused and fail in the aforementioned two cases.
This veriﬁes that our criterion can highlight the anomalies and provide distinct values for normal and abnormal points, making the detection precise and reducing the false-positive rate
[Figure]
InputTimeSeries ! !
[Figure]
Prior-association visualization During the minimax optimization
, the prior-association is learned to get close to the series-association.
Thus, the learned σ can reﬂect the adjacent-concentrating degree of time series.
As shown in Figure 6, we ﬁnd that σ changes to adapt to various data patterns of time series.
Especially, the prior-association of anomalies generally has a smaller σ than normal time points, which matches our adjacent-concentration inductive bias of anomalies.
Optimization strategy analysis Only with the reconstruction loss, the abnormal and normal time points present similar performance in the association weights to adjacent time points, corresponding to a contrast value closed to 1 (Table 3).
Maximizing the association discrepancy will force the series-associations to pay more attention to the non-adjacent area.
However, to obtain a better reconstruction, the anomalies have to maintain much larger adjacent association weights than normal time points, corresponding to a larger contrast value.
But direct maximization will cause the optimization problem of Gaussian kernel, cannot strongly amplify the difference between normal and abnormal time points as expected (SMD:1.15 → 1.27).
The minimax strategy optimizes the prior-association to provide a stronger constraint to series-association.
Thus, the minimax strategy obtains more distinguishable contrast values than direct maximization (SMD:1.27 → 2.39) and thereby performs better.
[Table]
C ONCLUSION AND F UTURE W ORK
This paper studies the unsupervised time series anomaly detection problem.
Unlike previous methods, we learn the more informative time-point associations by Transformers.
Based on the key observation of association discrepancy, we propose the Anomaly Transformer, including an AnomalyAttention with the two-branch structure to embody the association discrepancy.
A minimax strategy is adopted to further amplify the difference between normal and abnormal time points.
By introducing the association discrepancy, we propose the association-based criterion, which makes the reconstruction performance and association discrepancy collaborate.
Anomaly Transformer achieves the state-of-the-art results on an exhaustive set of empirical studies.
Future work includes theoretical study of Anomoly Transformer in light of classic analysis for autoregression and state space models.
A P ARAMETER S ENSITIVITY
We set the window size as 100 throughout the main text, which considers the temporal information, memory and computation efﬁciency.
And we set the loss weight λ based on the convergence property of the training curve.
Furthermore, Figure 7 provides the model performance under different choices of the window size and the loss weight.
We present that our model is stable to the window size over extensive datasets (Figure 7 left).
Note that a larger window size indicates a larger memory cost and a smaller sliding number.
Especially, only considering the performance, its relationship to the window size can be determined by the data pattern.
For example, our model performs better when the window size isfor the SMD dataset.
Besides, we adopt the loss weight λ in Equation 5 to trade off the reconstruction loss and the association part.
We ﬁnd that λ is stable and easy to tune in the range of 2 to 4.
The above results verify the sensitivity of our model, which is essential for applications
[Figure]
B I MPLEMENTATION D ETAILS
We present the pseudo-code of Anomaly-Attention in Algorithm 1.
Algorithm
Input: X ∈ R N × d
[Table]
: input; D = (cid:16)
[Table]
j − i )
[Table] [Table]
[Table] [Table]
[Table]
: relative distance matrix Layer params: MLP input : linear projector for input; MLP output
linear projector for output
[Table]
: Q , K , V , σ = Split (cid:16) MLP input
[Table] [Table]
=
[Table]
[Equation]
2: for
Q m , K m , V m , σ m
( Q , K , V , σ ) :
[Table]
[Equation]
[Table]
N ×
[Table]
3: for σ m = Broadcast
m ,
[Equation]
[Table]
[Equation]
[Table]
: for
m =
[Table]
exp
[Table]
(cid:17)
[Table]
[Equation]
[Table]
×
[Table]
: for P m = P m / Broadcast
Sum
[Table]
C M ORE S HOWCASES
To obtain an intuitive comparison of main results (Table 1), we visualize the criterion of various baselines.
Anomaly Transformer can present the most distinguishable criterion (Figure 8).
Besides, for the real-world dataset, Anomaly Transformer can also detect the anomalies correctly.
Especially for the SWaT dataset (Figure 9(d)), our model can detect the anomalies in the early stage, which is meaningful for real-world applications, such as the early warning of malfunctions
[Figure]
I npu t
T i m e
Se r i e s
B ea t G
A N
D ee p - S
V DD
[Figure]
A no m a l y
T r a n s f o r m e r
[Table]
[Figure]
[Table]
[Figure] [Figure]
I npu t
T i m e
Se r i e s
A ss o c
i a t i on - b
a s e d
C r i t e r
i
[Figure]
D A BLATION OF A SSOCIATION D ISCREPANCY
We present the pseudo code of the calculation in Algorithm 2.
D. A BLATION OF M ULTI -L EVEL Q UANTIFICATION
We average the association discrepancy from multiple layers for the ﬁnal results (Equation 6).
We further investigate the model performance under the single-layer usage.
As shown in Table 4, the multiple-layer design achieves the best, which veriﬁes the effectiveness of multi-level quantiﬁcation.
[Table]
D. A BLATION OF S TATISTICAL D ISTANCE
We select the following widely-used statistical distances to calculate the association discrepancy:
• Symmetrized Kullback–Leibler Divergence (Ours).
• Jensen–Shannon Divergence (JSD).
• Wasserstein Distance (Wasserstein).
• Cross-Entropy (CE).
• L2 Distance (L2).
[Table]
As shown in Table
[Table]
our proposed deﬁnition of association discrepancy still achieves the best performance.
We ﬁnd that both the CE and JSD can provide fairly good results
[Table]
which are close to our deﬁnition in principle and can be used to represent the information gain.
The L2 distance is not suitable for the discrepancy
[Table]
which overlooks the property of discrete distribution
The Wasserstein distance also fails in some datasets.
The reason is that the prior-association and series-association are exactly matched in the position indexes.
Still
[Table]
the Wasserstein distance is not calculated point by point and considers the distribution offset, which may bring noises to the optimization and detection
[Table] [Table]
P (cid:48)
∈ R L × N × N 2: S (cid:48)
=
Mean ( S , dim=1 )
(cid:46) S (cid:48)
∈ R L × N × N 3: R (cid:48) =
KL (cid:16) ( P (cid:48) , S (cid:48) ) , dim=-1 (cid:17) + KL (cid:16) ( S (cid:48) , P (cid:48) ) , dim=-1 (cid:17) (cid:46) R (cid:48)
∈ R L × N
[Table]
= Mean ( R (cid:48) , dim=0 )
(cid:46) R ∈
R N ×
[Table]
Return R (cid:46) Represent the association discrepancy of each time point
D. A BLATION OF P RIOR -A SSOCIATION
In addition to the Gaussian kernel with a learnable scale parameter, we also try to use the power-law kernel P ( x ; α ) =
x − α with a learnable power parameter α for prior-association, which is also a unimodal distribution.
As shown in Table 6, power-law kernel can achieve a good performance in most of the datasets.
However, because the scale parameter is easier to optimize than the parameter of power, Gaussian kernel still surpasses the power-law kernel consistently.
[Table]
E A BLATION OF A SSOCIATION - BASED C RITERION E. C ALCULATION
We present the pseudo-code of association-based criterion in Algorithm
[Table] [Table]
; association discrepancy AssDis( P , S ; X ) ∈ R N
[Table]
[Table] [Table]
(cid:16)
[Table] [Table]
[Table]
(cid:46)
[Equation]
C = C AD × C Recon (cid:46)
[Equation]
Return
E. A BLATION OF C RITERION D EFINITION
We explore the model performance under different deﬁnitions of anomaly criterion, including the pure association discrepancy, pure reconstruction performance and different combination methods for association discrepancy and reconstruction performance: addition and multiplication.
[Equation]
From Table
[Equation]
we ﬁnd that directly using our proposed association discrepancy can also achieve a good performance, which surpasses the competitive baseline THOC (Shen et al., 2020) consistently.
Besides, the multiplication combination that we used in Equation 6 performs the best, which can bring a better collaboration to the reconstruction performance and association discrepancy.
[Table]
F C ONVERGENCE OF M INIMAX O PTIMIZATION
The total loss of our model (Equation 4) contains two parts: the reconstruction loss and the association discrepancy.
Towards a better control of association learning, we adopt a minimax strategy for optimization (Equation 5).
During the minimization phase, the optimization trends to minimize the association discrepancy and the reconstruction error.
During the maximization phase, the optimization trends to maximize the association discrepancy and minimize the reconstruction error.
We plot the change curve of the above two parts during the training procedure.
As shown in Figures 10 and 11, both parts of the total loss can converge within limited iterations on all the ﬁve real-world datasets.
This nice convergence property is essential for the optimization of our model
[Figure]
[Figure]
G M ODEL P ARAMETER S ENSITIVITY
In this paper
[Figure]
we set the hyper-parameters L and d model following the convention of Transformers (Vaswani et al., 2017; Zhou et al., 2021).
Furthermore
[Figure]
to evaluate model parameter sensitivity
[Figure]
we investigate the performance and efﬁciency under different choices for the number of layers L and hidden channels d model .
Generally, increasing the model size can obtain better results but with larger memory and computation costs.
[Table] [Table]
H P ROTOCOL OF T HRESHOLD S ELECTION
Our paper focuses on unsupervised time series anomaly detection.
Experimentally, each dataset includes training, validation and testing subsets.
Anomalies are only labeled in the testing subset.
Thus
[Table]
we select the hyper-parameters following the Gap Statistic method (Tibshirani et al., 2001) in K-Means .
Here is the selection procedure:
• After the training phase, we apply the model to the validation subset (without label) and obtain the anomaly scores (Equation 6) of all time points.
• We count the frequency of the anomaly scores in the validation subset.
It is observed that the distribution of anomaly scores is separated into two clusters.
We ﬁnd that the cluster with a larger anomaly score contains r time points.
And for our model, r is closed to 0.1%, 0.5%, 1% for SWaT, SMD and other datasets respectively (Table 10).
• Due to the size of the test subset being still inaccessible in real-world applications, we have to ﬁx the threshold as a ﬁxed value δ , which can gaurantee that the anomaly scores of r time points in the validation set are larger than δ and thus detected as anomalies.
[Table]
Note that, directly setting the δ is also feasible.
According to the intervals in Table 10, we can ﬁx the δ as 0.1 for the SMD, MSL and SWaT datasets, 0.01 for the SMAP and PSM datasets, which yield a quite close performance to setting r .
[Table]
In real-world applications, the number of selected anomalies is always decided up to human resources.
Under this consideration, setting the number of detected anomalies by the ratio r is more practical and easier to decide according to the available resources.
I M ORE B ASELINES
In addition to the time series anomaly detection methods, the methods for change point detection and time series segmentation can also perform as valuable baselines.
Thus, we also include the BOCPD (Adams & MacKay, 2007) and TS-CP2 (Deldari et al., 2021) from change point detection and UTime (Perslev et al., 2019) from time series segmentation for comparison.
Anomaly Transformer still achieves the best performance.
[Table]
J L IMITATIONS AND F UTURE W ORK
Window size As shown in the Figure 7 of Appendix A, the model may fail if the window size is too small for association learning.
But the Transformers is with quadratic complexity w
[Table]
r.t. the window size.
The trade-off is needed for real-world applications.
Theoretical analysis As a well-established deep model, the performance of Transformers has been explored in previous works.
But it is still under-exploring for the theory of complex deep models.
In the future
[Table]
we will explore the theorem of Anomaly Transformer for better justiﬁcations in light of classic analysis for autoregression and state space models.
K D ATASET
Here is the statistical details of experiment datasets. [Table] TS
Various Anomalies
[Table]
L UCR D ATASET
UCR Dataset is a very challenging and comprehensive dataset provided by the Multi-dataset Time
Series Anomaly Detection Competition of KDD2021 (Keogh et al.
, Competition of International
Conference on Knowledge Discovery & Data Mining 2021).
The whole dataset contains 250 subdatasets, covering various real-world scenarios.
Each sub-dataset of UCR has only one anomaly segment and only has one dimension.
These sub-datasets range in length from 6,684 to 900,000 and are pre-divided into training and test sets.
We also experiment on the UCR dataset for a wide evaluation.
As show in Table 14, our Anomaly
Transformer still achieves the state-of-the-art in this challenging benchmark.
[Table]