A NOMALY T RANSFORMER : T IME S ERIES A NOMALY D ETECTION WITH A SSOCIATION D ISCREPANCY
Anonymous authors
Paper under double-blind review
A BSTRACT
Unsupervisedly detecting anomaly points in time series is challenging, which requires the model to learn informative representations and derive a distinguishable criterion.
Prior methods mainly detect anomalies based on the point-wise representations from recurrent networks.
However, the point-wise representation is less informative for complex temporal patterns and can be dominated by normal patterns, making rare anomalies less distinguishable.
We ﬁnd that in each time series, each time point can also be described by its associations with all time points, presenting as a point-wise distribution that is more expressive for temporal modeling.
We further observe that due to the rarity of anomalies, it is harder for anomalies to build strong associations with the whole series and their associations shall mainly concentrate on the adjacent time points.
This observation implies an inherently distinguishable criterion between normal and abnormal points, which we highlight as the Association Discrepancy .
Technically we propose the Anomaly Transformer with an Anomaly-Attention mechanism to compute the association discrepancy.
A minimax strategy is devised to amplify the normal-abnormal distinguishability of the association discrepancy.
Anomaly Transformer achieves state-of-the-art performance on six unsupervised time series anomaly detection benchmarks for three applications: service monitoring, space & earth exploration, and water treatment.
I NTRODUCTION
Real-world systems always work in a continuous way, which can generate several successive measurements monitored by multi-sensors, such as industrial equipment, space probe, etc.
Discovering the malfunctions from large-scale system monitoring can be reduced to detecting the abnormal time points from time series, which is quite meaningful for ensuring security and avoiding ﬁnancial loss.
But anomalies are usually rare and hidden by vast normal points, making the data labeling hard and expensive.
Thus, we focus on time series anomaly detection under the unsupervised setting.
Unsupervised time series anomaly detection is extremely challenging in practice.
The model should learn informative representations from complex temporal dynamics through unsupervised tasks. Still, it should also derive a distinguishable criterion, which can detect the rare anomalies from plenty of normal time points.
Various classic anomaly detection methods have provided many unsupervised paradigms, such as the density-estimation methods proposed in local outlier factor (LOF, Breunig et al.
, clustering-based methods presented in one-class SVM (OC-SVM, Sch¨olkopf et al.
and SVDD (Tax & Duin, 2004).
But these classic methods do not consider the temporal information and are difﬁcult to generalize to unseen real-world scenarios.
Beneﬁting from the great representation learning capability of neural networks, recent deep models (Su et al., 2019;
Shen et al., 2020; Li et al., 2021) have made remarkable advances.
They mainly focus on learning pointwise temporal representations through well-designed recurrent networks and are self-supervised by the reconstruction or autoregressive task. Here, a natural and practical anomaly criterion is the pointwise reconstruction or prediction error.
However, due to the rarity of anomalies, the point-wise representation is less informative for complex temporal patterns and can be dominated by normal time points, making anomalies less distinguishable.
Also, the reconstruction or prediction error is calculated point-wisely, which cannot provide a comprehensive description of the temporal context.
From a new perspective, we ﬁnd that in each time series, each time point can also be represented by its associations with all the time points, presenting as a distribution of association weights along the temporal dimension.
The association distribution of each time point can provide a more informative description for the temporal context, indicating dynamic patterns, such as the period or trend.
This association distribution is named by series-association , which can be discovered from the raw series.
Further, we observe that due to the rarity of anomalies and the dominance of normal patterns, it is harder for anomalies to build strong associations with the whole series.
The associations of anomalies shall concentrate on the adjacent time points that are more likely to contain similar abnormal patterns due to the continuity.
Such an adjacent-concentration inductive bias is referred to as the prior-association .
In contrast, the dominating normal time points can discover informative associations with the whole series, not limiting to the adjacent area.
Based on this observation, we try to utilize the inherent normal-abnormal distinguishability of the association distribution.
This leads to a new anomaly criterion for each time point, quantiﬁed by the distance between each time point’s prior-association and its series-association, named as Association Discrepancy .
As aforementioned, because the associations of anomalies are more likely to be adjacent-concentrating, anomalies will present a smaller association discrepancy than normal time points.
Taking the advantage of the great model capability of Transformers (Vaswani et al., 2017; Devlin et al., 2019; Brown et al., 2020), we introduce them to the unsupervised time series anomaly detection and propose the Anomaly Transformer for association learning.
To compute the Association Discrepancy, we renovate the self-attention mechanism to the Anomaly-Attention , which contains a two-branch structure to model the prior-association and series-association of each time point respectively.
The prior-association employs the learnable Gaussian prior to present the adjacentconcentration inductive bias of each time point, while the series-association corresponds to the selfattention weights learned from raw series.
Besides, a minimax strategy is applied between the two branches, which can amplify the normal-abnormal distinguishability of the Association Discrepancy and further derive a new association-based criterion.
Anomaly Transformer achieves strong results on six benchmarks, covering three real applications.
The contributions are summarized as follows:
• Based on the key observation of Association Discrepancy, we propose the Anomaly Transformer with an Anomaly-Attention mechanism, which can model the prior-association and series-association simultaneously to embody the Association Discrepancy.
• We propose a minimax strategy to amplify the normal-abnormal distinguishability of the Association Discrepancy and further derive a new association-based detection criterion.
• Anomaly Transformer achieves the state-of-the-art anomaly detection results on six benchmarks for three real applications.
Extensive ablations and insightful case studies are given.
R ELATED W ORK 2. U NSUPERVISED T IME S ERIES A NOMALY D ETECTION
As a vital real-world problem, unsupervised time series anomaly detection has been widely explored.
Categorizing by anomaly determination criterion, the paradigms roughly include the densityestimation, clustering-based, reconstruction-based and autoregression-based methods.
As for the density-estimation methods, the classic methods local outlier factor (LOF, Breunig et al.
and connectivity outlier factor (COF, Tang et al.
(2002)) calculates the local density and local connectivity for outlier determination respectively.
DAGMM (Zong et al., 2018) and MPPCACD (Yairi et al., 2017) integrate the Gaussian Mixture Model to estimate the density of representations.
In clustering-based methods, the anomaly score is always formalized as the distance to cluster center. SVDD (Tax & Duin, 2004) and Deep SVDD (Ruff et al., 2018) gather the representations from normal data to a compact cluster.
THOC (Shen et al., 2020) fuses the multi-scale temporal features from intermediate layers by a hierarchical clustering mechanism and detects the anomalies by the multi-layer distances. ITAD (Shin et al., 2020) conducts the clustering on decomposed tensors.
The reconstruction-based models attempt to detect the anomalies by the reconstruction error.
Park et al.
(2018) presented the LSTM-VAE model that employed the LSTM backbone for temporal modeling and the Variational AutoEncoder (VAE) for reconstruction.
OmniAnomaly proposed by
Su et al.
(2019) further extends the LSTM-VAE model with a normalizing ﬂow and uses the reconstruction probabilities for detection.
InterFusion from Li et al.
(2021) renovates the backbone to a hierarchical VAE to model the inter- and intra- dependency among multiple series simultaneously.
GANs (Goodfellow et al., 2014) are also used for reconstruction-based anomaly detection (Schlegl et al., 2019;
Li et al., 2019a;
Zhou et al., 2019) and perform as an adversarial regularization.
The autoregression-based models detect the anomalies by the prediction error.
VAR extends ARIMA (Anderson & Kendall, 1976) and predicts the future based on the lag-dependent covariance.
The autoregressive model can also be replaced by LSTMs (Hundman et al., 2018; Tariq et al., 2019).
This paper is characterized by a new association-based anomaly detection criterion, which is embodied by a co-design of the temporal models for learning more informative time-point associations.
2. T RANSFORMERS FOR T IME S ERIES A NALYSIS
Recently, Transformers (Vaswani et al., 2017) have shown great power in sequential data processing, such as natural language processing (Devlin et al., 2019; Brown et al., 2020), audio processing (Huang et al., 2019) and computer vision (Dosovitskiy et al., 2021; Liu et al., 2021).
For time series analysis, beneﬁting from the advantage of the self-attention mechanism, Transformers are used to discover the reliable long-range temporal dependencies (Kitaev et al., 2020;
Li et al., 2019b; Zhou et al., 2021; Wu et al., 2021).
Especially for time series anomaly detection, GTA proposed by Chen et al.
(2021) employs the graph structure to learn the relationship among multiple IoT sensors, as well as the Transformer for temporal modeling and the reconstruction criterion for anomaly detection.
Unlike the previous usage of Transformers, Anomaly Transformer renovates the self-attention mechanism to the Anomaly-Attention based on the key observation of association discrepancy and detects the anomalies based on our proposed association-based criterion.
M ETHOD
Suppose monitoring a successive system of d measurements and recording the equally spaced observations over time.
The observed time series X is denoted by a set of time points { x 1 , x 2 , · · · , x N } , where x t ∈ R d represents the observation of time t .
The unsupervised time series anomaly detection problem is to determine whether x t is anomalous or not without labels.
As aforementioned, we highlight the key to unsupervised time series anomaly detection as learning informative representations and ﬁnding distinguishable criterion.
We propose the Anomaly Transformer to discover more informative associations and tackle this problem by learning the Association Discrepancy , which is inherently normal-abnormal distinguishable.
Technically, we propose the
Anomaly-Attention to embody the prior-association and series-associations, along with a minimax optimization strategy to obtain a more distinguishable association discrepancy.
Co-designed with the architecture, we derive an association-based criterion based on the learned association discrepancy.
3. A NOMALY T RANSFORMER
Given the limitation of Transformers (Vaswani et al., 2017) for anomaly detection, we renovate the vanilla architecture to the Anomaly Transformer (Figure 1) with an Anomaly-Attention mechanism.
Overall Architecture Anomaly Transformer is characterized by stacking the Anomaly-Attention blocks and feed-forward layers alternately.
This stacking structure is conducive to learning underlying associations from deep multi-level features.
Suppose the model contains L layers with length- N input time series X ∈ R N × d .
The overall equations of the l -th layer are formalized as:
[Equation]
(1) where X l ∈ R N × d model , l ∈ { 1 , · · · , L } denotes the output of the l -th layer with d model channels.
The initial input
X
=
Embedding( X ) represents the embedded raw series.
Z l ∈ R N × d model is the l -th layer’s hidden representation.
Anomaly - Attention( · ) is to compute the association discrepancy.
Anomaly-Attention Note that the single-branch self-attention mechanism (Vaswani et al., 2017) cannot model the prior-association and series-association simultaneously.
We propose the AnomalyAttention with a two-branch structure (Figure 1).
For the prior-association , we adopt a learnable
Sigma L i n ea r
Q K V M a t M u l Sca l e S
o ft M ax GlobalRelation LocalRelation DiagonalFilling
M a t M u l
KL Distance Min KL Max KL
[Table]
KL
[Table]
Q
K
C on ca t
L i n ea r
AnomalyAttention Layer Norm Feed Forward Layer Norm + + Anomaly Attention
Global
[Table]
LocalKL Distance ReconstructionLoss Sigma L i n ea r
Q K V M a t M u l Sca l e S o ft M ax
Global-SeriesAssociation Adjacent-SeriesAssociation
D i ag on a li ze
M a t M u l
KL Distance MinKL MaxKL
[Table]
KL Distance Stop gradient
L i n ea r
AnomalyAttention Layer Norm Feed Forward Layer Norm + + Reconstruction L x
[Table]
N (1 , 21 )
N (2 , 22 )
N ( N, 2 N )
[Table]
N (
[Figure]
Gaussian distribution as the prior, centered at the corresponding index.
Beneﬁting from the unimodal property of the Gaussian family, this design can pay more attention to the adjacent horizon constitutionally.
We also use a learnable variance parameter σ for the Gaussian prior, making the prior-associations adapt to the various time series patterns, such as different lengths of anomaly segments.
The series-association branch is to learn the associations from raw series, which can ﬁnd the most effective associations adaptively.
Note that these two forms maintain the temporal dependencies of each time point, which are more informative than point-wise representation.
They also reﬂect the adjacent-concentration prior and the learned real associations respectively, whose discrepancy shall be normal-abnormal distinguishable.
The Anomaly-Attention in the l -th layer is: Initialization: [Equation] Prior-
[Equation]
Series-Association
[Equation]
Reconstruction [Equation] [Equation] where Q , K , V ∈ R N × d model , σ ∈ R N × 1 represent the query, key, value of self-attention and the learned variance respectively.
W l Q , W l K , W l V ∈ R d model × d model , W lσ ∈ R d model × 1 represent the parameter matrices for Q , K , V , σ in the l -th layer respectively.
Prior-association P l ∈ R N × N is generated based on the learned variance σ ∈ R N × 1 and the i -th element σ i corresponds to the i -th time point.
Concretely, for the i -th time point, its association weight to the j -th point is the probability density of the Gaussian distribution
N ( i, σ 2 i ) at the index | j − i | .
Further, we use Scale( · ) to transform the association weights to discrete distributions P l by dividing the row sum.
S l ∈ R N × N denotes the series-associations. Softmax( · ) normalizes the attention map along the last dimension. Thus, each row of S l forms a discrete distribution.
(cid:98)
Z l ∈ R N × d model is the hidden representation after the Anomaly-Attention in the l -th layer.
We use Anomaly - Attention( · ) to summarize Equation 2.
In the multi-head version that we use, the learned variance is σ ∈ R N × h for h heads.
Q m , K m , V m ∈
R N × d model h denote the query, key and value of the m -th head respectively.
The block concatenates the outputs { (cid:98) Z lm ∈ R N × d model h } 1 ≤ m ≤ h from multiple heads and gets the ﬁnal result (cid:98) Z l ∈ R N × d model .
Association Discrepancy We formalize the Association Discrepancy as the symmetrized KL divergence between prior- and series- associations, which represents the information gain between these two distributions (Neal, 2007).
We average the association discrepancy from multiple layers to combine the associations from multi-level features into a more informative measure as:
[Equation]
AssociationDiscrepancy b.
AssociationDiscrepancy d. Min KL Max KL a.
Min KL Max KL c.
Association Space Adjacet-series Association
Global-series Association Gaussian Family
Reconstruction Loss Constrained Space Normal Time Points Abnormal Time Points
Maximize Phase Minimize Phase AssociationDiscrepancy
Association Space Adjacet-seriesAssociation
Global-series
Adjacet-seriesAssociation Global-seriesAssociation
Constrained by Reconstruction Global-seriesAssociation Adjacet-seriesAssociation
Within the Gaussian Family Time series
Time series
OptimizationDirection
[Figure]
where KL( ·(cid:107)· ) is the KL divergence computed between two discrete distributions corresponding to every row of P l and S l .
AssDis( P , S ; X
) ∈ R N × 1 is the point-wise association discrepancy of X with respect to prior-association
P and series-association S from multiple layers.
The i -th element of results corresponds to the i -th time point of X .
From previous observation, anomalies will present smaller AssDis( P , S ; X ) than normal time points, which makes AssDis inherently distinguishable.
3. M INIMAX A SSOCIATION L EARNING
As an unsupervised task, we employ the reconstruction loss for optimizing our model.
The reconstruction loss will guide the series-association to ﬁnd the most informative associations.
To further amplify the difference between normal and abnormal time points, we also use an additional loss to enlarge the association discrepancy.
Due to the unimodal property of the prior-association, the discrepancy loss will guide the series-association to pay more attention to the non-adjacent area, which makes the reconstruction of anomalies harder and makes anomalies more identiﬁable.
The loss function for input series X ∈ R N × d is formalized as:
[Equation]
where (cid:98) X ∈ R N ×
d denotes the reconstruction of X . (cid:107)·(cid:107) F , (cid:107)·(cid:107) k indicate the Frobenius and k -norm.
λ is to trade off the loss terms.
When λ > 0 , the optimization is to enlarge the association discrepancy.
A minimax strategy is proposed to make the association discrepancy more distinguishable.
Minimax Strategy Note that directly maximizing the association discrepancy will extremely reduce the variance of the Gaussian prior (Neal, 2007), making the prior-association meaningless.
Towards a better control of association learning, we propose a minimax strategy (Figure 2).
Concretely, for the minimize phase , we drive the prior-association P l to approximate the series-association S l that is learned from raw series.
This process will make the prior-association adapt to various temporal patterns.
For the maximize phase , we optimize the series-association to enlarge the association discrepancy.
This process forces the series-association to pay more attention to the non-adjacent horizon.
Thus, integrating the reconstruction loss, the total loss functions of these two phases are: Minimize
[Equation]
Maximize [Equation] where λ > 0 and ∗ detach means to stop the gradient backpropagation of the association (Figure 1).
As P approximates S detach in the minimize phase, the maximize phase will conduct a stronger constraint to the series-association, forcing the time points to pay more attention to the non-adjacent area.
Under the reconstruction loss, this is much harder for anomalies to achieve than normal time points, thereby amplifying the normal-abnormal distinguishability of the association discrepancy.
Association-based Anomaly Criterion We incorporate the normalized association discrepancy to the reconstruction criterion, which will take the beneﬁts of both temporal representation and the distinguishable association discrepancy.
The ﬁnal anomaly score of X ∈ R N × d is shown as follows: [Equation]
[Equation]
where (cid:12) is the element-wise multiplication. AnomalyScore( X ) ∈ R N × 1 denotes the point-wise anomaly criterion of X .
Towards a better reconstruction, anomalies usually decrease the association discrepancy, which will still derive a higher anomaly score.
Thus, this design can make the reconstruction error and the association discrepancy collaborate to improve detection performance.
E XPERIMENTS
We extensively evaluate Anomaly Transformer on six benchmarks for three practical applications.
Datasets Here is a description of the six experiment datasets: (1) SMD (Server Machine Dataset,
Su et al. (2019)) is a 5-week-long dataset that is collected from a large Internet company with 38 dimensions.
(2) PSM (Pooled Server Metrics, Abdulaal et al.
(2021)) is collected internally from multiple application server nodes at eBay with 26 dimensions.
(3) Both MSL (Mars Science Laboratory rover) and SMAP (Soil Moisture Active Passive satellite) are public datasets from NASA (Hundman et al., 2018) with 55 and 25 dimensions respectively, which contain the telemetry anomaly data derived from the Incident Surprise Anomaly (ISA) reports of spacecraft monitoring systems.
(4) SWaT (Secure Water Treatment, Mathur & Tippenhauer (2016)) is obtained from 51 sensors of the critical infrastructure system under continuous operations.
(5) NeurIPS-TS (NeurIPS 2021 Time Series
Benchmark) is a dataset proposed by Lai et al.
and includes ﬁve time series anomaly scenarios categorized by behavior-driven taxonomy as point-global, pattern-contextual, pattern-shapelet, pattern-seasonal and pattern-trend.
The statistical details are summarized in Table 1.
[Table]
Implementation details Following the well-established protocol in Shen et al.
(2020), we adopt a non-overlapped sliding window to obtain a set of sub-series.
The sliding window is with a ﬁxed size of 100 for all datasets as shown in Table 1.
We label the time points as anomalies if their anomaly scores (Equation 6) are larger than a certain threshold δ .
The threshold δ is determined to make r proportion data of the validation dataset labeled as anomalies.
For the main results, we set r = 0 .
1% for SWaT, 0.5% for SMD and 1% for other datasets.
We adopt the widely-used adjustment strategy (Xu et al., 2018;
Su et al., 2019;
Shen et al., 2020): if a time point in a certain successive abnormal segment is detected, all anomalies in this abnormal segment are viewed to be correctly detected.
This strategy is justiﬁed from the observation that an abnormal time point will cause an alert and further make the whole segment noticed in real-world applications.
Anomaly Transformer contains 3 layers.
We set the channel number of hidden states d model as 512 and the number of heads h as 8.
The hyperparameter λ (Equation 4) is set as 3 for all datasets to trade-off two parts of the loss function.
We use the ADAM (Kingma & Ba, 2015) optimizer with an initial learning rate of 10 − 4 .
The training process is early stopped within 10 epochs with the batch size of 32.
All the experiments are implemented in Pytorch (Paszke et al., 2019) with a single NVIDIA TITAN RTX 24GB GPU.
Baselines We extensively compare our model with 18 baselines, including the reconstructionbased models: InterFusion (2021), BeatGAN (2019), OmniAnomaly (2019), LSTM-VAE (2018); the density-estimation models: DAGMM (2018), MPPCACD (2017), LOF (2000); the clusteringbased methods: ITAD (2020), THOC (2020), Deep-SVDD (2018); the autoregression-based models: CL-MPPCA (2019), LSTM (2018), VAR (1976); the classic methods: OC-SVM (2004), IsolationForest (2008).
Another 3 baselines from change point detection and time series segmentation are deferred to Appendix I. InterFusion (2021) and THOC (2020) are the state-of-the-art deep models.
4. M AIN R ESULTS
Real-world datasets We extensively evaluate our model on ﬁve real-world datasets with ten competitive baselines.
As shown in Table 2, Anomaly Transformer achieves the consistent state-of-the- [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure] [Figure] [Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure]
art on all benchmarks. We observe that deep models that consider the temporal information outperform the general anomaly detection model, such as Deep-SVDD (Ruff et al., 2018) and DAGMM (Zong et al., 2018), which veriﬁes the effectiveness of temporal modeling.
Our proposed Anomaly Transformer goes beyond the point-wise representation learned by RNNs and models the more informative associations.
The results in Table 2 are persuasive for the advantage of association learning in time series anomaly detection.
In addition, we plot the ROC curve in Figure 3 for a complete comparison.
Anomaly Transformer has the highest AUC values on all ﬁve datasets.
It means that our model performs well in the false-positive and true-positive rates under various pre-selected thresholds, which is important for real-world applications.
[Figure]
NeurIPS-TS benchmark This benchmark is generated from well-designed rules proposed by Lai et al.
(2021), which completely includes all types of anomalies, covering both the point-wise and pattern-wise anomalies.
As shown in Figure 4, Anomaly Transformer can still achieve state-of-the-art performance on various anomalies. This veriﬁes the effectiveness of our model on various types of anomalies.
We provide some showcases in Appendix C.
Ablation study As shown in Table 3, we further investigate the effect of each part in our model.
Our associationbased criterion outperforms the widely-used reconstruction criterion consistently.
Speciﬁcally, the association-based criterion brings a remarkable 18.76% (76.20 → 94.96) averaged absolute F1-score promotion.
Also, directly taking the association discrepancy as the criterion still achieves a good performance (F1-score: 91.55%) and surpasses the previous state-of-the-art model THOC (F1score: 88.01% calculated from Table 2).
Besides, the learnable prior-association (correspond- ing to σ in Equation 2) and the minimax strategy can further improve our model and get 8.43% (79.05 → 87.48) and 7.48% (87.48 → 94.96) averaged absolute promotions respectively.
Finally, our proposed Anomaly Transformer surpasses the pure Transformer by 18.34% (76.62 → 94.96) absolute improvement.
These verify that each module of our design is effective and necessary.
More ablations of association discrepancy can be found in Appendix D. [Table]
4. M ODEL A NALYSIS
To explain how our model works intuitively
[Table]
we provide the visualization and statistical results for [Figure] [Figure] [Figure]
[Figure]
[Figure]
I npu t
T i m e
Se r i e s
R e c on s t r u c t i on
C r i t e r i on
A ss o c
i a t i on - b
a s e d
C r i t e r i on
DetectionError DetectionError
[Figure]
[Table]
[Figure] (cid:57) in red segments. The wrongly detected cases are bounded by red boxes. Anomaly criterion visualization To get more intuitive cases about how association-based criterion works, we provide some visualization in Figure 5 and explore the criterion performance under different types of anomalies, where the taxonomy is from Lai et al.
We can ﬁnd that our proposed association-based criterion is more distinguishable in general.
Concretely, the associationbased criterion can obtain the consistent smaller values for the normal part, which is quite contrasting in point-contextual and pattern-seasonal cases (Figure 5).
In contrast, the jitter curves of the reconstruction criterion make the detection process confused and fail in the aforementioned two cases.
This veriﬁes that our criterion can highlight the anomalies and provide distinct values for normal and abnormal points, making the detection precise and reducing the false-positive rate
[Figure]
InputTimeSeries ! !
[Figure]
Prior-association visualization During the minimax optimization
, the prior-association is learned to get close to the series-association.
Thus, the learned σ can reﬂect the adjacent-concentrating degree of time series.
As shown in Figure 6, we ﬁnd that σ changes to adapt to various data patterns of time series.
Especially, the prior-association of anomalies generally has a smaller σ than normal time points, which matches our adjacent-concentration inductive bias of anomalies.
Optimization strategy analysis Only with the reconstruction loss, the abnormal and normal time points present similar performance in the association weights to adjacent time points, corresponding to a contrast value closed to 1 (Table 4).
Maximizing the association discrepancy will force the series-associations to pay more attention to the non-adjacent area.
However, to obtain a better reconstruction, the anomalies have to maintain much larger adjacent association weights than normal time points, corresponding to a larger contrast value.
But direct maximization will cause the optimization problem of Gaussian prior and cannot strongly amplify the difference between normal and abnormal time points as expected (SMD:1.15 → 1.27).
The minimax strategy optimizes the prior-association to provide a stronger constraint to series-association.
Thus, the minimax strategy obtains more distinguishable contrast values than direct maximization (SMD:1.27 → 2.39) and thereby performs better.
[Table]
C ONCLUSION
This paper studies the unsupervised time series anomaly detection problem.
Unlike previous methods, we try to tackle this problem with more informative association learning.
Based on the key observation of association discrepancy, we propose the Anomaly Transformer, including an AnomalyAttention with the two-branch structure to embody the association discrepancy.
A minimax strategy is adopted to further amplify the difference between normal and abnormal time points.
By introducing the association discrepancy, we propose the association-based criterion, which makes the reconstruction performance and association discrepancy collaborate.
Anomaly Transformer achieves the state-of-the-art on extensive benchmarks.
Comprehensive ablations and insightful analyses are included to verify the effectiveness of our design and elaborate on how the model works.
A P ARAMETER S ENSITIVITY
We set the window size as 100 throughout the main text, which considers the temporal information, memory and computation efﬁciency.
And we set the loss weight λ based on the convergence property of the training curve.
Furthermore, Figure 7 provides the model performance under different choices of the window size and the loss weight.
We present that our model is stable to the window size over extensive datasets (Figure 7 left).
Note that a larger window size indicates a larger memory cost and a smaller sliding number.
Especially, only considering the performance, its relationship to the window size can be determined by the data pattern.
For example, our model performs better when the window size isfor the SMD dataset.
Besides, we adopt the loss weight λ in Equation 5 to trade off the reconstruction loss and the association part.
We ﬁnd that λ is stable and easy to tune in the range of 2 to 4.
The above results verify the sensitivity of our model, which is essential for applications
[Figure]
B I MPLEMENTATION D ETAILS
We present the pseudo-code of Anomaly-Attention in Algorithm 1.
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Table]
[Equation]
[Table] [Table]
C M ORE S HOWCASES
To obtain an intuitive comparison of main results (Table 2)
[Figure]
we visualize the criterion of various baselines.
Anomaly Transformer can present the most distinguishable criterion (Figure 8).
Besides, for the real-world dataset, Anomaly Transformer can also detect the anomalies correctly.
Especially for the SWaT dataset (Figure 9(d)), our model can detect the anomalies in the early stage, which is meaningful for real-world applications, such as the early warning of malfunctions
[Figure]
npu
[Figure]
T i m e
Se r i e s
B e a t G
A N
D ee p - S
V DD
[Figure]
LS T MVA E
[Figure]
A
[Figure]
m a l
[Figure]
a
[Figure]
o
[Figure]
Pattern
[Figure]
Shapelet
Pattern
[Figure]
Seasonal
Pattern
[Figure]
[Figure]
I npu t
T i m e
Se r i e s
A ss o c
i a t i on - b
a s e d
C r i t e r i on
[Figure]
D A BLATION OF A SSOCIATION D ISCREPANCY
We present the pseudo code of the calculation in Algorithm 2.
Algorithm
[Table]
[Table]
D. A BLATION OF M ULTI -L EVEL Q UANTIFICATION
We average the association discrepancy from multiple layers for the ﬁnal results (Equation 6).
We further investigate the model performance under the single-layer usage.
As shown in Table 5, the multiple-layer design achieves the best, which veriﬁes the effectiveness of multi-level quantiﬁcation
[Table] [Table]
D. A BLATION OF S TATISTICAL D ISTANCE
We select the following widely-used statistical distances to calculate the association discrepancy:
• Symmetrized Kullback–Leibler Divergence (Ours).
• Jensen–Shannon Divergence (JSD).
• Wasserstein Distance (Wasserstein).
• Cross-Entropy (CE).
• L2 Distance (L2).
[Table]
As shown in Table [Table] our proposed deﬁnition of association discrepancy still achieves the best performance.
We ﬁnd that both the CE and JSD can provide fairly good results, which are close to our deﬁnition in principle and can be used to represent the information gain.
The L2 distance is not suitable for the discrepancy, which overlooks the property of discrete distribution.
The Wasserstein distance also fails in some datasets.
The reason is that the prior-association and series-association are exactly matched in the position indexes.
Still, the Wasserstein distance is not calculated point by point and considers the distribution offset, which may bring noises to the optimization and detection.
D. A BLATION OF P RIOR -A SSOCIATION
In addition to the Gaussian distribution N ( index , σ ) with a learnable variance, we also try to use the power-law distribution x − α with a learnable power parameter α for prior-association, which is also a unimodal distribution.
As shown in Table 7, power-law distribution can achieve a good performance in most of the datasets.
However, because the variance parameter is easier to optimize than the parameter of power, Gaussian distribution still surpasses the power-law distribution consistently.
[Table]
E A BLATION OF A SSOCIATION - BASED C RITERION E. C ALCULATION
We present the pseudo-code of association-based criterion in Algorithm 3. Algorithm
Input: time series length N ; input time series X ∈ R
N × d ; reconstruction time series (cid:98) X ∈ R N × d ; association discrepancy AssDis( P , S ; X
) ∈ R N × 1 ; 1: C AD =
Softmax ( − AssDis( P , S ; X ) , dim=0 ) (cid:46)
[Equation]
2: C Recon = Mean (cid:16) ( X − (cid:98) X ) 2 , dim=1 (cid:17) (cid:46)
C Recon ∈ R N ×3:
C = C AD × C Recon
∈ R N ×
E. A BLATION OF C RITERION D EFINITION
We explore the model performance under different deﬁnitions of anomaly criterion, including the pure association discrepancy, pure reconstruction performance and different combination methods for association discrepancy and reconstruction performance: addition and multiplication.
Association Discrepancy: AnomalyScore(
[Equation]
= Softmax (cid:16) − AssDis( P
[Equation]
S
[Equation]
X
[Equation]
(cid:17) ,
Reconstruction: AnomalyScore
{+( X ) = (cid:104) (cid:107)X i, : − (cid:98) X i, : (cid:107) 
 (cid:105) i+}
=1 , ··· ,N ,
Addition:
AnomalyScore( X ) = Softmax (cid:16) − AssDis( P , S ; X ) (cid:17) + (cid:104) (cid:107)X i, : − (cid:98) X i, : (cid:107) 22 (cid:105) i =1 , ··· ,N ,
Multiplication
(Ours): AnomalyScore( X ) =
Softmax (cid:16) − AssDis( P , S ; X ) (cid:17) (cid:12) (cid:104) (cid:107)X i, : − (cid:98)
X i, : (cid:107) 22 (cid:105) i =1 , ··· ,N .
[Table]
From Table
[Table]
we ﬁnd that directly using our proposed association discrepancy can also achieve a good performance, which surpasses the competitive baseline THOC (Shen et al., 2020) consistently. Besides [Table]
the multiplication combination that we used in Equation 6 performs the best, which can bring a better collaboration to the reconstruction performance and association discrepancy. F C ONVERGENCE OF M INIMAX O PTIMIZATION
The total loss of our model (Equation 4) contains two parts: the reconstruction loss and the association discrepancy.
Towards a better control of association learning, we adopt a minimax strategy for optimization (Equation 5).
During the minimization phase, the optimization trends to minimize the association discrepancy and the reconstruction error.
During the maximization phase, the optimization trends to maximize the association discrepancy and minimize the reconstruction error.
We plot the change curve of the above two parts during the training procedure.
As shown in Figures 10 and 11, both parts of the total loss can converge within limited iterations on all the ﬁve real-world datasets.
This nice convergence property is essential for the optimization of our model
[Figure]
− (cid:98) X(cid:107) 2 F in real-world datasets during training
[Figure]
G M ODEL P ARAMETER S ENSITIVITY
In this paper
[Figure]
we set the hyper-parameters L and d model following the convention of Transformers (Vaswani et al., 2017; Zhou et al., 2021).
Furthermore, to evaluate model parameter sensitivity, we investigate the performance and efﬁciency under different choices for the number of layers L and hidden channels d model .
Generally, increasing the model size can obtain better results but with larger memory and computation costs.
[Table]
[Table]
H P ROTOCOL OF T HRESHOLD S ELECTION
Our paper focuses on unsupervised time series anomaly detection.
Experimentally, each dataset includes training, validation and testing subsets.
Anomalies are only labeled in the testing subset.
Thus
[Table]
we select the hyper-parameters following the Gap Statistic method (Tibshirani et al., 2001) in K-Means .
Here is the selection procedure:
• After the training phase, we apply the model to the validation subset (without label) and obtain the anomaly scores (Equation 6) of all time points.
• We count the frequency of the anomaly scores in the validation subset.
It is observed that the distribution of anomaly scores is separated into two clusters.
We ﬁnd that the cluster with a larger anomaly score contains r time points.
And for our model, r is closed to 0.1%, 0.5%, 1% for SWaT, SMD and other datasets respectively (Table 11).
• Due to the size of the test subset being still inaccessible in real-world applications, we have to ﬁx the threshold as a ﬁxed value δ , which can gaurantee that the anomaly scores of r time points in the validation set are larger than δ and thus detected as anomalies.
[Table]
Note that, directly setting the δ is also feasible.
According to the intervals in Table 11, we can ﬁx the δ as 0.1 for the SMD, MSL and SWaT datasets, 0.01 for the SMAP and PSM datasets, which yield a quite close performance to setting r .
[Table]
In real-world applications
[Table]
the number of selected anomalies is always decided up to human resources.
Under this consideration, setting the number of detected anomalies by the ratio r is more practical and easier to decide according to the available resources.
I M ORE B ASELINES
In addition to the time series anomaly detection methods, the methods for change point detection and time series segmentation can also perform as valuable baselines.
Thus, we also include the BOCPD (Adams & MacKay, 2007) and TS-CP2 (Deldari et al., 2021) from change point detection and UTime (Perslev et al., 2019) from time series segmentation for comparison.
Anomaly Transformer still achieves the best performance.
[Table]