Paper: Time Series Anomaly Detection using Reconstruction and RBF Similarity Scores (ICLR-2024)

Reviewer comment (Experiments):
In review part, only Xu et al. (2021) is reviewed for describing current time-series anomaly detection methods, but several works in 2022 and 2023 are missed in reviewing and experimental parts, which have also performed greatly in multiple time-series anomaly detection datasets, such as:  
[1] Zhang Z, Li W, Ding W, et al. STAD-GAN: unsupervised anomaly detection on multivariate time series with self-training generative adversarial networks[J]. ACM Transactions on Knowledge Discovery from Data, 2023, 17(5): 1-18.  
[2] Xia F, Chen X, Yu S, et al. Coupled Attention Networks for Multivariate Time Series Anomaly Detection[J]. IEEE Transactions on Emerging Topics in Computing, 2023.  
[3] Ding C, Sun S, Zhao J. MST-GAT: A multimodal spatial–temporal graph attention network for time series anomaly detection[J]. Information Fusion, 2023, 89: 527-536.

Author response:
Following your suggestions, we have updated the related research section in the introduction of our paper to include the recent works and baselines. This addition has enriched our paper by providing a more comprehensive overview of the recent methods.

Reviewer comment (Experiments):
The experiments were only conducted on LSTM and Transformer, which are some out-of-date baselines; more experiments on current SOTA methods (as baselines) should be conducted for validating the performance of the proposed module, since the author mentioned that the proposed method is generic and can be applied to a wide range of deep learning architectures. For instance, the Anomaly Transformer can be regarded as a good baseline to testify the proposed RBF-based module.

Author response:
We understand your curiosity about the application of our SimRec method to more specialized anomaly detection models. We address your point in the following details:

- **Rationale Behind the Selected Baselines:** Our study specifically focuses on enhancing the anomaly detection in models based on Reconstruction Error (RE). This focus derives from the identified gap where these models often struggle with detecting subtle anomalies. Our selection of LSTM and Transformer baselines is relevant to the scope of the paper and is driven by their widespread use as RE-based benchmarks in anomaly detection literature. Many papers employ these models as reconstruction-error based benchmarks when comparing new methods. This choice allows us to demonstrate the significant improvements our SimRec method can bring to widely recognized models in the field of RE-based anomaly detection.

- **Exploring Beyond Initial Scope (Potential Application to Specialized Models):** In response to your suggestion, we appreciated the potential value of exploring the application of SimRec to more specialized models like Anomaly Transformer or Omny, despite them operating on different principles such as association discrepancy. This exploration was not part of our original research plan but was undertaken to address your feedback.
  - **Challenges with Implementation:** We encountered challenges while attempting to apply SimRec to the Anomaly Transformer model due to discrepancies between the published paper and the public code. Notably, the model's code includes a "temperature" parameter set to a value of 50, a critical detail that is absent in the original paper. When we adjusted this parameter to a value of 1, more in line with standard practice and implied by the paper, we observed a substantial drop in performance on datasets like MSL. For instance, the F1-score decreased from 0.95 to 0.88, and the AUC from 0.98 to 0.92. This discrepancy highlights the difficulties in replicating and adapting methods when crucial implementation details are omitted in the publication.

    Despite reaching out for clarification, we have not received a response, which makes it hard to properly integrate SimRec with their model. The limited time frame of the rebuttal period and complexities in adapting SimRec due to unclear published core and details constrained our ability to extend our research in this direction. This point brings us back to the choice of our baselines as they were not just arbitrary but a strategic decision to demonstrate the impact of SimRec within a well-recognized and relevant framework in the field.

- **Focused Approach with Openness to Future Research:** We recognize the potential benefits of integrating SimRec with the specialized models, but it should be noted that our research is dedicated to addressing challenges within the pure RE-based anomaly detection domain. We would like to keep the scope of the study in establishing the foundational effectiveness of SimRec in the targeted context of RE score. We believe this focused approach better illustrates the impact of SimRec. Nevertheless, exploring the applicability of SimRec to a broader range of models, including specialized ones, is an exciting direction for future research which can be now opened by our study. We updated the paper accordingly to show this point. 

We hope that the revisions and provided responses address your concerns adequately.

Reviewer comment (Experiments):
There is a lack of baseline. Since neither LSTM nor Transformer are models specialized for anomaly detection, I am curious about the results when applied to more specialized models. In particular, I would like to see the results of applying SimRec to the Anomaly Transformer. In addition, Omni Anomaly

Author response:
Thank you for your insightful feedback. We carefully addressed them point-by-point and tried to make possible revisions accordingly.

### **Q1. Lack of Baseline** (Application of SimRec in other baselines):

We understand your curiosity about the application of our SimRec method to more specialized anomaly detection models. We address your point in the following details:

- **Rationale Behind the Selected Baselines:** Our study specifically focuses on enhancing the anomaly detection in models based on Reconstruction Error (RE). This focus derives from the identified gap where these models often struggle with detecting subtle anomalies. Our selection of LSTM and Transformer baselines is relevant to the scope of the paper and is driven by their widespread use as RE-based benchmarks in anomaly detection literature. Many papers employ these models as reconstruction-error based benchmarks when comparing new methods. This choice allows us to show the significant improvements our method can bring to widely recognized models in the field of RE-based anomaly detection.

- **Exploring Beyond Initial Scope (Potential Application to Specialized Models):** In response to your suggestion, we appreciated the potential value of exploring the application of SimRec to more specialized models like Anomaly Transformer or Omny, despite them operating on different principles such as association discrepancy. This exploration was not part of our original research plan but was undertaken to address your feedback.

  - **Challenges with Implementation:** We faced challenges while trying to apply SimRec to the Anomaly Transformer model due to discrepancies between the published paper and the public code. Notably, the model's code includes a "temperature" parameter set to a value of 50, a critical detail that is absent in the original paper. When we adjusted this parameter to a value of 1, more in line with standard practice and implied by the paper, we observed a substantial drop in performance on datasets like MSL. For instance, the F1-score decreased from 0.95 to 0.88, or the AUC from 0.98 to 0.92. This discrepancy highlights the difficulties in replicating and adapting methods when crucial implementation details are omitted in the publication.

    Despite reaching out for clarification, we have not received a response, which makes it hard to properly integrate SimRec with their model. The limited time frame of the rebuttal period and complexities in adapting SimRec due to unclear published core and details constrained our ability to extend our research in this direction. This point brings us back to the choice of our baselines as they were not just arbitrary but a strategic decision to show the impact of SimRec within a well-recognized and relevant framework in the field.

- **Focused Approach with Openness to Future Research:** We recognize the potential benefits of integrating SimRec with the specialized models, but it should be noted that our research is dedicated to addressing challenges within the pure RE-based anomaly detection domain. We would like to keep the scope of the study in establishing the foundational effectiveness of SimRec in the targeted context of the RE score. We believe this focused approach better illustrates the impact of SimRec. Nevertheless, exploring the applicability of SimRec to a broader range of models, including specialized ones, is an exciting direction for future research which can be now opened by our study. We updated the paper accordingly to show this point.

Reviewer comment (Evaluation):
The effect of SimRec is clearly visible, but it seems to be more affected by threshold selection. I would like to see the reconstruction error graph of the anomaly transformer using the same threshold selection method.

Author response:
### **Q2. Effect of the Threshold**

- **Threshold Selection Methodology:** In our research, the threshold for anomaly detection was carefully chosen based on the well-accepted method used in the literature, specifically referencing the approach used in the Anomaly Transformer paper.

- **Use of ROC-AUC and AUC-PR:** To provide a more comprehensive evaluation that is not solely dependent on threshold selection, we have employed ROC-AUC and AUC-PR metrics which offer a threshold-independent assessment and a general view of the effect of our method.

- **Incorporation of New Evaluation Metrics:** Following the suggestion of reviewer \#GeHM, we have added new evaluation metrics to our paper that are not dependent on threshold selection. These additional results align with our initial findings and support the claims of our study. Results can be found in Table 1 in the updated paper.

- **Consistency Across Comparisons:** We have ensured that the comparison of our method with other models is fair by using the same threshold and settings. Also, to provide a robust evaluation, we have reported the mean performance over multiple runs in all our experiments. We kept this practice not only for our model but also in comparisons with other methods.

Reviewer comment (Experiments):
The effect of SimRec is clearly visible, but it seems to be more affected by threshold selection. I would like to see the reconstruction error graph of the anomaly transformer using the same threshold selection method.

Author response:
### **Q2. Effect of the Threshold**

- **Threshold Selection Methodology:** In our research, the threshold for anomaly detection was carefully chosen based on the well-accepted method used in the literature, specifically referencing the approach used in the Anomaly Transformer paper.

- **Use of ROC-AUC and AUC-PR:** To provide a more comprehensive evaluation that is not solely dependent on threshold selection, we have employed ROC-AUC and AUC-PR metrics which offer a threshold-independent assessment and a general view of the effect of our method.

- **Incorporation of New Evaluation Metrics:** Following the suggestion of reviewer \#GeHM, we have added new evaluation metrics to our paper that are not dependent on threshold selection. These additional results align with our initial findings and support the claims of our study. Results can be found in Table 1 in the updated paper.

- **Consistency Across Comparisons:** We have ensured that the comparison of our method with other models is fair by using the same threshold and settings. Also, to provide a robust evaluation, we have reported the mean performance over multiple runs in all our experiments. We kept this practice not only for our model but also in comparisons with other methods.

Reviewer comment (Novelty):
Novelty is somewhat low. The idea seems technically sound as sub-components already exist, but the technical novelty is somewhat low. For example, there is a lack of preliminaries to help understand if something is brand new, and several of the proposed methods are simple adaptations of original equations applied to this new problem (e.g., k-means-type optimization in equation 3, RBF kernel already exists, combination of two scores is trivial). The core idea of using similarity to centroids has been applied many times in previous work, perhaps not explicitly combined with the reconstruction error.

Author response:
### **Q1. Novelty is somewhat low:**

We appreciate the opportunity to further clarify and underscore the contributions of our work, considering your concerns regarding the novelty of our methodology:

- **Identification of a New Problem (Addressing a Specific Gap):** Our research has identified and addressed a previously unexplored challenge in anomaly detection models based on reconstruction error — specifically, their limited effectiveness in detecting subtle anomalies. Uncovering this issue is an important contribution to the field.

- **Innovative Application of Existing Methods:** We agree with your point regarding the established nature of methods like the RBF, and we have already acknowledged this in the introduction of our paper. However, one of the novelties of our work lies in the specific application of these methods to address a specific problem. The combination of the RBF similarity score with traditional reconstruction error, while conceptually straightforward, is a strategic choice. This is not just a trivial combination but a carefully considered approach to enhance the sensitivity and robustness of anomaly detection for subtle anomalies (addressing the gap).

    As you mentioned, this combination has not been explicitly explored in the literature, and our experimental results (provided by detailed comparative analysis) validate the effectiveness of our approach in addressing the identified gap.

    We have updated the paper to elaborate more on how the combination of RBF similarity score and reconstruction error is not trivial and it has specific advantages - This claim is now even more supported by empirical results (using the new evaluation metrics suggested by you).

- **Innovative Density Loss Term:** A key innovation in our approach is the introduction of a novel density loss term in the loss function, specifically designed for training the RBF kernel in this context. This term is detailed in the context of Formula 4 in our paper. We present the term here for reference:

    $$
    \text{Density Loss} = \lambda \frac{1}{N} \sum_{i=1}^{N} \frac{1}{T} \sum_{t=1}^{T} \log \left( \frac{1}{M} \sum_{m=1}^{M} (\mathbf{z}_{i,t})_m + \epsilon \right)
    $$

    This is a substantial technical contribution that enhances the effectiveness of our method and is not found in existing literature (as far as we were able to find).

- **Efficiency of Practical Implications:** We demonstrate that next to using the novel density loss term, random initialization of the RBF parameters is as effective as more complex well-established initialization methods like K-means. This finding has significant practical implications, making our method more accessible and easier to implement in various settings.

    &nbsp;

In general, we have updated the paper to give a better overview of the contribution and novelty of the work. We hope that these points, along with the additional details and empirical evidence provided in the revised paper, address your concerns regarding the novelty of our work.

&nbsp;

Reviewer comment (Reproducibility):
Unclear experimental settings. The settings and parameters for baselines are not clear, raising questions about fairness in comparisons when details of the experimental settings are omitted.

Author response:
### **Q2. Unclear experimental settings:**

We would like to clarify that detailed information about the settings and parameters is comprehensively provided in the appendix section of our paper. Additionally, we have made the full code of our experiments available to ensure transparency and facilitate reproducibility.

To address your concern, we have now updated the key experimental details regarding the primary parameters, model architectures, and settings with a better overview of our experimental approach.

We believe these updates will enhance the clarity of our experimental framework and ensure that our comparisons are both fair and easily replicable by other researchers in the field. We hope that these modifications address your concerns effectively.

Reviewer comment (Experiments):
Missing related work, datasets, and new benchmarks. The work focuses on a very narrow part of the literature (emerging due to the rise of DNN solutions) but omits recent progress on benchmarks, baselines, and new evaluation measures. For example, new benchmarks include ~20 datasets, ~2000 time series, and 10+ baselines ([a] TSB-UAD: an end-to-end benchmark suite for univariate time-series anomaly detection. Proceedings of the VLDB Endowment 15.8 (2022): 1697-1711). New evaluation measures include [b] "Volume under the surface: a new accuracy evaluation measure for time-series anomaly detection." Proceedings of the VLDB Endowment 15.11 (2022): 2774-2787.

Author response:
### **Q3. Missing related work, datasets, new benchmarks**

- **Incorporation of New Evaluation Metrics:** Following your recommendation, we have incorporated the new evaluation metric "Volume under the surface" from the referenced work [b]. This inclusion has not only validated our initial findings but has also strengthened the overall results and conclusions of our paper. We have updated our manuscript with the new results showcasing these results, demonstrating the added robustness and validity that this metric brings to our research. Your suggestion has significantly contributed to enhancing the quality of our work. (Due to space limitations, we only provide the new metrics here. See Table 1 from the updated paper for the complete results)

**Table 1**. Mean performance (std). Initialization methods are denoted as (R) for Random and (K) for K-means.

| Models                  | (SMD) VUS-ROC | | (SMD) VUS-PR | | (MSL) VUS-ROC | | (MSL) VUS-PR | | (PSM) VUS-ROC | | (PSM) VUS-PR |
|-------------------------|---------------|-|--------------|-|---------------|-|--------------|-|---------------|-|--------------|
| LSTM                    | 61(0)         | | 55(0)        | | 72(1)         | | 61(0)        | | 72(1)         | | 61(0)        |
| SimRec-LSTM (R)         | **65(2)**     | | **60(3)**    | | **78(1)**     | | **65(2)**    | | **78(1)**     | | **65(2)**    |
| SimRec-LSTM (K)         | **66(2)**     | | **61(2)**    | | **80(0)**     | | **66(2)**    | | **80(0)**     | | **66(2)**    |
|                         |               | |              | |               | |              | |               | |              |  <!-- This is the empty row -->
| Transformer             | 59(2)         | | 52(3)        | | 70(2)         | | 59(2)        | | 70(2)         | | 59(2)        |
| SimRec-Transformer (R)  | **64(3)**     | | **57(3)**    | | **80(1)**     | | **64(3)**    | | **80(1)**     | | **64(3)**    |
| SimRec-Transformer (K)  | **66(2)**     | | **61(4)**    | | **79(2)**     | | **66(2)**    | | **79(2)**     | | **66(2)**    |

- **Benchmark Recommendations:** We understand your point regarding the inclusion of more diverse benchmarks, such as the TSB-UAD suite for univariate time-series anomaly detection. While we acknowledge the value of such comprehensive datasets, our research has targeted more challenging datasets. This is why we selected the multivariate time series datasets which are widely acknowledged and used in this field. We believe that focusing on these datasets allows us to show the effectiveness of our method in more complex, real-world scenarios. Other research has shown that simpler models can often effectively address anomaly detection in univariate datasets (refer to [a1]). However, our work aims to tackle the subtleties and complexities inherent in more challenging benchmarks. We appreciate your suggestion and will consider exploring a broader range of datasets, including univariate ones, in our future works.

- **Expansion of Related Work:** We have updated the related research section in the introduction of our paper to include recent works and baselines in the field of anomaly detection. This addition ensures that our paper provides a comprehensive overview of the recent methods.

- **Clarification on the Scope of the paper:** While acknowledging the variety of innovative approaches in anomaly detection, our research specifically focuses on addressing issues within Reconstruction Error (RE) based methods. Therefore, we have chosen baselines (for applying SimRec score) that are directly relevant to this focus. For instance, STAD-GAN [a2] is based on reconstruction error anomaly score, but a baseline like OmniAnomaly [a3], although effective, diverge from our core research area as they are based on different principles such as reconstruction probability, distance-based methods, or association discrepancy.

    We believe that our focus on enhancing RE-based anomaly detection is a significant contribution to the field (both fundamentally and in terms of competitive performance compared to the recent SOTA method). Also, by identifying and addressing a specific limitation in these methods, our work opens up new opportunities for future research. For example, the integration of RBF kernels, as proposed in our method, could potentially benefit other baseline models.

References:
- [a1] Schmidl, Sebastian, et al. "Anomaly detection in time series: a comprehensive evaluation." Proceedings of the VLDB Endowment 15.9 (2022): 1779-1797.
- [a2] Zhang, Zhijie, et al. "STAD-GAN: Unsupervised anomaly detection on multivariate time series with self-training generative adversarial networks." ACM Transactions on Knowledge Discovery from Data 17.5 (2023): 1–18.
- [a3] Su, Ya, et al. "Robust anomaly detection for multivariate time series through stochastic recurrent neural network." Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019.