Paper: ChronoGAM: An End-to-End One-Class Time Series Gaussian Mixture Model (ICLR-2024)

Reviewer comment (Evaluation):
This paper introduces a convolutional approach to time series modeling with reference (maybe, as a standard method described in [Wang+2017]) to existing research on time series classification problems. It seems to me that the strengths of this approach (i.e., for what time series can well-fitted features be captured) and the weaknesses (what time series are difficult to represent) have not been adequately discussed.

Author response:
**“Is the adoption of the convolution approach the recommended setting by the authors? Or is it adopted because it is known as the most basic and standard in the field concerned? For example, if the user has prior knowledge of the time series of interest, is it easy to change the proposed method to a time series model specific to that interest (in this context, the design of the AE layer)?”**

As we carried out extensive testing on different data sets generated and transformed to adapt to a One-Class Learning problem, we adopted an autoencoder structure capable of generalizing some information from different sets without major problems. Therefore, the proposed architecture, its layers, and parameters are a general recommendation that can be changed in different sets, changing the structure of the autoencoder and parameters such as the number of layers, filters, and size of the applied kernels. Activation is a recommendation given in the article [1], which presents a theorem about neural networks and the ability to reconstruct periodic patterns, so we adopted it to preserve the periodicity of the series without giving up other advantages.

[1] Ziyin, Liu, Tilman Hartwig, and Masahito Ueda. "Neural networks fail to learn periodic functions and how to fix it." Advances in Neural Information Processing Systems 33 (2020): 1583-1594.

**“Is this a robust setting for local (near temporal) features of the time series of interest, as the standard convolution assumes? Or is it also possible to capture very distant temporal dependencies (in the extreme case, where the first frame determines the last frame) or periodic structures?”**

We evaluated the capacity of the model through the different sizes of the series available in the data sets, see Table 3. It is possible to observe that the performance is maintained for time series of up to 1000 observations, being impaired in series with a size greater than that or in series very small. Once this is observed, one of its causes is the size of the kernels applied during the convolutions. Note that optimizing different kernel sizes to find the best solution for each of the 652 generated datasets would result in an unfeasible experiment load. Therefore, we opted for a model with a generalist kernel size that obtains good performance for most solutions. However, this does not prevent other works from exploring these parameters to increasingly improve results in very large time series or treat this type of situation with other structures capable of capturing these long-distance dependencies.

Reviewer comment (Experiments):
Another major issue is that the paper does not demonstrate clearly the relevance of these modifications. The authors should conduct an ablation study in order to show how the implemented modifications impact the obtained results, including convolution layers of Fawaz et al. (2019) and the snake activation of Ziyin et al. (2020).

Author response:
**“Why there is no ablation study that allows to demonstrate the relevance of the proposed modifications ?”**

We included a more in-depth study of the results in the appendices. However, due to limited space, we focused on the main points during the analysis of the results in the paper. We show and compare the approach using the new proposed loss function and without using it (adapting only the autoencoder) in Figure 2 (CDD).

We also raised some analyses about the sets' characteristics and the autoencoder's performance in capturing temporal dependencies. To see the relevance of these results, simply compare ChronoGAM (wo regularizer) in Figure 2 with DAGMM, which implements the same loss function and differs in the autoencoder. The gain shows that adapting the model structures to work with temporal dependencies is enough to surpass classic Machine Learning models. However, it is still unable to perform without a significant statistical difference when we analyze it together with the complete ChornoGAM, which implements the new loss function.

Reviewer comment (Experiments):
Why is there no ablation study that allows to demonstrate the relevance of the proposed modifications?

Author response:
**“Why there is no ablation study that allows to demonstrate the relevance of the proposed modifications ?”**

We included a more in-depth study of the results in the appendices. However, due to limited space, we focused on the main points during the analysis of the results in the paper. We show and compare the approach using the new proposed loss function and without using it (adapting only the autoencoder) in Figure 2 (CDD).

We also raised some analyses about the sets' characteristics and the autoencoder's performance in capturing temporal dependencies. To see the relevance of these results, simply compare ChronoGAM (wo regularizer) in Figure 2 with DAGMM, which implements the same loss function and differs in the autoencoder. The gain shows that adapting the model structures to work with temporal dependencies is enough to surpass classic Machine Learning models. However, it is still unable to perform without a significant statistical difference when we analyze it together with the complete ChornoGAM, which implements the new loss function.

Reviewer comment (Evaluation):
Limited concrete evidence for how the numerical challenges in One-Class-Learning are released.

Author response:
Thank you for the reviews and pointed questions. Based on the demonstration of the problems found in (Ruff et al., 2018), we inserted an appendix to highlight the numerical challenges that involve the use of hyper-sphere loss, including the n-ball problem [2], as well as the experimental results that highlight the numerical instability of the DAGMM in Figure 3.

Reviewer comment (Theory):
By using an autoencoder, the numerical problems are resolved but with limited theoretical justification, and no ablation study is presented.

Author response:
The problems of numerical instability were solved by inserting a regularizing term of the covariance matrix different from that proposed in DAGMM, see the final part of Equation 4, as well as equations 2 and 3 that insert the use of the proposed Explu function.

Reviewer comment (Evaluation):
The impact of adopting an autoencoder is welcomed but requires concrete examples and evidence regarding what temporal and structural features are well captured.

Author response:
“what temporal and structural features are well captured.”

Unlike a simple MLP network, a convolutional filter can be used to obtain results for all timestamps of a time series, allowing the learning of invariant filters along the temporal dimension. These CNN networks allow the learning of these filters dynamically, optimizing the parameters for each data set used and learning filters that make it easy to discriminate between classes [1].

See in Figures 4 and 5 that this type of network optimizes the generation of more discernible embeddings between time series instances, making the representations more robust for the method.

[1] Ismail Fawaz, Hassan, et al. "Deep learning for time series classification: a review." Data mining and knowledge discovery 33.4 (2019): 917-963.

Reviewer comment (Evaluation):
The author claims the gain of the proposed method in 369 datasets out of a total of 652. The issue with this massive test is the lack of insights into the reasons for the proposed method's effectiveness and ineffectiveness on these many datasets.

Author response:
Firstly, we would like to thank you for the review and the suggestions. We will try to resolve as many of them as possible below to improve the quality and contributions of the paper.

We modified the discussion of the results to clarify discussions of why ChronoGAM (the proposed method) outperforms other methods.

“(...) Unlike traditional machine learning methods, such as IsolationForest and OCSVM, which interpret observations as attributes without temporal dependence, ChronoGAM uses convolutions to capture this dependence and thus obtains better results. Compared to its competitors that use deep learning strategies, DeepSVDD suffers from the problem of hypersphere collapse, leading to poor results which ChronoGAM easily overcomes due to not using a hypersphere loss.

ChronoGAM has been more successful than its main competitor (DAGMM) for two primary reasons. Firstly, its autoencoder architecture is more flexible, using convolutional structures that can detect temporal correlations in the data, as demonstrated in various studies on time series classification (Fawaz et al, 2019, Wang et al, 2017, Zong et al, 2018). In contrast, DAGMM only uses an MLP-based autoencoder, which cannot capture these dependencies. The second reason is the covariance matrix regularizer, while the DAGMM uses a 

$p=\sum^K_{k=1}\sum^d_{j=1}\frac{1}{\mathbf{\hat {\Sigma}}_{kjj}}$

which can easily cause very large values, making it difficult to correctly propagate the error to the task, leading to numerical instabilities and exploding gradients, ChronoGAM uses the penalty $p=\text{min} (\sum^K_{k=1}\sum^d_{j=1}-\log(\mathbf{\hat{\Sigma}}_{kjj}), \alpha)$ where $\alpha = 100$ , so values that would cause the gradient to explode and cause numerical instabilities as well as disregarding the other tasks that make up the objective function to simply minimize the covariance matrix are more easily avoided. (...)”

We emphasize that the innovation of the proposed method lies in the new loss function used, proposing the explu to avoid very high values as occur in the exp, in addition to the motivation of the term that will regularize the covariance matrix that is used by the method (avoiding division to extremely low values that causes the exploding gradients problem). As for the results, the detailed results of each dataset, label, and metric will be made available in the project repository. To clarify this issue, we modify the following sentence:

“(...) The experimental evaluation codes and results are publicly available1. (...)”
	
And later the link will be updated to the project's public repository (link omitted for double-blind purposes).

Reviewer comment (Novelty):
The proposed method appears to have no significant technical novelty; thus, it is a question of where the outperformance stems from. The authors are expected to provide a conceptual or theoretical explanation of why the proposed method can outperform in addition to experimental results.

Author response:
Firstly, we would like to thank you for the review and the suggestions. We will try to resolve as many of them as possible below to improve the quality and contributions of the paper.

We modified the discussion of the results to clarify discussions of why ChronoGAM (the proposed method) outperforms other methods.

“(...) Unlike traditional machine learning methods, such as IsolationForest and OCSVM, which interpret observations as attributes without temporal dependence, ChronoGAM uses convolutions to capture this dependence and thus obtains better results. Compared to its competitors that use deep learning strategies, DeepSVDD suffers from the problem of hypersphere collapse, leading to poor results which ChronoGAM easily overcomes due to not using a hypersphere loss.

ChronoGAM has been more successful than its main competitor (DAGMM) for two primary reasons. Firstly, its autoencoder architecture is more flexible, using convolutional structures that can detect temporal correlations in the data, as demonstrated in various studies on time series classification (Fawaz et al, 2019, Wang et al, 2017, Zong et al, 2018). In contrast, DAGMM only uses an MLP-based autoencoder, which cannot capture these dependencies. The second reason is the covariance matrix regularizer, while the DAGMM uses a 

$p=\sum^K_{k=1}\sum^d_{j=1}\frac{1}{\mathbf{\hat {\Sigma}}_{kjj}}$

which can easily cause very large values, making it difficult to correctly propagate the error to the task, leading to numerical instabilities and exploding gradients, ChronoGAM uses the penalty $p=\text{min} (\sum^K_{k=1}\sum^d_{j=1}-\log(\mathbf{\hat{\Sigma}}_{kjj}), \alpha)$ where $\alpha = 100$ , so values that would cause the gradient to explode and cause numerical instabilities as well as disregarding the other tasks that make up the objective function to simply minimize the covariance matrix are more easily avoided. (...)”

We emphasize that the innovation of the proposed method lies in the new loss function used, proposing the explu to avoid very high values as occur in the exp, in addition to the motivation of the term that will regularize the covariance matrix that is used by the method (avoiding division to extremely low values that causes the exploding gradients problem). As for the results, the detailed results of each dataset, label, and metric will be made available in the project repository. To clarify this issue, we modify the following sentence:

“(...) The experimental evaluation codes and results are publicly available1. (...)”
	
And later the link will be updated to the project's public repository (link omitted for double-blind purposes).