From: ~Wei-Cheng_Chang1
comment: We thank the reviewer for the constructive comments.

1) Indeed, optimizing Eq (3) involves a choice of sample size (window size). Ideally, in the offline detection which we have the entire data set, we could set the window size to be reasonably large to have a stronger statistical guarantee. For online real-time detection, we may not have the luxury to have a very large window size as it delays the real-time detection. Thus, we set the window size to be {25, 50, 75} for the real-time detection setting. 

2) It is true that tuning hyper-parameter via the availability of anomaly labels on validation could be non-universal in some applications. We would like to highlight that our method has a stopping condition that is irrelevant to the anomaly labels but solely based on the MMD distance between real distribution P and the generated fake distribution G_theta. With this stopping condition on the validation set, we can also tune the hyper-parameter lambda and beta. We will show the updated results on the revised version. 

3) We try our best to compare the proposed method with state-of-the-art methods with publicly available implementation.  As you mentioned, some methods have low AUC, even <0.5 on some datasets. It could due to two reasons. First, we basically tune one major hyper-parameter and set the rest as default in their implementation. Take RDR-KCPD as an example, the kernel bandwidth could have some impact on its performance. Second, it could be that some methods easily overfit on the training set with early stopping on the validation set. For example, LSTNet has 0.9127 AUC on Fishkiller while only 0.5077 AUC on HASC. 
title: Response to Reviewer3 comments and questions

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper42/AnonReviewer3
relevance: 4: high
impact: 4: high
novelty: 4: high

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper42/AnonReviewer3
evaluation: This paper presents a change-point detection methodology based on maximizing test statistics power by kernel selection. Overall, the paper is very well written. The description of the method is clear and structured well, making it easy to follow.

Several comments/questions:
1) In Eq 3., the authors present a Type I error regularization term based on an empirical estimator, which is claimed to bound the Type I error. However, this claim is correct only in the limit, making the practical implementation depend on sample size (window size in this implementation). How would you choose sample/window size to maintain the above bound?

2) The choice of hyperparameters (e.g. lambda) and kernel-set (or kernel parameters limits) is critical for a successful optimization. In this paper, these parameters are found during validation. However, this is not applicable for many time series (black box) problems. 

3) Results - AUC results for some datasets/methods are ~0.5 (and even <0.5). Are these even suitable methods for comparison or are there any issues with their implementation?