Kernel Change-point Detection with Auxiliary Deep Generative Models
Abstract
Detecting the emergence of abrupt property changes in time series is a challenging problem.
Kernel two-sample test has been studied for this task making fewer assumptions on the distributions than traditional parametric approaches.
However, selecting kernels is non-trivial in practice.
Although kernel selection for two-sample test has been studied, the insufﬁcient samples in change point detection (CPD) problem hinders the success of those developed kernel selection algorithms.
In this paper, we propose KL-CPD , a novel kernel learning framework that optimizes a lower bound of test power via an auxiliary generative model.
With deep kernel parameterization, KL-CPD endows kernel two-sample test with the data-driven kernel to detect different types of change-points in real-world applications.
The proposed approach signiﬁcantly outperformed other state-of-the-art methods in our comparative evaluation of benchmark datasets.
Introduction
Detecting changes in the temporal evolution of a system in time series analysis has attracted considerable attention in machine learning decades [4, 7].
In this work, we study the retrospective change-point detection (CPD) problem [29, 21], which allows a ﬂexible time window to react on the change-points.
Retrospective CPD not only enjoys robust detection [9] but embraces many real-world applications [26, 31, 36].
Albeit being developed for many years [16], many works are parametric with strong assumptions on the distributions [4, 15], including auto-regressive models [35] and state-space models [18] for tracking changes in various statistics.
On the other hand, kernel two-sample test has been applied to time series CPD that makes fewer assumptions on the distributions (e g. [17, 21]).
The performance of kernel methods, nevertheless, relies heavily on the choice of kernels.
[12, 13] conducted kernel selection for RBF kernel bandwidths via median heuristic.
While certainly straightforward, it has no statistical guarantees regarding to the test power of hypothesis testing.
[14] show explicitly optimizing the test power leads to better kernel choice for hypothesis testing under mild conditions.
Kernel selection by optimizing the test power, however, is not directly applicable for time series CPD due to insufﬁcient samples.
In this paper, we propose KL-CPD , a kernel learning framework for time series CPD, highlighting three contributions:
In Section 2, we discuss the inaptness of existing kernel learning approaches in a simulated example.
We then propose to optimize a lower bound of the test power via an auxiliary generative model, serving as a surrogate of the abnormal events.
In Section 3, we present a deep kernel parametrization of our framework, which endows a data-driven kernel for the kernel two-sample test.
KL-CPD induces composition kernels by combining RNNs and RBF kernels that are suitable for the time series applications.
In Section 4, we conduct extensive benchmark evaluation showing the outstanding performance of KL-CPD in real-world CPD applications.
Optimizing Test Power
Maximum mean Discrepancy (MMD) is a nonparametric probabilistic distance commonly used in two-sample-test [12, 13].
Given a kernel k , the MMD distance between two distributions P and
Q is M k ( P , Q ) :
= E P [ k ( x, x 0 )]  2 E P , Q [ k ( x, y )] + E Q [ k ( y, y 0 )] .
In practice, with ﬁnite samples X = { x 1 , . . .
, x m } ⇠ P and Y = { y 1 , . . .
, y m } ⇠ Q , we estimate M k
( P , Q ) with an unbiased estimator ˆ M k
( X, Y ) :
= 1 ( m 2 )
P i 6 = i 0 k ( x i , x i 0 )  2 m
P i,j k ( x i , y j ) + 1 ( m 2 )
P j 6 = j 0 k ( y j , y j 0 ) .
For any characteristic kernel k , M k ( P , Q ) =
iff P = Q .
However, the estimator ˆ M k
( X, X 0 ) may not beeven though X, X
⇠ P due to ﬁnite sample size.
Hypothesis test instead offers thorough statistical guarantees of whether two ﬁnite sample sets are the same distribution.
Following [13], the hypothesis test is deﬁned by the null hypothesis H 0 : P = Q and alternative H 1 : P 6 = Q , using test statistic m ˆ M k
( X, Y ) .
For a given allowable false rejection probability ↵ (i.e., Type I error), we choose a test threshold c ↵ and reject H
if m ˆ M k
( X, Y ) >
c ↵ .
The kernel selection objective to maximize the test power [14, 28] is presented as follows.
Under the alternative H 1 : P 6 = Q , ˆ M k is asymptotically normal with V m ( P , Q ) denoting the asymptotic variance.
The test power is then
[Equation]
where  is the CDF of the standard normal distribution.
Given a set of kernels K , We aim to choose a kernel k 2 K to maximize the test power, which is equivalent to maximizing the argument of  .
In time series CPD, we denote P as the distribution of usual events and Q as the distribution for the event when change-points happen.
The difﬁculty of choosing kernels via optimizing test power in
Eq.
(1) is that we have very limited samples from the abnormal distribution Q .
Kernel learning in this case may easily overﬁt, leading to sub-optimal performance in time series CPD.
2.1 Difﬁculties of Optimizing Kernels for CPD
To demonstrate how limited samples of Q would affect optimizing test power, we consider kernel selection for Gaussian RBF kernels on the Blobs dataset [14, 28], which is considered hard for kernel two-sample test.
P is a 5 ⇥grid of two-dimensional standard normals.
Q is laid out identically, but with covariance ✏ q  1 ✏ q +1 between the coordinates.
Right ﬁgure shows X ⇠ P (red samples), Y ⇠ Q (blue dense samples), ˜ Y ⇠ Q (blue sparse samples) with ✏ q = 6 .
Note that when ✏ q = 1 , P = Q .
For ✏ q 2 { 4 , 6 , 8 , 10 , 12 , 14 } , we take 10 K samples for X, Y and 200 samples for ˜ Y .
We choose kernels by: 1) median heuristic ; 2) max-ratio ⌘ k ⇤ ( X, Y ) =
arg max k ˆ M k
( X, Y ) / p
V m ( X, Y ) ; among 20 kernel bandwidths.
We repeat this process 1000 times and report the test power under false rejection rate ↵ = 0 .
As shown in Fig.
1, optimizing kernels using limited samples ˜
Y signiﬁcantly decreases the test power compared to
Y (blue curve down to the cyan curve).
This result not only veriﬁes our claim on the inaptness of existing kernel learning objectives for CPD task, but stimulates us with the following question, How to optimize kernels with very limited samples from Q , even none in an extreme?
2.2 A Practical Lower Bound on Optimizing Test Power
We ﬁrst assume there exist a surrogate distribution G that we can easily draw samples from ( Z ⇠ G , | Z |  | ˜ Y | ), and also satisﬁes the following property:
[Equation]
Besides, we assume dealing with non trivial case of P and Q where a lower bound 1 m v l 
V m,k
( P , Q ) , 8 k exists.
Since M k ( P , Q ) is bounded, there exists an upper bound v u .
With bounded variance v l m  V m,k ( P , Q )  v u m condition, we derive an lower bound  k ⇤ ( P , G ) of the test power max k 2 K
[Equation]
Just for now in the blob toy experiment, we artifact this distribution G by mimicking Q with the covariance ✏ g = ✏ q  2 .
We defer the discussion on how to ﬁnd G in the later subsection 2.3.
Choosing kernels via  k ⇤ ( X, Z ) using surrogate samples Z ⇠ G , as represented by the green curve in Fig.
1, substantially boosts the test power compared to ⌘ k ⇤ ( X, ˜ Y ) with sparse samples
˜ Y ⇠ Q .
Test Threshold Approximation Under H 0 : P = Q , m ˆ M k
( X, Y ) converges asymptotically to an unknown distribution depending on P [13, Theorem 12], yielding a non closed form test threshold c ↵ .
Even estimating c ↵ with permutation test or some kernel approximated distributions, it is difﬁcult to optimize c ↵ because it is a function of k and P .
Alternatively, since c ↵ is a function of ˆ M k ( X, X 0 ) that controls the Type I error, bounding ˆ M k ( X, X 0 ) could be an approximation of bounding c ↵ .
Therefore, we propose the following objective that maximizing a lower bound of test power
[Equation]
where  is a hyper-parameter to control the trade-off between Type-I and Type-II errors.
Note that the optimization of Eq.
(3) is solved using the unbiased estimator of M k ( P , G ) with empirical samples.
2.3 Surrogate Distributions using Generative Models
The remaining question is how to construct the surrogate distribution G .
As no prior knowledge nor empirical samples of Q , to ensure (2) holds for any possible Q (e.g. Q 6 = P but Q ⇡ P ), intuitively, we have to make G as closed to P as possible.
We propose to learn an auxiliary generative model
G ✓ parameterized by ✓ such that ˆ M k
( X, X 0 ) <
min ✓ M k ( P , G ✓ ) <
M k ( P , Q ) , 8 k 2 K .
To ensure the ﬁrst inequality hold, we set early stopping criterion when solving G ✓ in practice.
Moreover, the limited capacity of G ✓ (e.g. small neural networks) [3] and ﬁnite samples of P hinder us to fully recover P .
Thus, we result in a min-max formulation to consider all possible k 2 K when we learn G ,
[Equation]
and solve the kernel for the hypothesis test in the mean time.
Lastly, we remark that although the resulted objective (4) is similar to [20], the motivation and explanation are different .
One major difference is we aim to ﬁnd k with highest test power while their goal is ﬁnding G ✓ to approximate P .
KLCPD: Realization for Time Series Applications
To have a more expressive kernel for complex time series, we consider compositional kernels K = n
[Equation]
o .
The resulted kernel ˜ k is still characteristic if f is an injective function and k is characteristic [13].
Inspired by the recent success of combining deep neural networks into kernels [32, 1, 20], we parameterize f  by RNNs to capture the temporal dynamics of time series.
For an injective function f , there exists a function F such that F ( f ( x ))
= x, 8 x 2 X .
A practical realization of f is a RNN encoder parametrized by  while the function F is a RNN decoder parametrized by trained to minimize the reconstruction loss.
Thus, our ﬁnal objective is
[Equation]
Practical Implementation We consider two consecutive windows in mini-batch to estimate ˆ M f
X, X in an online fashion for efﬁciency.
The sample X ⇠ P is divided into the left window segment X ( l ) =
{ x t  w , . . .
, x t  1 } and the right window segment X ( r ) = { x t , . . .
, x t + w  1 } such that X =
{ X ( l ) , X
( r ) } .
We present an realization of KL-CPD in Algorithm 1 with the weight-clipping technique, where the generator g ✓ is also a Seq2Seq model aims at conditional generation.
The stopping condition is based on a maximum number of epochs or the detecting power of M f
P , G ✓ )  ✏ .
This ensure the surrogate G ✓ is not too close to P , as motivated in Sec.
input : learning rate ↵ , clipping range c , window size w , n c kernel learning update per iter while M k  f  ( P , G ✓ ) > ✏ do for t = 1 , . . .
, n c do
Sample a minibatch X t ⇠ P , denote X t = { X ( l ) t , X ( r ) t } , and !
⇠ P ( ⌦ ) gradient (  ) r
M k  f
P , G ✓
ˆ M k
f
X ( l ) t , X ( r ) t    E ⌫ ⇠ P [ G ✓ k ⌫  F  f  ( ⌫ )  k 22   + ↵ · RMSProp (  , gradient (  ))
clip (  ,  c, c )
Sample a minibatch X t
⇠ P , denote X t 0 = { X ( l ) t 0 , X ( r ) t 0 } , and !
⇠ P ( ⌦ ) gradient ( ✓ ) r ✓ M k  f
P , G ✓  ✓ ✓  ↵ · Adam ( ✓ , gradient ( ✓ ))
Experiment Results
We compare the proposed KL-CPD with seven representative baselines on benchmark datasets from real-world applications of CPD, including Bee-Dance [30], Fishkiller [30], HASC [23] and
Yahoo [34].
Detailed data description are available in Appendix B.1.
Following [19, 27, 23], the datasets are split into training/validation/test set by 60% , 20% , 20% ratio in chronological order.
Note that training is fully unsupervised for all methods while labels in the validation set are used for hyperparameters tuning.
We consider AUC under the ROC curves as the evaluation metric, which is commonly used in CPD literature [21, 23, 33].
We compare KL-CPD with real-time CPD methods ( ARMA , ARGP , RNN , LSTNet ) and retrospective CPD methods ( ARGP-BOCPD , RDR-KCPD , Mstats-KCPD ).
Details are in Appendix B.2.
Note that OPT-MMD is a deep kernel learning baseline which optimizes MMD by treating past samples as P and the current window as Q (insufﬁcient samples).
[Table]
KL-CPD shows signiﬁcant gain over the other mostly, except being in a second place on the
Yahoo dataset, with 2% lower AUC compared to the leading ARGP .
This conﬁrms the importance of data-driven kernel selection and effectiveness of our kernel learning framework.
Notice that OPTMMD performs not so good compared to KL-CPD , which again veriﬁes our simulated example in
Sec. 2 that directly applying existing kernel learning approaches with insufﬁcient samples may not be suitable for real-world CPD task.
Conclusion
We propose KL-CPD , a new kernel learning framework for two-sample test by optimizing a lower bound of test power with a auxiliary generator, to resolve the issue of insufﬁcient samples in changepoints detection.
The deep kernel parametrization of KL-CPD combines RNNs with RBF kernels that effectively detect a variety of change-points from different real-world applications.
Extensive evaluation of our new approach along with strong baseline methods on benchmark datasets shows the outstanding performance of the proposed method in retrospective CPD.