Detection and Localization of Changes in Conditional Distributions
Abstract
We study the change point problem that considers alterations in the conditional distribution of an inferential target on a set of covariates.
This paired data scenario is in contrast to the standard setting where a sequentially observed variable is analyzed for potential changes in the marginal distribution.
We propose new methodology for solving this problem, by starting from a simpler task that analyzes changes in conditional expectation, and generalizing the tools developed for that task to conditional distributions.
Large sample properties of the proposed statistics are derived.
In empirical studies, we illustrate the performance of the proposed method against baselines adapted from existing tools.
Two real data applications are presented to demonstrate its potential.
Introduction
The canonical setting for change point (CP) analysis is concerned with the detection and localization of changes in distributions of an independent sequence of observations { x t , t = 1 , 2 , · · · , n } .
In many applications, however, the observation at each index t is a pair ( x t , y t ) with x t a set of covariates and y t a variable of special interest for inference.
This setting incorporates many important problems in a variety of scientiﬁc ﬁelds.
For example, in ﬁnance, market data are often affected by monetary policies.
Thus, to more accurately predict stock prices using this additional information, one may want to ﬁrst understand whether and when the relationship between the target stock price and its associated covariates changes over time [37, 38, 28, 29, 49].
Another example comes from environmental analysis [6], where one goal is to learn the relationship between the levels of pollutants and the number of weekly hospital admissions for circulatory and respiratory diseases, which is subject to potential changes.
Before considering more complex models that incorporate time-varying relationships, one needs to understand ﬁrst whether the relationship indeed changes and if so, when.
Despite the existence of many applied scenarios where the relationship between a pair changes, change point methods speciﬁcally designed for paired data are quite scarce.
As far as we know, existing methods assume univariate y ’s and aim at changes in the conditional expectation of y given x .
Moreover, most papers [26, 24, 25, 39, 2, 4, 22, 43, 15, 40, 11] focus on the setting where the conditional expectation of y depends linearly on x .
The nonlinear and nonparametric cases, however, are relatively under-investigated.
The existing methods mainly focus on two settings.
One setting arises from challenges in econometric modelling, and they focus exclusively on the problem of testing for structural stability of the conditional mean function over time series [49, 19, 47, 46], while the (perhaps more) important problem of localizing the change point(s) is not investigated.
Another setting assumes that the covariate x t ’s are ﬁxed, univariate, and form an equi-spaced sequence lying in interval [0 , 1] [31, 32].
A detailed review of these related works is included in Section 2.
We are interested here in the case where both x and y can lie in a general space (not necessarily the real line), and we aim to identify changes in the whole conditional distribution, rather than only the mean.
This is of signiﬁcance to many applications: for example, in ﬁnance, it is beneﬁcial to understand changes in the volatility level of stock prices.
In this paper we want to both identify and localize such changes.
We will assume a random design setting.
Contrary to the ﬁxed design setting in [31, 32], we assume that the covariate x t ’s are randomly drawn from some unknown distribution.
This assumption is realistic and meaningful.
Consider the example where a sociologist wants to investigate whether the relationship between education and income has changed over years.
A survey is distributed every year, which records both the covariate (education level) and the response (income).
In this case, the covariates cannot be ﬁxed (as one cannot determine who will take the survey).
In the random design setting, one could argue that, if we combine x and y and treat them as as one variable, a sufﬁciently powerful change point method that can handle any change in distribution should perform well.
This approach is straightforward to implement; however, it targets changes in the joint distribution p ( x, y )
= p ( x ) p ( y | x ) , while we are only interested in changes in the conditional distribution p ( y | x ) .
An intriguing question is whether a method especially designed for change points in this conditional distribution will have better performance than those for joint distributions.
As the ﬁrst work to study these general changes, we answer this question by: (i) developing a mathematical formulation of this new problem and establishing its connection to a simpliﬁed task; (ii) investigating the advantages and limitations of applying existing methods to this new problem; and (iii) proposing new methodology.
Related Work
This section reviews relevant CP literature on paired observations.
Please refer to [34, 1] for reviews on the canonical (unpaired) setting.
More related work is discussed in the Appendix Section B.
Linear models.
Most publications on paired data assume that the conditional expectation of y is a linear combination of x [26, 24, 25, 39, 2, 4, 22, 43, 15, 40, 11], and attention has been focused on analyzing changes in the slope.
Recently, generalizations of this classic linear model have been investigated.
One strand of work focuses on analyzing changes in high dimensional linear regression models [30, 42, 50], while others include seasonality and/or correlated errors[43].
Structural stability test for conditional expectation.
Several econometrics papers [49, 19, 47, 46] focus on testing the stability of the conditional mean function using nonparametric methods.
Their task is simpler, as only testing is concerned, and moreover, only changes in the conditional expectation are considered.
Direct generalizations of their methods to our problem are difﬁcult, but they provide a good starting point.
Nonparametric methods for ﬁxed design.
[31, 32] consider univariate x t , y t ’s, where x t forms an equi-spaced sequence, and they focus on changes in E [ y t | x t ] .
Furthermore, [32] assumes that direction of the change is known.
We note that, although the ﬁxed and random design settings seem similar, a method which works well for one might not even work for another.
Intuitively, the ﬁxed design setting can be formulated as ﬁnding break points in an otherwise continuous/smooth curve, which is different from our focus.
Trend ﬁltering.
The recently proposed trend ﬁltering [27, 48, 51] also targets estimating conditional expectation in a ﬁxed design.
By solving a penalized least squares optimization problem, the estimated curve exhibit the structure of a piecewise polynomial function [48], the form of which is more restrictive than [31, 32]. Bayesian approaches.
Bayesian methods are also used for CP problems.
The most relevant are based on the Gaussian process change point analysis [44, 7, 23], which aims to detect change(s) in the mean and/or the covariance function of the Gaussian process regression.
In order to derive a tractable posterior, a Gaussian prior is usually placed on the mean functions.
The parametric formulations needed for Gaussian process CP approaches make them more restrictive than the methods proposed in this work.
Problem Statement
Task I: Analyzing changes in conditional distributions.
The ultimate goal is to analyze changes in the conditional distribution for a paired sequence { ( x t , y t ) , t = 1 , 2 , · · · , n } where x t 2 X and y t 2 Y with ( X , d ) a semi-metric space and Y a general space.
We assume that the covariates x t iid ⇠ F X , while the conditional distribution of y t given x t might go through an abrupt change.
Denote { F 0 Y | X = x , x 2 X} , { F 1 Y | X = x , x 2 X} to be two sets of conditional distributions that differ on at least one x 2 X .
Our task is to investigate:
• (Detection) Testing the null hypothesis
[Equation]
against the alternative
[Equation]
where ⌧ ⇤ =
d n ⇢ ⇤ e .
Here d a e denotes the least integer great than or equal to a .
• (Localization) When rejecting H 0 , obtain an estimator ˆ ⌧ of ⌧ ⇤ .
Task II:
Analyzing changes in conditional expectations.
We start by considering a simpler task.
Assume that Y ⇢ R (i.e., response y is a scalar), ( X , d ) is a Euclidean space, and the focus is only on potential changes of the conditional mean of y t given x t .
For this simpler task, suppose f 0 , f 1 : X !
R are two unknown functions with f 0 6 = f 1 , ✏ t ’s are independent random variables with
E ( ✏ t ) = 0 and Var ( ✏ t ) < 1 , and the ✏ t ’s are independent from x t ’s. Let F 0 ✏ , F 1 ✏ be two probability distributions (not necessarily equal).
We are concerned with the following two questions:
• (Detection) Testing the null hypothesis
[Equation]
against the alternative
[Equation]
where ⌧ ⇤ =
d n ⇢ ⇤ e
• (Localization) When rejecting H 0 , obtain an estimator ˆ ⌧ of ⌧ ⇤ .
We will adapt methods developed for Task II to solve Task I.
We start with some notations.
Notations.
Denote d !
as convergence in distribution.
For a sequence of real-valued random variables
X n and a sequence of real numbers u n , denote X n =
O a.s. ( u n ) if P ( 9 C < 1 , 9 n, 8 m > n, | X m |  Cu m )
= 1 , and X n =
o a.s. ( u n ) if P ( 8 C > 0 , 9 n, 8 m > n, | X m |  Cu n )
= 1 .
Denote 0 p , 1 p 2 R p as the vector of all 0’s and 1’s, respectively.
Denote I p 2 R p ⇥ p as the identity matrix and I as the indicator function.
Denote k · k 2 as Euclidean norm.
Methodology 4.1 Solution to Task II
The underlying idea is to transform the CP problem with paired data into the more amenable CP problem for univariate observations.
The best performing transformation, in our view, is reported next.
Another possible transformation that has good performance, but without theoretical guarantees, is reported in Appendix A.
The proposed transformation relies on proper estimates of the conditional expectation; speciﬁcally, we consider the Nadaraya-Watson (NW) estimator:
[Equation]
with k X a kernel function, and h X =
h n a sequence satisfying h n ! 0 as n ! 1 .
Introducing the transformed sequence ˜  t .
A natural direction is to ﬁnd a sequence { ˆ  t , t = 1 , 2 , · · · , n } such that ˆ  t 2 [0 , 1 ) represents the degree of difference in conditional expectation for data before t and data after t .
One such sequence is shown in Figure 1, where we see that without a
[Figure]
t
[Figure]
f
[Figure]
r
[Figure]
ed s equen c e change point ( panel), ˆ  t is ﬂat across all t ’s; and with a change point (right panel), max t ˆ  t will be larger and the maximizer arg max t ˆ  t roughly lies around t = ⌧ ⇤ .
Thus, inference for the CP problem could be based on the location and magnitude of the maximum among { ˆ  t } . To deﬁne the sequence { ˆ  t } used for inference (and for generating Figure 1), we introduce a natural sequence, { ˜  t } , as follows. At each t , consider dividing the data into two parts: before t (i.e., { ( x i , y i ) , i = 1 , 2 , · · · , t } ) and after t (i.e., { ( x i , y i ) , i = t + 1 , t + 2 , · · · , n } ).
We ﬁt two functions, the left-side estimate ˆ f  ( t, · ) , using only data { ( x i , y i ) , i = 1 , 2 , · · · , t } , and the right-side estimate ˆ f + ( t, · ) , using only data { ( x i , y i ) , i =
t + 1 , t + 2 , · · · , n } .
Intuitively, with some mild conditions, under the null, we have for all t ’s
[Equation]
while under the alternative, when t = ⌧ ⇤ ,
[Equation]
However, under the alternative, when t 6 = ⌧ ⇤ , both ˆ f + ( t, · ) and ˆ f  ( t, · ) will roughly lie between fand f 1 , and we expect the ‘difference’ between ˆ f  ( t, · ) and ˆ f + ( t, · ) to be smaller than that between f 0 and f 1 .
Formally, we deﬁne the ‘difference’ as
[Equation]
which is different from the usual l 1 or l 2 distance between functions by emphasizing on their difference at the value of x ’s with a larger density in F X .
In practice we do not know F X , and the expectation in (5) can be approximated by
[Equation]
(6) or more economically, a Monte Carlo estimate using fewer particles ( n
[Equation]
< n
[Equation]
(7) where x 0 t iid ⇠ ˆ F X and ˆ F X
=
(1 /n )
P ni =
x i with  x the delta measure centered at x .
Standardizing ˜  t into ˆ  t .
It can be shown that the sequence ˜
t satisﬁes the trend shown in Figure 1 (see Theorem C.1 in the Appendix), i.e., when sample size is sufﬁciently large, ˜  t ⇡ 0 under the null, while under the alternative, ˜  t increases with t before change point, while decreases after it.
While directly using ˜  t for solving CP problems might work, a better choice for localization is to consider a properly standardized version (denoted, ˆ  t ) which is comparable across all t ’s under the null.
This requires a more careful analysis of the limiting distribution of ˆ f  , ˆ f + under the null.
Proposition C.2 in the Appendix shows that under the null, with some mild assumptions, for any t =
d n ⇢ e with ⇢ 2 [ ⇢ 0 , ⇢ 1 ] and any ﬁxed x , as n ! 1 ,
[Equation]
where c 1 ( h X ) , c 2 ( x ) are constants depending on h X and x separately.
With Deﬁnition (6), result (8) immediately suggests setting
[Equation]
such that ˆ  t follows the same limiting distribution (and thus is comparable) for all t ’s. Notice that the standardized sequence { ˆ  t , t = 1 , 2 , · · · , n } is used in drawing Figure 1.
The proposed method is named KCE (Kernel-based change point analysis for Conditional Expectations).
Before summarizing KCE and introducing the detailed algorithm, we would like to discuss the similarity of of the proposed method with the well established CUSUM [35], as the deﬁnition (5) and standardization in (9) might look very similar to those in CUSUM.
Indeed, one can think of KCE as a generalization of CUSUM where the estimate for mean is replaced by the estimate for conditional expectations; a similar argument holds for the KCD algorithm proposed next to detect changes in conditional distributions.
Summary of algorithm and complexity.
As mentioned before, after calculating ˆ  t , the estimator for ⌧ ⇤ is set to ˆ ⌧ = arg max n 0  t  n 1 ˆ  t .
Here n 0 = d n ⇢ 0 e , n 1 = d n ⇢ 1 e , 0 < ⇢ 0  ⇢ ⇤  ⇢ 1 < 1 are pre-determined in order to prevent selecting too few samples near the ends [8, 9].
We have not discussed yet the detection step.
Traditionally, either analytic formulas or re-sampling methods have been used to calculate p-values for CP detection.
The (limiting) distribution of max ˆ  t is complicated, and we adopt a resampling-based method to obtain p-values.
The complete procedure to solve Task II is summarized in Algorithm 1.
Note that we store some pre-computed matrices for speed-up, and the overall time complexity is O ( n 2 ) and space complexity O ( n 2 ) . input: observations ( x t , y t ) } nt =
[Table]
, signiﬁcance
[Table]
, parameters n pre-compute: 1.
K X =
[ k X  h  1 X d ( x i , x j )  ] ni,j =1 2 R n ⇥ n . 2. A
P =1 [ ˆ f  t x i
4.2 Solution to Task I
So far we have derived the solution for Task II, the conditional expectation change point problem.
We show next how to adapt the proposed procedure to solve the more difﬁcult Task I, the conditional distribution change point problem.
Generalizing ˜  t .
Note that the NW estimator is linear on y t ’s, and thus ˜  t can be written as t
X i,j = w ( t, i, j, x i , x j ) y i y j + n X i,j = t + w t, i, j, x i , x j ) y
i
y
j t X i = n X j = t + w ( t, i, j, x i , x j ) y i y j , (10) where denote for simplicity
k X  h  1 X d ( x i , x j )  = k X ( i, j ) , then w ( t, i, j, x i , x j ) = 8> >< >>:
[Equation]
n
P nl
[Equation]
P
[Equation]
P
[Equation]
n
P nl
[Equation]
[Equation]
X
[Equation]
P
[Equation] t [Equation] k X
[Equation]
P nr
[Equation]
[Equation]
[Equation]
n P nl [Equation] P tr
[Equation]
[Equation]
k
[Equation]
l )
P nr
[Equation] [Equation] t. We replace the inner product terms y i y j in Equation (10) by k Y ( y i , y j ) (a standard kernel technique), yielding
[Equation]
Some notes on (12): if x t ⌘ c for all t ’s (which can be viewed as the canonical unpaired setting, observing only { y t } nt =1 ), ˜  t equals the maximum mean discrepancy [16] for testing equality between the distribution of { y i , i  t } and that of { y i , i > t } .
In the paired setting, the weights (11) depend also on the value of x t ’s: the more similar x i , x j are, the larger the weight on k Y ( y i , y j ) becomes.
Standardizing ˜  t .
Similar to Task II, one can show under some mild assumptions that ˜  t deﬁned in (12) has the same trend as in Figure 1 (see Theorem 5.1 in Section 5).
Standardizing ˜
t , however, is much less obvious. We defer the rigorous derivation to Theorem 5.1 in the following section.
Here we give some high-level intuition: standardization essentially relies on variance of ˜  t ; when deriving it, because of the randomness of x i ’s, a uniform bound that does not depend on x i ’s is needed. To establish uniform bounds, we need to put some assumptions on the space of x ’s. In order to regulate the complexity of the space X , we introduce Kolmogorov’s entropy, deﬁned as:
[Equation]
with N ✏ ( X ) the minimal number of open balls of radius ✏ in order to cover X . Now we are ready to present the standardization for ˜  t .
Recall the re-scaling factor in (9) is 1 /t + 1 / ( n  t ) , and we expect the re-scaling factor here to be inﬂated by the Kolmogorov entropy of space X in some way. The simplest case is when X is compact and X has a density with respect to the Lebesgue measure: we can still set ˆ  t using deﬁnition (9); and in the general case we suggest setting [Equation]
Intuitively, for any ﬁxed t = d n ⇢ e , the larger the sample size, the smaller the variance of ˜  t , and thus the smaller the rescaling factor.
For any ﬁxed n and t , the more complex the space X , the larger X , and the larger the rescaling factor.
A more detailed derivation on this standardization is included in the next section.
Summary of algorithm and complexity.
The complete procedure for solving task I is summarized in Algorithm 2.
Again we store some pre-computed matrices to save time.
The time complexity of Algorithm 2 is O ( n 3 ) and space complexity O ( n 2 ) .
The proposed method is named KCD (Kernel-based change point analysis for Conditional Distributions).
Theory
We present only some relevant theory for Task I, and defer the rest to Section C and all proofs to
Section D in the Appendix.
From the Moore-Aronszajn theorem, the function k Y ( y, · ) satisﬁes k Y ( y, y 0 ) =
h k Y ( y, · ) , k Y ( y 0 , · ) i H where H is a reproducing kernel Hilbert space, as long as kernel k
Y ( · , · ) is positive-deﬁnite.
Denote the conditional mean maps before and after change point as
[Equation]
in the sense that h f i ( x ) , k Y ( y, · ) i H =
E Y ⇠ F iY | X = x
[ k Y ( Y, y )] for i = 0 , 1 .
Deﬁne the norm k · k H =
p h · , · i H , and let
[Equation]
Let B ( x, h ) =
{ x 0 2 X , d ( x 0 , x )  h } be a closed ball in X .
We need the following assumptions to establish the asymptotic behavior of ˜  t .
Assumption 1 (small ball probability of F X ) .
There exists a non-decreasing function m ( · ) such that
[Equation]
input: observations { ( x t , y t ) } nt =1 , signiﬁcance level ↵ , parameters n 0 , n 1 .
output: estimated change point location ˆ ⌧ .
( ˆ ⌧ = n implies no signiﬁcant change point)
pre-compute: 1.
K X
=
[ k X ( h  1 X d ( x i , x j ))] ni,j =1 2 R n ⇥ n , K Y =
[ k Y ( y i , y j )] ni,j =1 2 R n ⇥ n .
A 2 R n ⇥ n , where A ij =
{+P jl =
+}
[ K X ] il .
B = (1 /n )
K X K > X 2 R n ⇥ n . 4.
C 2 R n ⇥ n , where C ij =
B ij [ K Y ] ij .
for t = n 0 , n 0 + 1 , · · · , n 1 do calculate Q 2 R n ⇥ n with
Q ij = 8>> < >>:
C ij A it A jt , if i, j  t, C ij
A it ( A jn  A jt ) , if i  t, j > t, C ij ( A in  A it )( A jn  A jt ) , if i, j > t. calculate ˜  t =
P ti,j =1 Q ij +
P ni,j = t +1 Q ij
P ti =
P nj = t +1 Q ij .
end for detection: obtain p -value for max n 0  t  n 1 ˆ  t using permutations or bootstrap.
localization: if p -value < ↵ , estimate ˆ ⌧ = arg max n 0  t  n 1 ˆ  t ; else, estimate ˆ ⌧ = n .
Assumption 2 (Lipschitz continuity of f 0 , f 1 ) .
Functions f 0 , f 1 satisfy 9 C 3 < 1 , 9 b > 0 , 8 x, x 0 2 X , k f 0 ( x )  f 0 ( x 0 ) k H + k f 1 ( x )  f 1 ( x
[Equation]
Assumption 3 (regularization on kernel k X ) .
k X ( · ) is a nonnegative, bounded and Lipschitz continuous function with support [0 , 1) , and if k X (1) =
the following conditions have to be satisﬁed
[Equation]
Assumption 4 (regularization on kernel k Y ) .
k Y ( · , · ) is a continuous, symmetric, positive deﬁnite, and uniformly bounded kernel.
Assumption 5 (topological complexity of X ) .
The topological complexity X ( · ) , together with the small ball probability function m ( · ) , satisfy
[Equation]
Remark 5.1.
Assumptions 1-5 are technical, but, in general, they are mild and not that restrictive.
For example, when the space X is compact, and k X , k Y are Gaussian kernels, Assumption 1,3, 4 are all satisﬁed.
If we further assume X is Euclidean and the random variable x has a density function,
Assumption 5 is satisﬁed.
Assumption 2 is probably the most abstract, but it essentially states that the conditional distribution p ( Y | X = x ) changes smoothly in the sense that p ( Y | X = x 0 ) is close to p ( Y | X = x 1 ) when x 0 is close to x 1 .
Theorem 5.1.
Suppose Assumptions 1, 2, 3, 4, 5 hold. (1) Under the null, for any t =
d n ⇢ e with ⇢ 2 [ ⇢ 0 , ⇢ 1 ] , ˜  t =
O a.s.
[Equation]
(2) Under the alternative, for any t =
d n ⇢ e with ⇢ 2 [ ⇢ 0 , ⇢ ⇤ ) \ ( ⇢ ⇤ , ⇢ 1 ] , ˜
t   ( ⇢ )  =
o a.s. (1) , where
[Equation]
Remark 5.2.
Assumptions 1, 2, 3, 4 are standard conditions for obtaining pointwise convergence of
NW estimators.
Assumption 5 comes from [13] and regulates the topological structure of the inﬁnite dimensional space X .
An example of m, X : if X is a compact subset of R p , X ( ✏ ) =
O (log(1 / ✏ )) , and, if X has a density with respect to the Lebesgue measure, m ( h ) =
O ( h p ) .
[Table]
[Table]
Remark 5.3.
Theorem 5.1 shows that as long as  6 = 0 , ˆ  t satisﬁes the trend in Figure 1.
Thus, performance of KCD is crucially affected by  , which is characterized in the next proposition.
Proposition 1.
Under Assumption 2, if k Y is a characteristic kernel, and F 0 Y | X = x 6 = F 1 Y | X = x for at least one x 2 supp ( F X ) , i.e., the support of F X , we have  6 = 0 .
Remark 5.4.
Proposition 1 shows that  acts as a discrepancy measure between two sets of conditional distributions, { F 0 Y | X
= x , x 2 supp ( F X ) } and { F 1 Y | X = x , x 2 supp ( F X ) } .
It is reminiscent of the maximal conditional mean discrepancy (MCMD) between two conditional distributions in [36].
Actually,  equals the expected value of squared MCMD between F 0 Y | X , F 1 Y | X with X ⇠ F X .
A detailed discussion on the link with [36] is included in the Appendix (Section B).
Experimental Results
This section investigates performance in synthetic data.
We report representative results on different forms of X , Y and types of changes, with additional results included in the Appendix.
Baselines.
We consider three baselines: one existing (the ﬁxed design CP method [31]), and two adapted from existing abrupt CP methods for unpaired data (denoted by D XY and D Y ).
D XY applies an existing CP method by treating ( x t , y t ) as the observation, while D Y does so by discarding x t .
There are numerous CP methods to choose from, and we use S 1 introduced in [33], which uniﬁes many existing nonparametric CP methods and can be viewed exactly as Algorithm 2 by setting x t ⌘ c .
For KCD (or KCE), D XY , D Y , we set k X ( u ) =
exp {  u 2 } and the choice of k Y depends on Y .
For the method in [31], there are restrictions on the kernel function that are violated by the
RBF kernel.
So, following [31], we set k ( u ) = 1 .
u 2 )
I (0  u  1) .
All bandwidths used for all methods are tuned among S h
= { 0 . 001 , 0 . 01 , 0 . 1 , 1 , 10 } on 10 independently generated data sets.
Evaluation metrics.
We report the l 1 localization error | ˆ ⌧  ⌧ ⇤ | and the empirical power, both averaged over 20 independent simulations.
Power is calculated under signiﬁcance level ↵ = {+
+}
.with p-value determined by 500 bootstraps.
Experiment A: y 2 R , E
[ y | x ] changes.
We ﬁrst consider task II, and we set F X = N (0 , 1) ,
F 0 ✏ = F 1 ✏ = N (0 , 1) , n = 1000 , ⇢ ⇤ = 0 .
7 , ⌧ ⇤ = 700 , ⇢ 0 = 0 .
05 , ⇢ 1 = 0 .
Before the change point, we set f 0 ( x ) = x , while after change point, we investigate different options for f 1 .
For
D Y , set k Y ( y, y 0 )
=
exp { k y  y 0 k 22 /h 2 Y } ; for D XY , set k Z ( z, z 0 )
=
exp { k z  z 0 k 22 /h
Z } with z t = ( x t , y t ) 0 .
KCE uses Algorithm 1 with d ( · , · ) set to Euclidean distance.
Localization comparisons are reported in Table 1a.
Note that the method in [31] and D Y perform not so well, which is expected as the former is designed for equi-spaced covariates, and the latter only looks at the marginal distribution of y t ; D XY performs better, but is still slightly worse than KCE.
KCE performs quite well, obtaining the most accurate estimator in most settings.
Power comparisons are similar and are included in Table 3a in the Appendix.
Experiment B: y 2 R , p ( y | x ) changes.
Here we investigate task I.
We start with y 2 R , E
[ y | x ] does not change, yet p ( y | x ) changes.
Speciﬁcally, we consider the model
[Equation]
where ✏ t iid ⇠ N (0 , 1) , v 0 ( x ) = x
and we consider different forms of v 1 ’s. We set F X = N (0 , 1) , n = 1000 , ⇢ ⇤ = 0 .
7 , ⌧ ⇤ = 700 , ⇢ 0 = 0 .
05 , ⇢ 1 = 0 .
KCD uses Algorithm 2 with k Y
( y, y 0 )
=
exp { k y  y 0 k 22 /h 2 Y } and d ( · , · ) set to Euclidean distance.
D Y uses k Y ( y, y 0 )
=
[-exp { k y  237 y 
-]
exp { k y  y 0 k 22 /h 2 Y } and D XY uses k Z ( z, z 0 )
=
exp { k z  z 0 k 22 /h
Z } with z t = ( x t , y t ) 0 .
Localization results are summarized in Table 1b.
We observe that KCD performs the best among all alternatives, followed by D XY , D Y , and lastly [31].
Experiment C: general y 2 Y , p ( y | x ) changes.
In the most general setting, let us consider task I with x 2 R p with p = 5 , F X = N ( 0 p , I p ) and different forms of Y ’s. When designing experiments, we deliberately let the conditional mean of y given x to be invariant.
We set k Z ( z, z 0 ) =
k X ( h  1 X d ( x, x 0 ))
k Y ( y, y 0 ) for D XY , and k Y used in all methods is tailored to each Y .
[31] is not applicable to non-scalar y t ’s and is thus omitted.
The other settings are identical to Experiment B.
The forms of Y ’s that are considered include:
(1) Y =
R p , and we set y t ⇠ N ( 0 p , I p ) before change point, and y t ⇠ N ( B 0 x t , I p ) with B =
[  1 ,  2 , · · · ,  5 ] and  i = 0 .
1 ⇥ i ⇥  ⇥ 1 p with different magnitude of  after the change point.
We set k Y ( y, y 0 ) =
exp { k y  y 0 k 22 /h 2 Y } .
(2) Y =
R p ⇥ p .
Denote Y ij as the element in the i -th row and j -th column of Y .
Before change point, we set Y ij =
X i X j where X i ,
X j denote the element in the i th and j th coordinate of X , while after change point, we set it to a different quantity.
We set k Y ( Y, Y 0 ) =
exp { k
[- Y 
-][Table]
/h [Table] Y } with Frobenius norm [Table]
(3) Y = P , the set of all univariate normal distributions with variance 1.
Each datum y t is m =observations independently drawn from N ( µ ( x t ) , 1) .
We consider µ ( x t ) =
(  / 5) x 0 t 1 5 with different magnitude  ’s. We set k Y ( Y, Y 0 )
=
exp { k Y
Y
k 2 W /h 2 Y } with 2-Wasserstein metric k · k W .
[Table]
Results for experiment C are summarized in Table 2.
In general, KCD (or KCE) outperform D XY and D Y , often with signiﬁcant improvements.
This observation is consistent across all experiments.
The trade-off is that KCD takes O ( n 3 ) time complexity, while KCE, D XY , D Y take only O ( n 2 ) .
Real Data Applications
UK stock index.
The Financial Times Stock Exchange 100 Index (FTSE 100) is a share index of the 100 largest companies listed on the London Stock Exchange.
We investigate potential changes in UK economy from January to August 2016, a period containing the Brexit vote (06/23/2016).
Following [20, 21], we consider changes in the daily value of FTSE 100 stock index, compared to two “reference” time series: the NYSE Composite Index and the NIKKEI 225 stock index.
The NYSE Composite is a stock market index of all common stocks listed on the New York Stock Exchange, and the NIKKEI 225 is a stock market index for the Tokyo Stock Exchange operating in the Japanese Yen.
They are used here as indicators for the US and Japan economies.
Following [20], we divide the closing value of each index by its corresponding daily Euro exchange rates.
All data are downloaded from https://www.marketwatch.com/. Data from the three stock indices are shown in Figure 2a.
The processed dataset contains n = 157 sample points, with y 2 R the FTSE 100 divided by
Euro exchange rate to British pound, and x 2 R 2 the vector consisting of NYSE and NIKKEIdivided by Euro exchange rate to US dollars and Japanese Yen, respectively.
Using KCD with k X ( u ) =
exp {  u 2 } , d ( x, x 0 ) =
k x  x 0 k 2 , h 2 X = 0 .
1 , k Y
( y, y 0 )
=
exp { k y  y 0 k 22 /h 2 Y } , h
Y = 0 .
1 and n 0 = 5 , n 1 = n  5 , we identify 06/23/2016 as the change point, with a pvalue < 0 .
At the signiﬁcance level 0.05, we conclude that there is a change point, which is consistent with [
[Figure] [Figure]
Market interest rates.
We consider changes in the relationship between market interest and discount rates over 1973-1989.
The discount rate is the rate “at which the Federal Reserve System (Fed) lends and is set by the Fed” [4].
Market interest rates usually respond to changes in discount rates, but the differentials in response might be different, reﬂecting different conditions of the economy.
We examine the response of market interest rates to discount rates over time.
The data we use are collected by [10], where the yields of three-month T-bills are treated as market interest rates.
The data set contains n = 56 sample points, with covariate x 2 R the discount rate, response y 2 R the market interest rate.
We note that discount rates are changed irregularly, thus the sequence is not equi-spaced in terms of time.
The sequence { ( x i , y i ) } ni =1 is visualized in Figure 2b.
Using KCD with k X ( u ) =
exp {  u 2 } , d ( x, x 0 ) =
k x  x 0 k 2 , h 2 X = 0 .
1 , k Y
( y, y 0 )
=
exp { k y  y 0 k 22 /h 2 Y } , h 2 Y = 0 .
1 and n 0 = 5 , n 1 = n  5 , we estimate a change point 08/17/1979.
This is identical to the estimation in [4], and is close to the October 1979 change in the Fed’s operating procedures ([4]).
However, the p-value is 0.25, indicating that the change is not signiﬁcant.
This is similar to the result obtained using the vanilla symmetric response model in [4], but different from that using adjusted or asymmetric response model [4].
Conclusion and Discussion
To identify changes in the conditional distribution of paired sequences, we investigate several approaches, some based on existing algorithms and one novel.
Our nonparametric method applies to general covariates and responses, and outperforms existing methods.
Future directions of research include relaxing distributional assumptions and improving the O ( n 3 ) time complexity of Algorithm 2. As suggested by one reviewer, the question of detecting changes in conditional distributions with multiple or gradual change points is also important but unresolved. A rigorous treatment of this generalized setup requires nontrivial modiﬁcations of the method and a re-formulation of the mathematical set-up but we provide a simple windowing approach as a possible solution for multiple change points in the Appendix.
There are no foreseeable negative societal impacts of this work.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 8.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 8.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] The assumptions are stated when the theory is presented.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]