Contrastive Multivariate Singular Spectrum Analysis
Abstract
We introduce Contrastive Multivariate Singular Spectrum Analysis, a novel unsupervised method for dimensionality reduction and signal decomposition of time series data.
By utilizing an appropriate background dataset, the method transforms a target time series dataset in a way that evinces the subsignals that are enhanced in the target dataset, as opposed to only those that account for the greatest variance.
This shifts the goal from ﬁnding signals that explain the most variance to signals that matter the most to the analyst.
We demonstrate our method on an illustrative synthetic example, as well as show the utility of our method in the downstream clustering of electrocardiogram signals from the public MHEALTH dataset
Introduction
Unsupervised dimensionality reduction is a key step in many applications, including visualization [8] [10], clustering [5] [11], and preprocessing for downstream supervised learning [13].
Principal
Component Analysis (PCA) is one well-known technique for dimensionality reduction, which notably makes no assumptions about the ordering of the samples in the data matrix X ∈ R N × D .
Multivariate
Singular Spectrum Analysis (MSSA) [7] is an extension of PCA for time series data, which been successfully applied in applications like signal decomposition and forecasting [6] [9] [12].
In MSSA, each row is read at a certain time step, and thus is inﬂuenced by the ordering of the samples.
MSSA works primarily by identifying key oscillatory modes in a signal, which also makes it useful as a generalpurpose signal denoiser.
However, MSSA (and PCA) is limited to ﬁnding the principal components that capture the maximal variance in the data.
In situations where the information of interest explains little overall variance, these methods fail to reveal it.
Recently, extensions like contrastive PCA (cPCA)
[1] have shown that utilizing a background dataset Y ∈ R M × D can help better discover structure in the foreground (target)
X that is of interest to the analyst.
Contrastive Multivariate Singular Spectrum Analysis (cMSSA) generalizes cPCA and applies it to time series data.
Figure 1 visualizes the relationships between the four methods.
As a contrastive method, cMSSA emphasizes salient and unique sub-signals in time series data rather than just those that explain most of the signal variance.
While standard MSSA is useful for denoising a signal, cMSSA additionally “denoises” signals of structured but irrelevant information.
Contrastive Multivariate Singular Spectrum Analysis 2.1 Standard MSSA
Consider a centered one-channel times series x ∈ R T .
We construct a Hankel matrix H x ∈ R T (cid:48)
× W with window size W as follows:
[Equation]
where T (cid:48) =
T − W + 1 .
To extend to the multivariate case, let X ∈ R T
× D be a D -channel time series that runs for T steps.
We construct the Hankelized matrix H X with window W by horizontally concatenating the per-channel Hankel matrices into a T (cid:48)
-by- DW matrix: H X =
[ H x (1) ; H x (2) ; . . .
H x ( D ) ] .
Next we compute the covariance matrix C X ∈ R DW × DW for H X .
The next step is to perform the eigendecomposition on C X , yielding DW eigenvectors.
Of these we take the top K vectors with the largest corresponding eigenvalues.
We denote e ( k ) as the eigenvector with the k th largest eigenvalue.
We collect the vectors into a matrix E ∈ R DW × K .
To transform our original time series X , we have two options: (a) Project X into the principal component (PC) space deﬁned by E : A = H X E or (b) use A to compute the k th reconstructed component (RC) R ( k ) as done in the SSA literature:
[Equation]
where L t = max(1 , t − T + W ) , U t = min( t, W ) , and W t =
U t − L t + 1 .
Summing up the reconstructed components reproduces a denoised version of the original signal.
For our purposes, we opt instead to take the horizontal concatenation of the reconstructed components as the second transform: R =
[ R (1) ; R (2) ; . . .
; R ( K ) ] .
To handle multiple time series, we vertically stack each
Hankelized matrix.
The algorithm proceeds identically from there.
2.2 Contrastive MSSA
The modiﬁcation to MSSA we introduce is via a new variable α ≥
we call the contrastive hyperparameter.
We construct H Y for another D -channel times series Y (the background data) via the same process.
It is not required that X and Y run for the same number of time steps, only that their channels are aligned.
We compute a contrastive covariance matrix C =
C X − αC Y and perform the eigendecomposition on C instead of C X .
The intuition for this is that by subtracting out a portion of the variance in Y , the remaining variance in X is likely to be highly speciﬁc to X
but not Y .
This is the key additional mechanism behind cMSSA — if α = 0 , then no contrast is performed, and cMSSA reduces down to just MSSA.
The choice of α is non-trivial.
We outline a routine for auto-selecting a small number of promising values for α in the appendices.
Experiments 3.1 Synthetic example
To illustrate cMSSA, we present a simple synthetic example.
We generate an artiﬁcial one-channel signal Y by sampling 500 sinusoids with different frequencies, amplitudes, phases, and vertical shifts.
White Gaussian noise sample from N (0 , 1) is added in as well.
We generate X in the same manner, but add in a very speciﬁc sub-signal (Figure 2a) that has comparatively low variance compared to the whole time series.
The signals X and Y are generated independently as to rule out simple signal differencing as an explanation.
We take X as foreground and Y as background.
We set W = 100 , α = 2 , and use only the top K = 2 RCs.
Figure 2 displays the reconstructions computed by MSSA versus cMSSA, alongside the sub-signal that was injected into X .
Speciﬁcally,
[Figure]
we see that the cMSSA reconstruction shown in Figure 2c yields a noisy approximation of the subsignal of interest, Figure 2a.
Note that the variance of the noise is comparable to the variance of the sub-signal—more noise would eventually overpower cMSSA’s ability to extract the sub-signal.
3.2 Clustering of electrocardiograms
Data: The data used in our experiments is taken from the public MHEALTH dataset [3].
In the dataset, 10 individuals were asked to perform 12 physical activities as various sensors recorded various motion data.
In addition, the researchers also collected two-lead electrocardiogram (ECG) readings, which we take as dual-channel time series data.
In addition to the 12 activities, there is a 13th NULL class that represents ECG signals collected between each activity but which don’t have labels themselves.
To increase the number of individual time series, we partition each one in half.
For our experiments, the foreground data are all time series labelled as either J OGGING , R UNNING ,
J UMPING , or C YCLING , 20 time series each for a total of 80.
These four, being the more cardiointensive of the 12, had much more signal activity that would be needed to be sifted through, exactly the type of environment cMSSA is intended to handle.
For background data, we take all 272 time series belonging to the NULL class.
Setup: To evaluate the effectiveness of cMSSA over its non-contrastive counterpart, we run both cMSSA and MSSA with a variety of hyperparameter settings.
For each ﬁtted model, we transform the foreground data to both the PC and RC spaces.
Once the transformations are had, we perform spectral clustering into 4 clusters and compare the resulting clusters to the activity labels on the time series data, which were hitherto withheld from the algorithms.
There arehyperparameters: the window size W ∈ { 8 , 16 , 32 , 84 , 18 } , the number of desired components
K ∈ { 1 , 2 , 4 , 6 , 8 , 10 , 12 , 14 , 16 , 18 , 20 } , and the contrastive parameter α .
We set K only if the value is less than or equal to DW (where D = 2 in this case).
For α , we used our automatic routine to compute ﬁve key values to try for each setting of W and K .
Of these ﬁve, one of them is zero, representing standard MSSA.
Altogether, we run 530 experiments, 106 of which are standard MSSA, and the remaining cMSSA.
The spectral clustering requires an afﬁnity matrix S ∈ R N × N which contains the similarities between any pair of time series, where N is the number of times series we wish to cluster.
Let X ( i ) and X ( j ) be two time series.
Using the FastDTW metric [14] with a euclidean norm 1 , we deﬁne the similarity
S ij to be 1 F AST DTW ( X ( i ) ,X ( j ) )
+1 .
The cluster evaluation uses the well-rounded BCubed metric [2] to compute the precision, recall, and F1 harmonic mean for a particular cluster prediction.
We also perform the evaluation in the model-free sense where we simply cluster the time series with no transformation as a basic baseline.
[Table]
The points look at only those where the
Results: Table 1 reports the best representative contrastive and non-contrastive models, comparing both to the model-free baseline.
We observe a number of things.
First, both MSSA and cMSSA outperform the model-free baseline.
Second, cMSSA has 9-10 point gains over cMSSA in each of precision, recall, and F1.
Third, both ﬁnd that using A over R as the transform yielded better results.
Finally, and most interestingly, is the number of PCs used.
Of the DW number of PCs available, MSSA gets its best performance using half (16 out of 32).
The ratio is very different for cMSSA, which only uses one PC out of the maximum of 256 available.
This highlights an interesting efﬁciency of cMSSA.
By ﬁltering out unnecessary components, the remaining not only account for less signal variance, but provide diminishing returns with each subsequent component used.
Figure 3 shows a more granular view of the general gains to be had from using cMSSA.
For a particular setting of
W and K , we plot the F1 score for the non-contrastive case vs the contrastive case.
Due to the four values of α s used in the contrastive case, we take the model that had the greatest F1.
The line is a visual guide – points below the line mean that the contrast was useful for a particular setting of the hyperparameters.
Conclusion
We have developed cMSSA as a general tool for dimensionality reduction and signal decomposition of temporal data.
By introducing a background dataset, we can efﬁciently identify subsignals that are enhanced in one time series data relative to another.
In an empirical experiment, we ﬁnd that for virtually any setting of the hyperparameters, cMSSA is more effective at unsupervised clustering than MSSA, contingent on appropriate choices for the foreground and background data.
Some basic heuristics should be kept in mind when choosing to use cMSSA as an algorithm.
First, the data ideally should exhibit periodic behavior, as MSSA (and by extension, cMSSA) is particular well suited to ﬁnding oscillatory signals.
Second, X and Y should not be identical, but should share common structured signal such that the contrast retains some information in the foreground.
As an example, the ECG foreground data consisted of subjects performing very speciﬁc activities, whereas the background consisted of a corpus of unlabelled ECG signals in which the participants performed no speciﬁc activity.
We would expect a good amount of overlap in signal variance, but signals speciﬁc to the four activities would be under-represented in the background.
Thus contrast is a plausible way to extract this signal.
Finally, we note that the only hyper parameter of cMSSA is the contrast strength, α .
In our default algorithm, we developed an automatic scheme for selecting the most informative values of α (see the Appendix A).
The experiments performed for this paper use the automatically generated values, and we believe this default will be sufﬁcient in many use-cases of cMSSA.
The user may also input speciﬁc values for α if more ﬁne-tuned exploration is desired. Appendices A Routine for candidate α generation Algorithm Require:
Minimum α to consider α min , maximum α to consider α max , total number of α s to consider n , and number of α s to return m , foreground signal X , background signal Y , window
W , and number of components K .
1: procedure G ENERATE
A LPHAS ( α min , α max , n , m , X , Y , W , K ) 2:
C ← L OG S PACE ( α min , α max , n ) (cid:46)
Get n α s spaced evenly in log-space.
C ← C ∪ { 0 } (cid:46) Include zero in candidate set.
4: 5: for α ( i ) ∈ C do 6:
E ( i ) ← G ET E IGEN ( X , Y , W , K , α ( i ) )
(cid:46) Use cMSSA to compute eigenvectors.
7: end for 8: 9:
S ← E MPTY ( R n +1 × n +1 )
(cid:46) Initialize afﬁnity matrix.
10: for i ∈ { 1 , . . .
, n + 1 } do 11: for j ∈ { i, . . .
, n + 1 } do 12: s ← (cid:13)(cid:13)(cid:13) E ( i ) T E ( j ) (cid:13)(cid:13)(cid:13) ∗ (cid:46) Take nuclear norm of matrix product.
S i,j ← s 14:
S j,i ← s 15: end for 16: end for 17: 18:
Z ← S PECTRAL ( S , C , m ) (cid:46) Get m clusters.
C ∗ ← {0} (cid:46) Set of best α s to return.
Zero always included.
21: for z ∈ Z do 22: if 0 / ∈
z then (cid:46)
We ignore all α s that were clustered with zero.
23: α ∗ ← M EDIOD ( z , S )
(cid:46)
[Equation]
[Equation]
28: end
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]