Scalable Hybrid Hidden Markov Model with Gaussian Process Emission for Sequential Time-series Observations
1. Introduction
Employing non-linear function with Gaussian Process (GP) prior as the emission function of HMM has some advantages in characterizing the sequences of time-series observations; it can model the pair of input-output observations such as observed time and the corresponding response (Frigola, 2015; Nakamura et al., 2017; Nagano et al., 2018).
Moreover, employing SM kernel (Wilson and
Adams, 2013) for the covariance of GP prior can characterize the dataset based on diversely characterized stationary kernel (Ulrich et al., 2014, 2015).
However, due to the scalability issue for the training, this model is limited to modeling a small size of dataset.
In this work, we introduce a hybrid Bayesian HMM with GP emission using SM kernel, which we call HMM-GPSM, for modeling sequences of single-channel time-series observations.
Then, we propose a scalable inference method to train the HMM-GPSM with large-scale sequences of timeseries data D = { x t , y t } Tt =1 with x t and y t ∈ R N t having (1) long sequences of state transitions ( T is large) and (2) a large number of time-series observation from each state ( N t is large).
• To address issue (1), we employ stochastic variational inference (SVI) based on (Hoffman et al., 2013; Foti et al., 2014) to efﬁciently update the parameters of HMM-GPSM with the long sequences dataset.
To be speciﬁc, we propose the approximate evidence lower bound for the full T sequences of time-series observations which can be computed using the randomly sampled L sub-sequences of time-series observations.
This approximation can linearly reduce the computational complexity for computing the evidence lower bound from O ( T ) to O ( L ) .
• To address issue (2), we propose the approximate GP emission using spectral points sampled from the spectral density of SM kernel and the efﬁcient inference for the kernel hyperparameters of approximate GP emission and corresponding HMM-GPSM.
To be speciﬁc, we approximate the SM kernel by employing the spectral points sampled from Gaussian mixture spectral density based on random Fourier feature (RFF) (Rahimi and Recht, 2008).
We then introduce the variational distribution on the spectral points while treating these points are random variables and derive the regularized lower bound of GP emission likelihood.
Using this lower bound, we derive the evidence lower bound of the HMM-GPSM that can be scalably computed for a large number of time-series observations in each sequence.
This approximation reduces the computational complexity for computing GP emission likelihood from
O ( N 3 t ) to O ( N t M 2 ) with M sampled spectral points under M (cid:28) N t .
The proposed methods can be jointly used to efﬁciently update the parameters of HMM-GPSM with the dataset having both (1) and (2) issues.
We validate the proposed method on the synthetic using the clustering accuracy and training time as the performance metrics.
2. Methodology 2.1. Hybrid Bayesian HMM with GP Emission
Hybrid Bayesian HMM with GP emission has the structure of Input-Output HMM (Bengio and
Frasconi, 1995) with the emission function being modeled by GP.
To introduce the model, we assume that T sequences of inputs and outputs pairs X
= { x t } Tt =1 and Y =
{ y t } Tt =1 are given where y t ∈ R N t is the N t dimensional output corresponding to the input x t ∈ R N t obtained at time t .
We denote z t ∈ { 1 , .., K } representing the hidden state for x t and y t .
To explain the relation between y t and x t , the target function f t with the conditional GP prior given hidden state z t and covariance k
z t is deﬁned as
[Equation]
where θ z t denotes the hyperparameters of the SM kernel k z t having Q z t mixture components.
Under the Gaussian noise assumption, we deﬁne the emission function as the conditional marginal likelihood p ( y t | x t , z t ) given the hidden state z t and the input x t as
[Equation]
Then, the joint likelihood of Hybrid HMM with GP emission is deﬁned as
[Equation]
where p ( π ) =
Dir( π | α π ) is a prior distribution for the initial parameter π ∈ R K and p ( A )
=
(cid:81)
Ki =1 Dir( A i | α Ai ) is a prior distribution for transition matrix A ∈ R K × K .
A i denotes the i th row of A .
2.2. Variational Inference
Under Mean-Field assumption, let us assume the variational distribution q ( π, A, Z ) =
q ( π
[Equation]
q ( A
[Equation]
q ( Z
[Equation] [Equation]
where w π and { w Aj } Kj =1 are variational parameters for q ( π ) and q ( A ) .
Then, using Jensen inequality, we derive the variational objective L called by evidence lower bound (ELBO) as
[Equation]
where E q ( Z,A,π ) [log p ( Y, Z | X, A, π )] is expressed as
[Equation]
For the training, L is maximized by alternatively updating the local hidden variables q ( Z ) , variational parameters w π and { w Aj } Kj =1 for q ( π, A ) , and kernel hyperparameters { θ k } Kk =1 based on variational EM algorithm.
The detailed update procedures are described in the appendix.
2.3. Scalable Variational Inference 2.3.1. SVI A PPROACH FOR L ONG SEQUENCES ( T IS LARGE )
Given the T sequences of time-series, we randomly sample L consecutive sequences of inputs and outputs X sL = { x k − 1+ l } Ll =1 and Y sL =
{ y k − 1+ l } Ll =1 , where k is sampled uniformly from k ∈ { 1 , .., T − L + 1 } .
We linearly approximate the ELBO of full sequences by considering the expected log joint likelihood of { Z sL , Y sL } given X sL as
[Equation]
Based on Eq.
(7), the batch factors C
As and C θs for calibrating the ELBO of { X sL , Y sL } with full
ELBO of { X, Y } are obtained as
[Equation]
The detailed derivation of batch factors are explained in appendix.
With the batch factor C As in Eq.
(8) and the estimated q ∗ ( Z sL ) for Y sL and X sL , the variational parameters w π for q ( π ) and w A for q ( A ) are updated by stochastic natural gradient descent as
[Equation]
where p n is n -th iteration learning rate.
SM kernel hyperparameters θ = { θ k } Kk =1 are updated by maximizing the following expected log marginal likelihood coordinated with the batch factor C θs as
[Equation]
2.3.2. A PPROXIMATE GP E MISSION FOR A LARGE NUMBER OF TIME - SERIES ( N t IS LARGE )
For the SM kernel k SM ( x − y )
= (cid:80) Qq =
1 w q exp (cid:0) − 2 π 2 (( x − y )
T σ q ) 2 (cid:1) cos (cid:0) 2 π ( x − y )
T µ q (cid:1) with the parameters { w q , µ q , σ 2 q } Qq =1 , we approximate the SM kernel based on RFF.
Let S =
[ S 1 , 1 , .., S 1 ,m , .., S Q, 1 , .., S Q,m ] be the random spectral points and q ( S ) be the variational distribution of S deﬁned as
[Equation]
Using the set of the spectral points s = ∪ Qq =1 { s q,i } mi =1 ∼ q ( S ) sampled by reparametrization trick as s q,i = µ q + σ q ◦ (cid:15) i with (cid:15) i ∼ N ( (cid:15) ; 0 , I ) , we deﬁne the feature map φ SM ( x ; s ) as
[Equation]
to approximate the SM kernel k SM ( x − y ) by φ SM ( x ; s ) φ SM ( y ; s ) T and gram matrix k SM ( x t , x t ) ∈
R N t × N t by Φ SM ( x t ; s )
Φ SM ( x t ; s ) T with the feature matrix Φ SM ( x t ; s
) ∈ R N t × 2 Qm .
Then, using Jensen’s inequality, we derive the regularized lower bound of log p ( y t | z t , x t ) as
[Equation]
where p ( y t | x t , z t , s ( k ) ) is computed as N ( y t | m z t ( x t ) , Φ SMz t ( x t ; s ( k ) )
Φ SMz t ( x t ; s ( k ) )
T + σ 2 (cid:15)
zt I ) with the set of spectral points s ( k ) sampled from q ( S ) at k times.
Eq (14) enables efﬁcient computation because the inversion of the approximate gram matrix is efﬁciently computable (L´azaro-Gredilla et al., 2010).
The prior distribution of p ( S ) is assumed as (cid:81) Qq =1 (cid:81) mi =1 N ( S q,i ; (cid:101) µ q,i , (cid:101) σ 2 q,i ) .
Applying the result of Eq.
(14) to the conditional log likelihood log p ( y t | x t , z t ) in Eq.
(6), we derive the lower bound L asm , which can be efﬁciently computable, as
[Equation]
under the condition that the parameters of prior p ( S ) are assumed as { (cid:101) µ q.i , (cid:101) σ 2 q,i } Mi =
= { µ q , σ 2 q } .
Here, E q ( Z,A,π ) [log p ( Y, Z | X, A, π )] is the expected joint likelihood over q ( Z, A, π ) using the approximate GP emission p ( y t | x t , z t ) deﬁned as
[Equation]
Optimizing the L asm in Eq.
(15) can be efﬁciently conducted in the same way as L in Eq.
(5) is optimized, using the less number of operation for updating the parameters.
It can be used together with section 2.3.1 SVI approach for long sequences dataset.
3. Experiments Evaluation Metric
• Accuracy : The ratio of correct hidden state estimation for sequences of time-series observations.
Since the estimated hidden states { ˆ z t } Tt =1 are possibly equivalent to the true hidden states up to the permutation of hidden states, we reorder them by Munkres
algorithm (Munkres, 1957); For example, if two sequences of the estimated hidden states are given as z 11:3 =
(1 , 2 , 3) and z 21:3 = (2 , 3 , 1) , and
there exists the correspondence π between hidden states z 11:3 and z 21:3 to satisfy π (1) = 2 , π (2) = 3 , and π (3) = 1 , those estimated hidden states z 11:3 and z 21:3 are equivalent.
Then, we calculate the ratio of how many reordered states are matched with the total T true hidden states.
• 1 iteration time : single iteration time (seconds) required to update local and global parameters.
Simulation data
We generate the sequences of time-series observations with eight different hidden states.
We assume that state transition follows Markov process.
Speciﬁcally, we construct two groups of hidden states whose dynamics are different: (1 → 2 → 3 → 4) and (5 → 6 → 7 → 8) .
In the ﬁrst group, each
[Figure]
a cc u r a c y
[Figure]
state except state 4 follows the staying probability 0 .
7 and the moving probability 0 . 3 .
Similarly, in the second group, each state except state 8 has staying probability 0 .
3 and the moving probability 0 .
The special states { 4 , 8 } connect the two group with probability 1, i.e., (4 → 5) and (8 → 1) .
Given the hidden state s ∈ { 1 , .., 8 } , the time-series observation f s ( t ) is generated by
[Equation]
(17) where the weights α s =
[ α s 1 , .., α s 6 ] are sampled from Dir( α s | α 0 ) , and the frequency components ω sq are sampled from the interval [0 , W Hz ] .
3.1. Long Sequences ( T is large)
We investigate how the SVI approach reduces the training time required to train HMM-GPSM with long sequences of time-series dataset ( T is large).
We compare the following inference methods:
• VI (T) :
Variational inference for full T sequences in appendix • SVI (L) :
Stochastic Variational inference with L sampled sub-sequences in section 2.3.
We generate the sequences of time-series observations using Eq.
(17) having 150 sequences of time-series.
We ﬁx the number of data points in each time-series observation as N t = 200 .
We limit spectral range as W Hz = 100 to make sure that time-series with N t = 200 contains enough information to infer the spectral characteristics of the time-series according to the Nyquist–Shannon sampling theorem (Oppenheim, 1999).
We train the ﬁrst 150 sequences of times-series ( T = 100 , N t = 200 ) and test the left sequences ( T = 50 , N t = 200 ); VI approach uses the whole sequences ( T = 100 ) of the training data, while the SVI approach uses the sub-sequences with the length L ∈ { 10 , 20 , 50 } .
In addition, to see how the model complexity affects the training time, we consider the number of mixture for the SM kernel as Q ∈ { 3 , 4 } .
Figure 1(a) compares the training performance of the SVI training approach for varying batch length L ∈ { 10 , 20 , 50 } with the ﬁxed Q = 3 .
Figure 1(b) shows the same results for Q = 4 .
[Figure]
In this plot, each test accuracy for VI and SVI is evaluated every iteration during total 30 training iterations.
Figures show that SVI approach with the smaller batch length L ∈ { 10 , 20 , 50 } takes less training time to achieve a similar level of test accuracy compared to the VI approach.
3.2. Large number of observations ( N t is large)
In this experiment, we investigate how the derived ELBO L asm in Eq.
(15) reduces the training time of the HMM-GPSM and affects the accuracy of hidden state estimation when N t is large.
To this end, we compare the following two inference methods:
• SVI (L) :
Stochastic Variational inference with L sampled sub-sequences in section 2.3.1.
• SVI-AGPE (L, R) :
Stochastic Variational inference with L sampled sub-sequences in section 2.3.1.
and approximate GP emission with the sampling rate R in section 2.3.2.
During the experiment, we change the number of mixtures Q for SM kernel and the ratio of the sampled spectral points R used for the SM kernel approximation to investigate how these parameters affect the accuracy and training time; the number of sampled spectral points M is set as M ∝ N t R .
We generate 150 sequences of time-series observation by Eq.
(17) with N t = 1000 and W Hz = 500 .
The ﬁrst 100 sequences ( T = 100 , N t = 1000) and the left 50 sequences ( T = 50 , N t = 1000) are used for training and test, respectively.
For the number of repetition K for SM kernel approximation in Eq.
(16), we use K = 1 for training and K = 3 for test.
Figure 2(a) compares the training performance of the SVI-AGPE approach for varying sampling rates R ∈ { . 10 , .
30 } with the ﬁxed L = 10 and Q = 5 .
Figure 2(b) shows the same results for
Q = 6 .
Each test accuracy for SVI and SVI-AGPE approaches is evaluated every iteration during 50 training iterations.
Figures show that SVI-AGPE with the rate R ∈ { . 10 , .
30 } takes less training time to achieve a similar level of test accuracy for SVI.
4. Conclusion
In this work, we propose the scalable learning method for HMM-GPSM and validate that the proposed learning algorithm scalably trains HMM-GPSM with a large-scale dataset while maintaining the clustering performance.
SUPPLEMENTARY MATERIAL Appendix A. Derivation A.1. ELBO Derivation for Eq. (5)
[Equation]
where cross entropy H (
[Equation] [Equation]
[Equation]
A.2. Batch factor Derivation for SVI Approach for Eq. (8)
Given the sampled i ∈ { 1 , .., T − L + 1 } uniformly, let Y sL =
{ y i , .., y i + L − 1 } be sampled observation with the length L and X sL be corresponding inputs and Z sL corresponding hidden states.
The expected log joint likelihood of { Z sL , Y sL } given X sL is approximated as
[Equation]
transition term
[Equation]
observation term
[Equation]
This implies that transition and observation term of E q [log p ( Y sL , Z sL | X sL )] for the sampled i ∈ { 1 , .., T − L + 1 } can be approximated as
[Equation]
Thus, the batch factors, C As and C θs , to calibrate the approximated ELBO are obtained as
[Equation]
The transition term in expectation in ( ∗ ) can be approximated as
[Equation]
approximate term
[Equation]
  ≈
[Equation]
=
E
[Equation]
L
T
[Equation]
t = log
[Equation]
1 , A )
[Equation]
Here, the observation term in expectation in ( ∗ ) can be approximated as
T − L (cid:88)
i
[Equation]
=
E
[Equation]
 
[Equation]
=
E
[Equation]
approximate term
[Equation]
≈ E q (cid:34)
[Equation]
= E q (cid:34)
L
[Equation]
A.3. SM kernel Approximation for Eq.
Given the parameters of SM kernel { w q , µ q , σ q } Qq =
[Equation]
we sample spectral points s q = { s q,i } mi =1 from
Gaussian distribution N ( S ; µ q , σ q ) by reparametrization trick as
[Equation]
where (cid:15) i ∼ N ( (cid:15) ; 0 , I ) for i = 1 , .., m .
If we deﬁne the feature map φ s q ( x ) as
[Equation]
, then φ s q ( x ) φ s q ( y ) T can approximate k q ( x − y ) which is the inducted kernel from Gaussian Spectral density N ( S ; µ q , σ q ) by Bochner’s theorem
[Equation]
Using the above derivation
[Equation]
if we deﬁne sampled spectral points s = ∪ Qq
=1 { s q,i } mi =1 with s q,i ∼
N ( µ q , σ 2 q ) and the feature map φ SM ( x ) = (cid:104) √ w 1 φ { s 1 ,i } mi =1 ( x ) , .., √ w Q φ { s Q,i } mi =1 ( x ) (cid:105) , then φ SM ( x ) φ SM ( y ) T is an unbiased estimator of k SM ( x, y ) as
[Equation]
A.4. Regularized Lower bound for Eq. (14)
Let q ( S ) be variational distribution deﬁned in Eq.
We can derive the lower bound L as follows:
[Equation]
where s ( i ) is i -th sampled spectral points from q ( S ) .
A.5. ELBO Derivation for Eq. (15)
L = E q
( Z,A,π ) [log p
( Y, Z | X, A, π )] − KL ( q ( A, π ) || p
( A, π ))
=
E q ( π ) [log p ( z 0 | π )]
+ E q
( Z,A ) (cid:34) T (cid:88) t
= log p ( z t | z t − 1 , A ) (cid:35) + E q ( Z ) (cid:34) T (cid:88) t = log p ( y t | x t , z t ) (cid:35) − KL ( q ( A, π ) || p
( A, π )) ≥ E q ( π ) [log p ( z 0 | π )]
+ E q
( Z,A ) (cid:34) T (cid:88) t
= log p ( z t | z t − 1 , A ) (cid:35) + E q ( Z ) (cid:34) T (cid:88) t =
K
K (cid:88) k
= log p ( y t | z t , x t , s ( k ) )
(cid:35) − E q ( Z ) (cid:34) T (cid:88) t =
KL ( q ( S | z t ) || p ( S
| z t ))
(cid:35) − KL ( q ( A, π ) || p
( A, π )) ≥ E q ( π ) [log p ( z 0 | π )]
+ E q
( Z,A ) (cid:34) T (cid:88) t
= log p ( z t | z t − 1 , A ) (cid:35) + E q ( Z ) (cid:34) T (cid:88) t =
K
[Equation]
The ﬁrst inequality holds by applying the lower bound bound in Eq.
(14) to log p ( y t | x t , z t ) in the derived ELBO in Eq.
The second inequality is derived using q ( z t = k ) ≤ 1 for all t, k .
[Equation]
where E q ( Z,A,π ) [log p ( Y, Z | X, A, π )] is the expected joint likelihood with respect to q ( Z, A, π ) using 1 K (cid:80)
Kk =1 log p ( y t | z t , x t , s ( k ) )
as log likelihood of GP emission instead of log p ( y t | z t , x t ) .
The additional KL divergence term related to KL ( q ( S | z t = k ) || p ( S | z t = k )) is computed as (cid:80)
Qq =1 (cid:80) mi =1 KL( N ( u q , σ 2 q ) ||
N ( (cid:101) µ q,i , (cid:101) σ 2 q,i )) with the parameters { µ q , σ q } Qq =1 corresponding to hidden state z t = k .
If we consider { (cid:101) µ q.i , (cid:101) σ 2 q,i } M i =2 = { µ q , σ q } , we reduce the KL regulaizer as
[Equation]
1 KL( N ( u q , σ 2 q ) || N ( (cid:101) µ q, 1 , (cid:101) σ 2 q, 1 )) .
=
E q
[Equation]
Z,A,π ) [log p ( Y, Z | X, A, π )] − T
[Equation]
N
[Equation]
, π ) || p
( A, π ))
For the corresponding approximate GP emission, we can obtain (cid:81)
Kk =1 p ( y t | z t , x t , s ( k ) )
1 k because of
[Equation]
If { s ( k ) } Kk =1 are sampled such that p ( y t | z t , x t , s ( k ) )
≈ p ( y t | z t , x t ) , the approximate GP emission (cid:81)
Kk =1 p ( y t | z t , x t , s ( k ) )
1 k becomes the original GP emission p ( y t
| z t , x t ) .
This can be feasible when the number of sampled spectral points Qm for s ( k ) is large enough for Φ SM ( x t ; s ( k ) )
Φ SM ( x t ; s ( k ) )
T to be equal to the true SM kernel gram matrix K SM ( x t , x t ) .
Appendix B. Parameters update procedure for section 2.2 Variational Inference B.1. Local Variables
The optimal local variable q ∗ ( Z ) is proportional as
[Equation]
by Mean ﬁeld approximation (Bishop, 2006; Beal, 2003).
We deﬁne the following auxiliary variables (cid:101) π as exp (cid:0) E q ( π ) [log π j ] (cid:1) and (cid:101)
A j,i as exp (cid:0) E q ( A ) [log A j,i ] (cid:1) required to evaluate Eq.
(18), which are computed as
[Equation]
where ψ ( · ) is the digamma function.
To compute the marginal distribution q ∗ ( z t = k ) and q ∗ ( z t = j, z t = i ) necessary for updating variational parameters for global variables, forward-backward algorithm known as Baum–Welch algorithm (Beal, 2003) is used.
Deﬁning α t,i = p ( z t
= i | y 1: t , x 1: t ) and β t,i = p ( y t
+1: T | z t
=
i, x t +1: T ) with α
= π and β T =
[1 , .., 1] T ∈ R K , we compute { α t , β t } Tt =1 as α t,i =
K (cid:88)
j =
[Equation]
t
[Equation] [Equation]
A
[Equation]
i
[Equation]
y t |
[Equation]
t
[Equation]
x t )
[Equation]
t
[Equation]
(cid:88)
i
[Equation]
A j
[Equation]
i p
[Equation]
y t +
[Equation]
|
[Equation]
t +
[Equation]
=
[Equation]
x t +1 )
β t +1 ,i .
These computed likelihoods { α t , β t } Tt =1 are used to compute q ∗ ( z t = i ) and q ∗ ( z t =
j, z t = i ) for t = 1 , .., T and j, i = 1 , .., K as
[Equation]
A detail derivation for Forward-backward algorithm can be found in (Beal, 2003).
Appendix C. Computational Complexity
To analyze the computation complexity for the proposed learning algorithm, we split the algorithm mainly into three parts; computation of log marginal likelihood for observations, local update, and global update.
We proceed with the analysis of our computation under a single batch assumption because the repetitive batch sampling for SVI and the spectral points sampling of Eq.
(14) increases the total computation linearly.
For the brevity, we assume N t = N for all t .
In original VI approach, computing the log marginal likelihood of T × N observations with K hidden state costs O ( KT N 3 ) .
However, our approximation approach costs O ( KLNM 2 ) where the length of the sampled sequence is L , and the total M sampled spectral points are used for SM kernel approximation.
This is because SVI approach reduces to O ( L ) from O ( T ) and SM kernel approximation reduces to O ( NM 2 ) from
O ( N 3 ) under M (cid:28) N .
For the update of local variables, VI and SVI take O ( K 2 T ) and O ( K 2 L ) for the ForwardBackward algorithm, respectively.
Updating the global variable is dominated by updating kernel hyperparameters.
Computing the derivative of log marginal likelihood for each parameter costs
O ( N 3 ) (Rasmussen, 2004).
Thus, in the case of SM kernel, VI approach costs O ((3 Q + 1) N 3 KT ) where all SM kernels take Q Gaussian mixture components.
However, our scalable approach takes
O ((3 Q + 1) NM 2 KL ) .
In summary, our learning method scalably trains the large dataset when we control M and L such that NM 2 (cid:28) N 3 and L (cid:28) T .
Appendix D. Additional results of Experiment D.1. Long Sequence ( T is large)
[Table]
[Figure]
D.2. Large number of observations ( N t is large)
[Table]
This table summarizes how the proposed kernel approximation approach (SVI-AGPE) affects the training time and the accuracy for HMM-GPSM with Q ∈ { 3 , 5 , 6 } , comparing to the SVI approach without kernel approximation (SVI).
For this comparison, we set L = 10 for SVI approach and vary the ratio of spectral points R ∈ { . 05 , .
The statistical results are obtained fromrepetitive experiments.
[Figure] [Figure]
D.3. Why is SM kernel considered
SM kernel is a ﬂexible kernel to model any stationary covariance function.
Bochner’s theorem (Bochner, 1959) states that for the inputs x 1 and x 2 ∈ R P and its distance τ = x 1 − x 2 , the stationary kernel k ( τ ) , which is invariant to translation of the inputs, can be obtained by taking a inverse Fourier transform to the corresponding spectral density of p ( S ) as
[Equation]
This theorem implies that if p ( S ) approximates well the true spectral density for dataset, the target function f using GP prior with the corresponding kernel k ( τ ) models well the true signal.
Wilson et al.
(Wilson and Adams, 2013) implement this idea to devise a new kernel known as the spectral mixture (SM) kernel.
They represent the spectral density p ( S ) using a mixture of symmetric Gaussian distribution with parameters { w q , µ q , Σ q
[Equation]
=1 as
[Equation]
where µ q = [ µ q, 1 , .., µ q,P ] , σ q = [ σ q, 1 , .., σ q,P ] ∈ R P and Σ q = diag( σ 2 q ) ∈ R P × P because a mixture of Gaussian distribution can approximate any continuous function by universal approximate theorem (Plataniotis and Hatzinakos, 2017).
Then, they obtain the SM kernel by taking an inverse
Fourier transform to p ( S ) by Eq.
(20) as
[Equation]
Flexibility of GP emission using SM kernel
We have investigated that GP emission using SM kernel characterizes the hidden state of time-series observation more ﬂexibly over GP emission using the conventional kernel including RBF, Periodic, and the combination kernel.
The considered baseline models are as follows:
• HMM-GPRBF :
HMM-GP using RBF kernel • HMM-GPPER :
HMM-GP using Periodic kernel • HMM-GPCombKernel : HMM-GP using combination kernel (RBF + PER + PER + PER) • HMM-GPSM (Q) :
HMM-GP using Q-mixture SM kernel
All models are trained by variational inference for full T sequences as explained in section 3.2.
For this experiment, we generate the sequences of time-series observations using Eq.
(35) having 150 sequences of time-series.
During the experiment, we ﬁx the number of data points in each time-series observation as N t = 200 .
We limit spectral range as W Hz = 100 to make sure that time-series with N t = 200 contains enough information to infer the spectral characteristics of the time-series according to the Nyquist–Shannon sampling theorem (Oppenheim, 1999).
For training the parameters of the model, we use ﬁrst 100 sequences of time-series, i.e T =(training set).
We run 30 iterations to update the local variables q ( Z ) and the variational parameters of q ( π ) and q ( A ) , and kernel hyperparameters for global variables.
The kernel hyperparameters are updated by the Adam optimizer with learning rate .005.
After training, we evaluate how accurate the trained models estimate the hidden state for the left 50 sequences of time-series observations (test set).
[Table] [Figure]