FlowHMM: Flow-based continuous hidden Markov models
Abstract
Continuous hidden Markov models (HMMs) assume that observations are generated from a mixture of Gaussian densities, limiting their ability to model more complex distributions.
In this work, we address this shortcoming and propose novel continuous HMM models, dubbed FlowHMMs, that enable learning general continuous observation densities without constraining them to follow a Gaussian distribution or their mixtures.
To that end, we leverage deep flow-based architectures that model complex, non-Gaussian functions and propose two variants of training a FlowHMM model.
The first one, based on gradient-based technique, can be applied directly to continuous multidimensional data, yet its application to larger data sequences remains computationally expensive.
Therefore, we also present a second approach to training our FlowHMM that relies on the co-occurrence matrix of discretized observations and considers the joint distribution of pairs of co-observed values, hence rendering the training time independent of the training sequence length.
As a result, we obtain a model that can be flexibly adapted to the characteristics and dimensionality of the data.
We perform a variety of experiments in which we compare both training strategies with a baseline of Gaussian mixture models.
We show, that in terms of quality of the recovered probability distribution, accuracy of prediction of hidden states, and likelihood of unseen data, our approach outperforms the standard Gaussian methods.
Introduction
Hidden Markov models (HMMs) are a standard tool in modeling and analysis of time series data.
Although structurally simple, they have been successfully applied in a wide variety of applications, ranging from finance [1], speech recognition [2] to computational biology [3] and climate modeling [4].
HMMs are capable of solving complex problems, but their adoption is limited due to several reasons.
First, the training process of HMMs typically relies on the Baum-Welch algorithm [5] — a particular case of expectation-maximization (EM) method, which offers a relatively slow convergence to a local maximum.
To reduce this burden, several discrete HHMs introduce the so-called co-occurrence matrix that aggregates information about the probability of jointly observed values in the chain and estimate it together with the parameters of the whole model [6, 7, 8, 9, 10].
Although the co-occurrence matrix is computed only once and then used to significantly reduce convergence time during optimization, its application is strictly limited to discrete HMMs and cannot be easily generalized to their continuous variants.
Thanks to this, they can adjust to complex, non
Secondly, continuous HMM models are restricted to follow standard parametrized distributions when modeling the observations.
Most of them use either
Gaussians or other parametric families of distributions [11, 12], while the others rely on the mixtures of Gaussians or apply semiparametric distribution estimation [13, 14].
As a result, the existing HMMs have limited ability to model complex observations that do not follow the distributions mentioned above.
This, in turn, hinders their application in real-life use cases, e.g. , in human action recognition [15].
In this work, we address the shortcomings of the existing HMMs and introduce FlowHMM, a novel continuous hidden Markov model that learns a general continuous distribution of observations by exploiting the properties of flow-based models [16].
The core idea of our approach is to model the distributions of the observations with normalizing flows, instead of Gaussians or their mixtures.
Flows map a simple Gaussian prior to more complex distributions using a parametrized neural network.
This formulation enables flow-based models to overpass parametric models in their flexibility to handle data samples of unknown distributions.
Moreover, flow-based models naturally extend to a multimodal setting, which effectively renders obsolete the tedious process of tuning the number of hidden states.
Practically, training the model with flow-based components requires a gradient-based optimization.
To achieve that, we propose two variants of training the FlowHMM model.
The first approach is based on maximum likelihood (ML) technique and can be applied directly to continuous multidimensional data.
In the second method, we discretize the continuous values during training and leverage the co-occurrence matrix.
As a result, we provide an end-to-end training procedure that jointly optimizes the flow-based component parameters and the co-occurrence matrix using standard gradient-based techniques.
Since the model is trained in a discretized form, the optimization process is simple and more efficient than the competing HMM training procedures, while during inference, we can still use a continuous version of our FlowHMM model.
To summarize, the main contribution of our paper is a novel continuous HMM method dubbed
FlowHMM with two alternative training scenarios, capable of modeling complex multimodal distributions of observations without constraining them to follow Gaussians.
Not only does it outperform the competing approaches, but it also increases the efficiency of the required optimization procedure.
Background 2.1 Hidden Markov models
Let { X k } k ≥ 0 be an ergodic, time homogeneous Markov chain over hidden states S
= { s 1 , . . .
, s L } with a stationary distribution π = ( π 1 , . . .
, π L ) , and a transition matrix A , i.e. , for any k we have P ( X k +1 =
s j |
X k = s i )
=
A ( i, j ) .
Let { Y k } k ≥ 0 be a sequence of random variables taking values in V (the observation space ), which can be continuous or discrete.
Let us fix some time horizon
T .
Random variables Y 0: T =
( Y 0 , . .
, Y T ) are independent conditionally on the state sequence
X 0: T =
( X 0 , . . .
, X T ) , i.e. : p ( Y 0: T = y 0: T | X 0: T = x 0: T ) = T (cid:89) k
= p ( y k | x k ) , (1) where p ( y | x ) ≡ P ( Y = y | X = x ) represents the so-called emission probabilities for discrete observations.
We will shortly write p ( y 0: T ) for p ( Y 0: T = y 0: T ) and similarly e.g. , p ( y 0: T | x 0: T ) for p ( Y 0: T = y 0: T | X 0: T = x 0: T ) to simplify the notation.
Considering the continuous case, p ( y | x ) represents the conditional emission density function for a given state x .
For a Gaussian Mixture HMM (which we simply call Gaussian HMM) we assume that p ( y | x ) = (cid:80)
Kk
{+=
 α+} x,k N ( y ; µ x,k , σ 2 x,k ) , where α x, · is a distribution on { 1 , . . .
, K } .
The values of parameters { α x,k , µ x,k , σ 2 x,k } Kk =1 are determined by the conditioning value x .
Note that we assume that each mixture has the same number of components K .
For K =
we have a classic Gaussian HMM.
HMM model can be parametrized by θ = { π , A , β } , where β = { β 1 , . . .
β L } , and β l stays behind the parameters of p ( y | l ) , for a given state l .
For discrete case, β l represents the parameters for categorical distribution, while for a Gaussian HMM we have β l = { α l,k µ l,k , σ 2 l,k } Kk =1 .
The probability of observing the sequence of observations y [-
 :-]0: T can be expressed as:
[Equation]
where p ( y 0: T | x 0: T ; β ) is given by Eq.
(1), and p ( x 0: T ; A , π ) can be expressed as:
[Equation]
2.2 Normalizing Flows
Normalizing flows [16] are generative models that can be efficiently trained via direct likelihood estimation thanks to the application of the change-of-variable formula.
Practically, they utilize sequence of parametric and invertible transformations:
y = h n ◦ · · · ◦ h 1 ( z ) .
The goal of the transformation is to map z from the known, normal distribution N ( z ; 0 , I ) to the more complex distribution described by a density function f ( y ) from the observation domain.
The log-probability for y can be expressed as:
[Equation]
One of the main challenges while designing the normalizing flows is selection of a proper form of transformation functions h n .
The sequence of discrete transformations can be replaced by continuous equivalent by application of Continuous Normalizing Flows (CNFs) [17], where the aim is to solve differential equation of the form dzdt =
g β ( z ( t ) , t ) , where g β ( z ( t ) , t ) represents the function of dynamics, described by parameters β .
Our goal is to find a solution of the equation in t 1 , y :
=
z ( t 1 ) , assuming the given initial state z :
=
z ( t 0 ) with a known prior.
The transformation function h β and its inverse are defined as defined as:
[Equation]
The log-probability of y can be computed by (where h − 1 β ( y )
=
z ):
[Equation]
The choice of using CNFs is motivated by the fact, that we are focused on modeling distributions of one or low-dimensional data.
CNFs were successfully applied in such models as NGGP [18],
PointFlow [19], or StyleFlow [20], where the dimensionality and characteristic of data are similar. It is somehow confirmed by the empirical results provided by the authors of FFJORD ([17] Table 2), the proposed approach performs better than discrete flows like RealNVP [21] or Glow [22] for low-dimensional data in terms of normalized log-likelihood. Such models were used in [23]. Moreover, flows that use coupling layers (RealNVP, Glow) and autoregressive flows (MAF [24]) do not make sense for 1D data.
While operating on 1D data, we do not care about simplifying the estimation of the Jacobian, and any invertible differentiable transformation can be applied.
At the same time, we need a complex, well-parameterized transformation that is delivered by a dynamic function of CNF.  FlowHMM In this section we introduce FlowHMMs - HMM variants of continuous flow capable to model the observations using complex, non-Gaussian distributions.
The idea behind this approach is to model each of conditional densities p ( y | x
= s l ) =
f β l ( y ) , for each of the considered states, s l ∈ S , using a separate CNF module.
In practice, p ( y | x
= s l ) can be calculated using formula given by Eq.
(6) with the parameters β l dedicated for the state s l .
The idea of our approach is illustrated in Fig.
In FlowHMM we assume that { X k } is stationary, i.e. , the stationary distribution π is also its initial distribution.
In such a case, instead of using A , the model can be equivalently represented by the state joint probabilities:
[Equation]
Note that π i can be computed as (cid:80) j S ( i, j ) , what can be written as π i = 1 i S1 T , where 1 denotes a vectors consisting of ones and 1 i consists of 1 on position i and zeros otherwise.
For such a formulation, the FlowHMM model can be parametrized by θ = { S , β 1 , . . .
β L } .
We propose two variants of FlowHMM models, which differ mainly in a training process: gradientbased model F ML which is trained with an maximum likelihood approach directly on continuous data, and co-occurrence matrix-based model F Q , that utilizes co-occurrence matrix and is trained in end-to-end setting using discretized sequences.
The first of the proposed methods does not require discretisation step, but is costly and ineffective for larger sequences.
F Q eliminates that problem, and makes the training time independent of the length of the training sequence (only estimating empirical co-occurrence matrix depends on T , but the time is marginal).
In upcoming sections we are going to introduce both models in more details.
3.1 Training FlowHMM F ML model
Given a sequence of observations y 0: T = ( y 0 , . . .
, y T ) , the model is trained by optimizing the log-likelihood log p ( y 0: T ; θ ) (see Eq.
(2)), where we aim at finding the optimal values of S ∗ (and thus A ∗ and π ∗ ) and parameters of flow models β ∗ l , such that θ ∗ =
{ S ∗ , β ∗ 1 , . . .
, β ∗ L } = arg max θ log p ( y 0: T ; θ ) .
In order to satisfy the constraints on S ( i.e. , that 1S1 T = 1 and each entry is non-negative, what we denote by S ≥ 0 )
we parametrize the matrix S by a real-valued matrix ˜ S also of size L × L , using the following softmax function:
[Equation]
Thanks to that representation, we train the S and β l ’s iteratively with gradient-based approach, by maximizing the incomplete log-likelihood log p ( y 0: T ; θ ) , calculated directly using the forward algorithm.
3.2 Training FlowHMM F Q model
Considering F Q model we introduce the co-occurrence matrix that represents the joint distribution of two consecutive observations: Q ( y 1 , y 2 )
= p ( Y k +1 =
y 2 , Y k = y 1 ) . The matrix represents a categorical distribution, ( i, j ) -th entry represents the probability of observing a pair of states ( v i , v j ) at some fixed two consecutive steps k and k + 1 . Note that it is independent of k , since throughout the paper we assume that underlying Markov chain on hidden states is stationary, thus a bivariate distribution of ( X k , X k +1 ) is independent of k . The matrix can be rewritten as:
[Equation] [Equation]
Assuming the discrete set of observations, V = { v 1 , . .
. , v M } , it can be further expressed as Q = P T SP , where P collects probabilities of all possible observations at each hidden state in a matrix
P ( s i , v j ) = p ( v j | s i ) .
In this case there are M 2 possible observation pairs ( y 1 , y 2 ) ∈ V × V .
Note that matrices Q , S , P are of sizes M × M , L × M and L × L , respectively.
Moreover, we have (cid:80) v i ,v j ∈V Q ( v i , v j )
=
(cid:80) s i ,s
j S ( s i , s j ) = 1 and (cid:80) v ∈V P ( s i , v ) = 1 for all s i ∈ S .
In a matrix form, these can be written as 1Q1 T = 1S1 T = 1 and P1 T = 1 T (note that 1 must be of an appropriate size).
Given observations y 0: T the matrix Q can be empirically estimated by:
[Equation]
for all pairs ( v i , v j ) ∈ V × V .
The problem of training the HMM is to find such parameters (matrices
S and P ) so that the co-occurrence matrix Q is close (in some sense) to the empirical co-occurrence matrix ˆ Q .
We can formulate the problem as (for some distance dist )
[Equation]
This problem formulation has a couple of advantages compared to standard likelihood-based optimization.
First, the empirical co-occurrence matrix ˆ Q is independent of the sequence length T .
Second, the given objective can be easily optimized using gradient-based techniques.
On the other hand, the constraints for matrices P and S should be satisfied.
The set V is discrete, while we aim to design the model for continuous observations.
In order to satisfy the constraints for matrix S we use the representation given by Eq.
In order to represent matrix P we use Flow-based emission probabilities.
Flow-based emission probabilities.
With each hidden state l we associate the flow model described by a density function f β l , that can be calculated from Eq.
(6), where β l is a set of trainable parameters of the flow.
We construct P based on these models (in such a case P ≡ P β ).
The i -th row of the matrix is a density f β i ( · ) evaluated at v 1 , . .
. , v M and normalized as follows:
[Equation]
Similarly, the optimization problem (11) becomes:
[Equation]
As a distance function, we propose to use Kullback–Leibler divergence ,
[Equation]
where Q β = P T β SP β .
We postulate to apply divergence instead to L 2 distance used e.g. , in [7]., because it is more natural measure for comparing two distributions.
We observed during empirical evaluation, that using this divergence gives better stability and convergence of the training process.
The final training procedure of FlowHMM F Q model is as follows.
Let y train0: T be the training set and let y test0: T ′ be the test set.
For the fixed M we construct the grid Γ =
( γ 1 , . . . , γ M ) using one of the approaches described in A.1.
Next, we create the discretized training data y train , Γ 0: T , where y train , Γ i
[Equation]
|| 2 .
The values of the grid represent further the set V , V = Γ .
Next, we calculate the empirical co-occurrence matrix ˆ Q :
[Equation]
and matrices S and P β (using Eqs.
that are further used to calculate the loss given by Eq.
Practically, we add Gaussian perturbations to the grid values, while calculating the matrix P :
[Equation]
where ϵ is a random sample from N (0 , σ 2 noise ) , and σ 2 noise is a hyperparameter of the method. This is one of the standard tricks applied while training normalizing flows, that imitates the situation where we have an access to an infinite number of training examples.
These perturbations prevent from overfitting of the flow-based components caused by observing the same grid while training.
Next, all of the parameters of FlowHMM are updated with gradient based approach.
The procedure is repeated until convergence.
The procedures of training is presented in Algorithm 1.
Require: ˆ Q : — empirical co-occurrence matrix from Eq.
Parameters: β = { β 1 , . .
. , β L } — flow parameters, ˜ S parameters representing un-normalised co-occurrence matrix.
Hyperparameters: L - number of hidden states, α - step size, noise variance σ 2 noise .
1: function T RAIN ( ˆ Q , L , α ) 2:
Initialize β , and ˜ S .
3: while not convergent do 4:
Calculate S from Eq.
and P β from Eq.
Calculate loss function L from Eq.
(14) using ˆ Q .
6: ˜ S ← ˜ S − α ∇
˜ S L 7: for each l ∈ { 1 , . . .
, L } do 8: β l ← β l − α ∇ β l L 9: end for 10: end while 11: return β 1 , . . . , β L , ˜ S 12: end function
As we postulated before, our model is designed for continuous problems, due to the fact, that both of the training techniques return estimated S , and the parameters, that represent emission distributions for each of the states. Thus, during the inference stage, we simply calculate A and π from S and apply forward procedure on the test data y test0: T ′ to calculate p ( y test0: T ′ ; θ ) . We can also determine the hidden state values using the Viterbi procedure. Concluding, FlowHMM can be used in the same applications as standard continuous HMM models, but with no restrictions to emission distributions.
FlowHMM for multidimensional observations. Our approach can be easily extended to multidimensional observations, i.e. , to the case where y train0: T are from R d .
For such a case, the F ML model is straightforward, let us focus on a description of F Q .
On one hand the extension is straightforward: we construct some grid Γ =
( γ 1 , . . . , γ M ) of d -dimensional points, we create discretized training set y train , Γ 0: T and the empirical co-occurrence matrix ˆ Q from (15), afterwards we compute P β from Eq.
(16) and the gradient ∇ β l L of L given in (14).
In other words we proceed with an Algorithm 1.
Note that CNFs are well-suited for multivariate observations.
On the other hand, there is a challenging aspect of training F Q model for multidimensional data: an effective discretization technique.
We elaborate on that in A.1 proposing a new grid search method.
We suggest using F Q model for lower-dimensional data, especially with long observation sequence (recall, the matrix Q is computed once) – in such a case F ML will usually be very slow, each epoch loops through a whole observation sequence.
On the other hand, for F ML is better suited for high-dimensional data and short observation sequence – for longer observation sequences, in order to shorten the execution time, we apply small trick , at each epoch we sample a subsequence of fixed length, e.g. , 10 3 .
There is a trade-off between execution time and performance, see Table 7 in A.7.
Related work
One of the most popular way to train HMM models is the Baum-Welch algorithm (a name for EM applied to HMMs).
However, one of (several) drawbacks of using Baum-Welch algorithm for training is that it is prohibitively slow for long sequences.
The complexity of the forward-backward algorithm (which must be run for each epoch) is O
( N 2 T ) , where N is the number of hidden states and T is the length of the observation sequence.
In order to eliminate this issue, the authors [6] propose to use pairwise co-occurrence probabilities (also higher order statistics are also discussed there) and prove that it is possible to recover the structure of an HMM based only on co-occurrences.
An interesting extension (both, of pairs and triplets) was considered in [25].
Authors used there non-consecutive tuples, which outperforms consecutive tuples in some cases.
Our approach is, in a sense, close to the technique used in [7], with however significant differences: authors use alternating least square methods for optimization (in our case, all the matrices are realvalued and, softmax is applied whenever needed) and for the continuous case, they assume that the observation densities are a mixture of the predefined number of kernels.
The "softmax trick" is also used in [9] (their model learns so-called dense representations of hidden states and observation probabilities – it is designed only for discrete HMMs).
As already mentioned, typically continuous HMMs assume that observations were sampled from a Gaussian distribution or a mixture of such distributions.
For a non-Gaussian distribution, one usually assumes some parametric family of distributions.
In [11] authors propose a decoupling method to learning the parametric HMMs which are stationary.
Instead of estimating the parameters of hidden states and observations jointly, they learn the parameters of observation densities using some parametric mixture learner, and then hidden states by solving some convex quadratic programming problem.
In [13] a method is proposed for a case where at least one state-dependent distribution is modeled with some nonparametric technique ( e.g. , maximum likelihood estimation under shape constraints), some broader review of methods is presented in [26].
In [14] authors use a Gaussian copula to model the dependence structure.
A semiparametric data transformation is also proposed to ensure one may indeed use such copulas (the final observation distribution is a finite mixture of the copula models).
In [27] authors present an efficient learning for parametric continuous HMMs sampled at finite irregular time instants (they incorporate some ideas from the theory of continuous-time Markov chains).
In [28], the authors consider the extension of a Kalman filter (which you can think of as a version of a HMM with continuous both, observations and hidden states), where the pseudo-observations (as authors call it) are transformed through a normalizing flow producing the actual observation.
The most similar model to our approach was introduced in Ghosh et al. [29] and further developed in [23], where the authors employ normalizing flows in observation densities and apply it to classification problems.
However, they utilize the flow with coupling layers that cannot be directly applied to one-dimensional data.
We use CNFs that can be easily used for any type of data, and are characterized by better quality for low-dimensional examples compared to coupling-based flows.
Moreover, their approach relies on EM training, which makes it impractical for longer sequences of data.
We propose two variants of training models: a) F Q model which relies on co-occurrences of discretized data and can successfully be applied on very long sequences, and b) F ML model which applies gradient-based optimization not only to the parameters of the flow (as in [29] and [23]), but also to the transition matrix on hidden states (which - because the underlying chain is stationary – is uniquely determined by the matrix representing joint states probabilities and parametrization given in Eqs.
(8) and (9)) – consequently, in each iteration, only one forward step is needed, while Ghosh et al. model requires two forward passes (one to fit frozen weights (no gradient) and one gradient-based to update weights).
For one-dimensional examples we show the advantages of the proposed F Q empirically (compared to F ML model).
Finally, the authors of [29] apply their model only on sequence classification as black box .
At the same time, we investigate the proprieties of specific examples in order to discover the capabilities of distribution adjustment and discover the hidden states of data unobserved during training.
Experimental results
[Figure]
Here we consider our
FlowHMM models F ML and F Q with training procedures described in Sections 3. and 3.2 respectively.
We compare our results with classic HMM models utilizing Gaussian mixtures observation densities with K = 1 , 10 , 20 components; we denote them by G ( K ) and for simplicity we write G ≡ G (1) .
We used hmmlearn library 1 for the computations of all G ( K ) models and provide the source code for FlowHMM models 2 .
We evaluate the quality of the proposed methods computing
Normalized log-likelihood ( normLL ) of unseen observations, Total variation distance ( d tv ) between learned and true emission probabilities, and accuracy of predicted hid- T Example Example 2a
[Table]
[Figure]
[Table] den states ( accuracy ). We define all the metrics and motivate their choice in A.2, the experimental settings are described in details in A.3. 1D Synthetic sequences.
We consider HMMs with L = 3 hidden states and experiment with transition matrices A 1 (used in [7, Eq.
(11)]) or A 2 , where
[Equation]
We consider 2 examples.
Example 1 - transition matrix A 1 and emission probabilities: two Gaussian and one Uniform.
Example 2a - transition matrix A 2 , emission probabilities: one Gaussian, one
Uniform, and Beta distribution.
In addition, we consider Example 2b , where we aim at adjusting the models assuming only two hidden states.
The examples are described in details in Appendix A.4.
For each example we sample train and test observations of the same length T ∈ { 10 3 , 10 4 , 10 5 } .
We train F ML and F Q models using 500 (Example 1) and 1000 (Examples 2a and 2b) epochs.
For F Q we used a grid of size M = 30 .
For each case ( i.e. , fixed example and T ) we perform 10 simulations and report means of normLL s and total variation distances in Table 1.
For observations of length
T ≥ 10 4 flow models clearly outperform Gaussian ones in all the examples.
Note that in these examples we had a grid of size M = 30 , thus we estimated the matrix Q with 900 entries (number of observation pairs).
It is intuitively clear that T = 10 3 observations is not enough then for F Q .
In this case F ML outperforms F Q in terms of normLL s; models are comparable for larger T .
For T ≥ 10the F Q model "recovers" original distributions much more accurately, the small values of d tv from
Table 1 are confirmed in Fig.
2, where distributions learned by our flow models trained with T = 10observations are depicted (from single simulations).
In A.4 in Fig.
5, the trained distributions for
Examples 1, 2a-b trained with T = 10 3 observations are depicted. In A.7 we investigate the impact of length T on quality of the model.
We encourage the reader to confront Fig.
2 for F Q and T = 10 4 for Example 1 with [7, Fig. 4].
In Fig.
3 we depict the distributions learned models with only L = 2 hidden states for T = 10observations (recall observations were sampled from a model with 3 different distributions).
Since flow models are capable of fitting multimodal distributions, they outperform Gaussian ones.
Moreover, the co-occurrence matrix-based model F Q outperforms the maximumum-likelihood-based model F ML . Recall that, by construction, our models F ML and F Q assume that the underlying Markov chain is stationary.
We want to point out that in our examples we used at most L = 3 states. In such a case, even if the assumption of stationarity does not hold, it does not have a large impact on results. This is because for such chains the rate of convergence to stationarity is exponential (in number of steps). To be more specific, e.g. , for the chain with t.m.
A 2 given in (17), after 10 steps the total variation between the stationary distribution (which is π (1) = 5 / 14 , π (2) = 4 / 14 , π (3) = 5 / 14 ) and the distribution of X 10 is at most (for any initial distribution) 4 .
22 · 10 − 6 .
[Table]
Real datasets.
Example 3.
S&P 500 and Dow Jones are popular measures of market performance.
We retrieved data [30] for a period of 9/1977-8/2017 that consist ofpoints.
We trained models on T = 1000 observations (and computed normLL s for the remaining data).
Example 4.
We used mean air pressure from [31] dataset collected between 02/15/18 and 02/28/19.
There were 54 531 observations, the models were trained on the first
T = 40 000 ones and tested them on the remaining observations.
In both, Example 3 and Example 4, we applied a standard difference transform (to remove a trend), i.e. , we considered time series y [-
-]′
k = y k − y k − 1 .
We trained the models with L ∈ { 2 , 3 , 4 } hidden states, models F ML and F Q were trained for 500 epochs, for the latter we used a uniform grid of size M = 30 . During training F ML model we randomly sampled subsequences of length 10 3 for each epoch.
The values of normLL are reported in Table 2.
As can be seen, in all cases F Q yields the best results and for Example 3 model F ML is second best.
Note also that although a number of observations for Example 3 is not large (we used only T = 1000 observations, while for M = 30 we have 900 observation pairs)
F Q outperforms F ML .
Fig.
6 in A.4 shows that the learned observation densities for L ∈ { 2 , 3 , 4 } are non-Gaussian.
We want to remark one thing: in financial data one usually works with log-returns log( y k /y k − 1 ) , which tend to be " more Gaussian ". This is also the case with the examples we considered: F ML , F Q and Gaussian models returned very similar normLL s (which is no surprise in case of Gaussian data, other models easily fitted to them). We chose and reported the differences y k − y k − 1 to compare the models in case of non-Gaussian data.
2D synthetic sequences.
We consider HMM models with L = 3 states.
We consider 2 examples:
Example 5 with emission probabilities: two " Moons " and one Uniform and Example 6 with emission probabilities: one bivariate Gaussian, one Uniform and one related to geometric Brownian motion.
Details provided in A.5.
In both examples we consider the transition matrix A 1 (variant (a) ) and A(variant (b) ), defined in (17). [Table] Example 6a Example 6b normLL
[Table]
[Table]
[Table] [Table] [Figure] We compare Gaussian baseline models, our F ML and F Q models, as well as a model from Ghosh at al.
[23] – we used authors’ official implementaion 3 .
For the latter we computed only normLL s.
In Table 3 values of normLL (for all models) and accuracy (for all models except Ghosh at al.
[23]) are reported, whereas in Fig. 4 the trained distributions for Example 5a for T = 10 4 are depicted.
Similar plots for Example 6a are presented in A.5 in Fig.
We used non-uniform grids of size M = 30(cartesian product of two one-dimensional grids of size 30 on each coordinate – for other strategies for choosing a grid see A.1)
One can observe that for all examples except Example 5b model F ML gives the best normLL s, Ghosh et al. outperforms our models only in Example 5b. For 2D examples we can spot a drop of quality – in terms of normLL – for F Q model. We performed additional simulations for this model for T = 10 3 observations and grids of sizes 35 2 and 40 2 , the results are provided in Table 4.
[Table]
As we can see they are still worse than F ML model (which is no surprise: we have 30 4 , 35 4 and 40 4 pairs and only 10 3 ), but we see a tendency: increasing a grid size increases normLL in all cases.
One can observe that in all the considered cases and variants, model F ML gives the best normLL values. When it comes to the other metric, i.e. , the accuracy , all the classic Gaussian mixture models, as well as F ML , yield almost the same accuracy if transition matrix A 1 was used (variant (a) ).
However, for the transition matrix A 2 (variant (b) ), the models F ML and F Q outperform the classic models with significant margin (note that A 2 yields a "more complex" model compared to A 1 ). [Table] High dimensional (6D) real dataset. Example 7 : from [31] dataset (used in Example 4) we chose 6 features related to humidity ( mean, stdev ) and air pressure ( min, max, mean, stdev ).
Similarly as in Example 4 we considered differences of the observations. Again, the length of training set was set to 40 k , the remaining ∼ 14 k observations constituted a test set. We used F ML model – in each epoch we randomly sampled a subsequence of length 10 3 . In all cases ( L = 2 , 3 , 4) the model significantly outperfoms Gaussian models, as can be seen in Table 5.
Conclusions and limitations
In this work, we proposed a continuous hidden Markov model that leverages deep flow-based network architectures to model complex, non-Gaussian distributions.
Although our approach outperforms several baselines and competing approaches, it relies on the co-occurrence matrix that can be only computed with the assumption that the Markov chain on a set of hidden states is stationarya limitation not present in the EM-based approaches.
Addressing this constraint, e.g. , by re-computing temporally stationary co-occurrence matrix during training, is part of our future work.
So is in-depth validating our model on multivariate time series or distributions, where our flexible approach can potentially offer more benefits over the existing works.
While we do not identify any straightforward negative societal implications of our work, we acknowledge that its application to large data corpora can lead to significant energy consumption, despite our main contributions being focused on increased efficiency. Acknowledgements This work was supported by Foundation for Polish Science (grant no POIR.04.04.00-00-14DE/18-00) carried out within the Team-Net program co-financed by the European Union under the European Regional Development Fund, as well as the National Centre of Science (Poland) Grant No. 2020/39/B/ST6/01511.
The work conducted by Maciej Zieba was supported by the National Centre of Science (Poland) Grant No. 2021/43/B/ST6/02853.
For the purpose of Open Access, the author has applied a CC-BY public copyright licence to any Author Accepted Manuscript (AAM) version arising from this submission.
A Appendix A.1 Continuous observations and discretization.
For the purpose of the training process of F Q , the continuous sequence of observations must be discretized first.
Assume our observations are from R d , d ≥ 1 .
Let Γ =
( γ 1 , . . . , γ M ) , γ i ∈ R d , be some grid of size M .
The grid may be predefined, created uniformly, or may represent the centroids of clustering approach (of the training set).
Having continuous observations y 0: T = ( y 0 , . .
, y T )
we round each y i to the nearest point from Γ obtaining a discretized sequence y Γ0: T = ( y Γ0 , . . .
, y Γ T ) , where:
[Equation]
The values of the grid represent the set of discrete observations V , i.e. , V = Γ .
The simplest uniform grid would be following: take min and max of each coordinate of observations and for each dimension construct a uniform grid of size m .
Then as the final grid we may take a Cartesian product of this one-dimensional grids.
The resulting grid would be of size M = m d and the co-occurence matrix ˆ Q would have m 2 d entries.
On one hand, m cannot be too small, since the discretised observations should be fairly close to original ones.
On the other hand, m cannot be too large, since the size of ˆ Q would be prohibitively large then.
That is why we use the uniform grid for our univariate examples (Example 1 and 2a-b), whereas we propose (and use for two-dimensional Examples 5 and 6) the following strategy.
Strategy for grid choice .
Fix some relatively small m and ρ ∈ (0 , 1) .
{+For simplicity assume that m 
+}
= ρm is an integer.
Let y train0:
T ( i ) denote a vector of i -th feature of observations, from which we construct a one-dimensional grid
Γ i,a = ( γ i,a 1 , . . .
, γ i,aρm 0 ) , taking γ i,a= min( y train0: T ( i )) and γ i,aρm = max( y train0:
T ( i )) .
Additionally, construct a grid Γ i,b in the following way: perform k -means on y train0:
T ( i ) with k = (1 − ρ ) m and take the centers of the clusters as the grid.
Then we set Γ i
=
Γ i,a ∪ Γ i,b .
As the final grid we take the Cartesian product
[Equation]
In this strategy we still have M = | Γ | = m d , however m can be significantly smaller than in the uniform case.
We always assure that there are some relevant points (close to continuous observations) and some far away points (we need e.g. , pairs of relevant and far away points for proper training).
A.2 Metrics used for evaluation
Normalized log-likelihood.
In all the examples, flow model F Q is trained on y train , Γ 0: T , whereas F ML and Gaussian models are trained on y train0: T .
For each model we compute normLL – the normalized log-likelihood of continuous observations y test0: T ′ , i.e. , log p ( y test0: T ′ ; θ ) /
( T ′ + 1) (see Eq.
To compute it for flow models we used the standard forward algorithm.
Moreover, for univariate synthetic examples (for which we know the observations densities p ( y | s i ) )
we also determine how well the model learned the observation densities and how well it inferred the hidden states (since we know the actual hidden states at each step).
Total variation distance.
Note that we may evaluate f β l ( y ) at finite number of points y ∈ R , we must thus do it on some grid.
We will use another, more dense, uniform grid Γ ′ =
( γ ′ 1 , . . .
, γ ′ r ) , where γ ′ 1 = γ 1 , γ ′ r = γ M .
We take r = 10 M in all the examples.
For two continuous distributions g 1 and g 2 we compute the total variation distance as
[Equation]
[Equation]
(where we actually normalize g 1 and g 2 so that they are distributions on Γ ′ ).
To compare distributions p ( ·| s i ) with with f β i ( · ) , i = 1 , . .
, L ( i.e. , with distribution learned by flow models), we compute the mean of total variation distances
[Equation]
Similarly, for Gaussian models we compute
[Equation]
where { α i,k , µ i,k , ( σ i,k ) 2 } Kk =1 , i = 1 , . . .
, L are the parameters learned by G K .
Accuracy of predicted hidden states.
For synthetic examples we know the true hidden states x test0: T .
We implemented the Viterbi algorithm to compute predicted hidden states x pred0: T for our F Q and F ML models, then we simply compute the accuracy between x test0: T ′ and x pred0: T ′ .
A.3 Experimental settings
We use ADAM [32] as an optimizer with lr = 0 .
01 , default betas (0 . 9 , 0 . 999) , and weight decay equal 0 . 0001 .
In all experiments, we use two blocks of CNFs with concatsquash layers, batch norm, and the configuration of the hidden neurons equal 16 − 16 .
A.4 Examples 1D - details
We sample the observations from the following distributions: uniform distribution U ( a, b ) , normal distribution N ( µ, σ 2 ) , and
beta distribution Beta( α, β ) with shape parameters α > 0 , β > 0 .
The latter has the density function p ( y )
∝ y α − 1 (1 − y )
β − 1 for y ∈ (0 , 1) .
Example 1: we used p ( ·| s 1 ) ∼ N (11 , 2) , p ( ·| s 2 ) ∼ N (16 , 3) and p ( ·| s 3 ) ∼ U (16 , 26) ,
Example 2: we used p ( ·| s 1 ) ∼ Beta (7 , 1 . 1) , p ( ·| s 2 ) ∼ U (0 . 2 , 0 . 4) and p ( ·| s 3 ) ∼ N (0 . 6 , 0 . 0046) .
To be consistent with discretization in [7], in Example 1, we take the integer grid points γ = { 0 , 1 . . .
, M − 1 } , while in Example 2, we consider uniform grid Γ = ( γ 1 , . . .
, γ M ) with γ 1 = min( y train0: T ) and γ M = max( y train0: T ) ; we set the grid size M = 30 .
In Fig.
5 we present the trained distributions for Examples 1, 2a-b for models trained on T =
10observations.
It is worth noting that the distributions learned by G 10 are similar to those learned by G .
More precisely, each mixture of G 10 is, roughly speaking, a mixture of 10 almost identical normal distributions with almost equal weights.
Recall, in Fig.
2 the distributions learned for Examplesand 2a for T = 10 3 were presented.
In Fig.
6 the distributions trained by F Q for Examples 3 and 4 are depicted.
A.5 Examples 2D
We consider HMM models with L = 3 hidden states and the observations sampled in the following ways:
Example 5:
• p ( ·| s 1 ) and p ( ·| s 2 ) are sampled using make_moons (which we call " Moons ") from sklearn.datasets library, • p ( ·| s 3 ) is a uniform distribution on [0 . 8 , 1 . 5] × [ − 1 , 1] .
Example 6:
• p ( ·| s 1 ) ∼ N ( µ , Σ ) with µ = (0 . 5 , 0 . 5) T , Σ
= (cid:18) 0 .
1 − 0 .− 0 .0 .(cid:19) , • p ( ·| s 2 ) is a uniform distribution on [0 . 25 , 0 .
5] ×
[Figure]
[Figure]
[Figure]
• p ( ·| s 3 ) is a density of ( S (0 . 5) , S (1)) , where S ( t ) is a geometric Brownian motion, i.e. , S ( t ) =
S (0) exp( µ ∗ t + σB ( t )) , and B ( t ) is a standard Brownian motion; we used the parameters: S (0)
= 0 .
3 , volatility σ = 0 .
5 and interest rate r = 0 .
µ ∗ =
− σ 2 /
All observations were centered – centers were computed from the training set and then they were subtracted from both, the training and test set.
In both examples we used strategy for grid choice described in A.1 with m = 30 and ρ = 0 . 5 .
Thus, the grid was of size M = 30 2 = 900 .
In Fig.
7 the trained distributions for Example 6a for models trained with T = 10 4 observations are depicted.
Recall, similar plots for Example 5a were presented in Fig.
A.6 Training time evaluation
We analyse the training time, by comparing the time/per epoch in sec.
between F Q , and F ML .
The experiment was performed on NVIDIA GeForce RTX 2070 8GB .
The results of the test are provided in Table 6.
The F ML can be efficiently trained for T ≤ 10 3 , therefore, for T > 10 3 we randomly sample subsequences of a given size T = 10 3 at each step.
[Table]
{+10 1 0.366 ( ± 0.031) 0.
+}
( ± 0.038) 10 2 0.414 ( ± 0.015) 0.529 ( ± 0.041) 10 3 0.436 ( ± 0.034) 1.070 ( ± 0.021)
[Table]
A.7 Impact of T length
To investigate the influence of length of a sampled subsequence in each epoch of F ML training, we additionally tested what happens in cases of lengths 400 , 2 k, 4 k, 10 k, 40 k (note, the latter actually means that all observations were used in each epoch).
The results for 500 epochs are reported in
Table 8.
As we can see, the case of T ′ =
2 k is the best in case L = 3 , 4 and T ′ =
40 k in case
L = 2 .
It is worth noting that in this case F ML normLL s are worse than those of F Q .
This is even the case if we perform more epochs: for L = 2 and training F ML for 2000 epochs and T ′ =
40 k we obtained normLL =0.756, whereas, as reported in Table 2, F Q yielded normLL =0.823.
At the same time, training F ML took 16 hrs 22 min, whereas training F Q took 4 m 49 s (the differences for
L = 3 or L = 4 are of similar order).
To investigate if F ML is more competitive with longer samples sequences and fewer epochs, we additionally trained model for 100, 200, 300 and 500 epochs and T ′ =
40 k .
The resulting normLL s and execution times are reported in Table 7.
[Table]
[Table]
In a sense
[Table]
training for fewer epochs and on whole observation sequence is competitive: e
[Table]
g. , forepochs normLL s are better, the price is longer execution time: for L = 3 or L = 4 it is c
[Table]
a. 5 times larger, whereas for L = 2 it is c.a. 26 times larger.
B Broader impact
HMM models are still applied on many domains in research and industry, they achieve competitive results compared to heavy deep learning models.
They may have very positive impact in drug discovery, or various medical applications.
On the other hand they can be used in some inappropriate ways, for instance in surveillance tools.
Moreover, we propose effective method of training our model, that reduces the computational costs for larger sequences.