Paper: On the Identifiability of Switching Dynamical Systems (ICLR-2024)

Reviewer comment (Theory):
Does the number of states need to be specified as prior information?

Author response:
> Does the number of states need to be specified as prior information?

TL;DR: No.

Just as in the non-temporal finite mixture case (e.g. Gaussian Mixture Model), the number of states **is identifiable from observations**, both for the MSM and SDS. This information can be found in **Definition 3.1**, where the notion of an identifiable MSM requires the number of states to be identifiable.

Note that other recent ICA works which use discrete latent variables (HMM-ICA [2], IIA-ICA [3], or SNICA [1]), rely on Gasiat et al. (2016) [4], which requires the number of states to be known a priori. Therefore, our result here is much more stronger.

[2] Hälvä, H. and Hyvarinen, A., 2020, August. Hidden markov nonlinear ica: Unsupervised learning from nonstationary time series. In Conference on Uncertainty in Artificial Intelligence (pp. 939-948). PMLR.

[3] Morioka, H., Hälvä, H. and Hyvarinen, A., 2021, March. Independent innovation analysis for nonlinear vector autoregressive process. In International Conference on Artificial Intelligence and Statistics (pp. 1549-1557). PMLR.

[4] Gassiat, É., Cleynen, A. and Robin, S., 2016. Inference in finite state space non parametric hidden Markov models and applications. Statistics and Computing, 26, pp.61-71.

Reviewer comment (Theory):
Whether the number of states is identifiable and how to select it practically.

Author response:
> Whether the number of states is identifiable and how to select it practically.

As mentioned, the number of states is identifiable. Selecting the number of states in practice is a different notion than identifiability, which refers to the estimation method. 

As an example, we can think about the equivalent non-temporal case where we need to find the number of mixture components in a clustering problem. One can then use the MLE objective with models initialized with different numbers of states. The identifiability conditions paired up with the general consistency of the MLE solution guarantee that the model with the correct number of states will have the highest data log-likelihood.

Note that this only works when the number of states is identifiable.

Reviewer comment (Theory):
It would be better if there has a simple example to show the identifiability of the model in theory.

Author response:
> It would be better if there has a simple example to show the identifiability of the model in theory.

In the first paragraph of section 3, we refer to our contribution as the temporal extension of a finite mixture model. In section 3.1, we adapt the notation of an MSM to finite mixture models, making the analogy where we now have a mixture of K^T trajectories. 

“This notation shows that the MSM extends finite mixture models to temporal settings as a finite mixture of $K^T$ trajectories composed by (conditional) distributions … “

With this in mind, the identifiability of the MSM is close to the identifiability of a finite mixture model. Direct application of finite mixture model theory shows that linear independence of the $K^T$ mixings is a sufficient condition for identifiability. This explanation is depicted in the Proof sketch of **Theorem 3.1**, where we give the reader an overall idea of the proof technique, and how to prove linear independence using induction.

Reviewer comment (Writing):
Moreover, the definition of the identifiability in this work should be stated clearly.

Author response:
> Moreover, the definition of the identifiability in this work should be stated clearly.

We thank the reviewer for the suggestion. However, we believe this notion is already stated in Definitions 3.1 and 3.3, where we state the definition of identifiable Markov Switching Models, and the identifiability of prior distributions up to affine transformations respectively.

For functions parameterized with neural networks, we refer to the paragraph after **Theorem 3.1**, second sentence, where we mention using analytic neural networks: “... In the latter case, the identifiability result applies to the functional form only, since network weights do not uniquely index the functions that they parameterise …”. For further clarification, we will update the manuscript re-stating this latter notion of identifiability on neural network functions for the SDS as well.

Reviewer comment (Novelty):
**The identifiability approach/results follow quite directly from previous works with some results already known previously** -- one could have assumed the results here to be already implied based on earlier works: the approach for proving MSM using mixtures model presentation is well known from the general work in [1] (which could be cited). While this specific class of models covered in this work is not explicitly mentioned, the results therein could be seen as already implying the identifiability of this type of models -- more explicit acknowledgement that this approach has been taken before would probably be sufficient (for the MSM results). The identifiabiltiy of SDSs with nonlinear activation function is just an application of the Kivva paper on this specific model class. Further identifiability of SDSs (in nonlinear ICA form) has actually already been established -- below point covers this more.

Author response:
> The identifiability approach/results follow quite directly from previous works with some results already known previously

We are aware of Allman et al. (2009) results, which directly lead to HMM identifiability in Gassiat et al. (2016). In fact our initial attempts followed Allman et al. (2009), but in the end we found it not applicable to non-parametric MSMs due to the following reasons. 
- The Allman et al. (2009) technique is based on expressing the marginal distribution as a mixture of tensors, where each tensor is an outer product of 3 vectors, and each of these vectors is effectively $p(z_t | s_t)$. Therefore this technique can only be applied to HMMs where the emission probability is dependent on the discrete state only. On the other hand, MSMs introduce explicit dependencies on the data via the transition dynamics $p(z_t|z_{t-1},s_t)$.
- The Allman et al. (2009) technique needs to have at least $T=3$. It won’t work for $T=2$ and it’s easy to give counterexamples here. In contrast, our technique for MSM works for $T=2$ under our assumptions.
- The key restriction of Allman et al. (2009) technique (see its Section 4), as well as in many subsequent papers including Gassiat et al. (2016), comes from the use of Kruskal’s “Three-Way Arrays” [1]. 

Observing the limitations of Allman et al. (2009), we decided to go back to the very first finite mixture model identifiability result (Yakowitz & Spragins, 1968) as the starting point. This leads to our key and novel theoretical result (Lemma B.1) which is crucial for our MSM identifiability proof in the non-parametric case.

[1] Kruskal, J.B., 1977. Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics. Linear algebra and its applications, 18(2), pp.95-138.

Reviewer comment (Novelty):
**There are lacking citations and incorrect interpretations of previous works** -- if these are acknowledged appropriately the contributions of this paper would appear less significant. Most importantly, this paper claims that previous works have not considered identifiability in nonlinear dynamical systems with nonlinear emission function: specifically the authors state that "In contrast, Hälvä et al. (2021) assumes linear SDSs," -- this is not correct, their identifiability results do not appear to make any such assumptions (based on my reading of their Theorems 1 and 2); their practical algorithm does seem to assume that but that has nothing to do with the identifiability. Identifiability of SDSs is thus achieved in their work. For identifiability of models with autoregressive transitions, see also [2], and the IIA-HMM model particularly -- I don't see this paper mentioned and discussed even though it has some similarity. In terms of estimation algorithm, the seminal work of [3] is not mentioned -- you need to at least explain why their work does not apply here or why the current approach is superior.

Author response:
> There are lacking citations and incorrect interpretations of previous works.

We thank the reviewer for carefully comparing our work to existing literature. We will make sure previous work is properly acknowledged (including Morioka et al. (2021)) and modify the reference to other studies using general SDSs (which can be covered by Halva et al. 2021) by highlighting the assumptions on temporal dependence, rather than the assumptions on the transition model. 

Nonetheless, we believe our contribution is significant even though other works have studied similar models.
- The SDS assumptions of Halva et al. (2021) are a combination of the ones used for HMM-ICA (Halva et al. 2020), and Hyvärinen et al. (2017) [2], thereby inheriting their limitations.
  - Regarding HMM-ICA (Halva et al. 2020), the discrete states $s_t$ are assumed to follow a first-order stationary Markov chain with a full-rank transition matrix. The number of states are also required to be known. Instead, **our work makes no assumptions on knowing the number of states, or the structure of the dynamic model of $s_t$,** meaning it can be e.g., non-stationary and/or higher-order Markov.
  - Regarding Hyvärinen et al. 2017 [2], while it is true that they do not impose linearity the assumptions are placed on the temporal dependencies between sources. Our strategy works in a different spirit, where we directly establish restrictions on the transition mapping itself, which is more intuitive.
- The identifiability results of Halva et al. (2021), although stronger in terms of achieving disentanglement of the sources, are weaker in terms of identifying the transitions in the latent space. 
  - Their main results (Thms 1 & 2) prove the identifiability of the non-linear mixing and source distribution, but make **no identifiability statements about the transition dynamics**. 
  - The Linear SDS assumed in Halva et al. (2021) is overcomplete ( observation dimensions < latent dimensions), which is problematic in terms of achieving transition identifiability.
- We believe the following claim “Identifiability of SDSs is thus achieved in Halva et al. (2021)” is not accurate. 
  - Identifiability results for HMMs include identifying the discrete transitions and source emission [3] (which would be transitions for SDSs). Halva et al. (2021) provides no result in this regard for SDS. 
  - Existing HMM identifiability results do not transfer to MSMs due to the explained limitation of Allman et al. (2009) technique. 
- Morioka et al. (2021) provides transition identifiability for vector autoregressive models (VAR), where only one transition function is used. On the contrary, Markov Switching Models propose using many transition functions conditioned on the discrete state variable. The extension IIA-HMM introduces an identifiable HMM from [3] thanks to source identification, but does not provide additional transition functions conditioned on the discrete variables.

Reviewer comment (Theory):
**The identifiability results are weaker than acknowledged, or at least not sufficiently discussed** -- By directly relying on the result of Kivva to prove the identifiability of the nonlinear emission function / latent components, the results unfortunately inherit its weaknesses. 1) The identifiability requires one to know the family of the noise distribution which is a cumbersome assumption (but see Q2 below). 2) The results rely on piecewise linear emission function and are therefore less general than many other works that allow e.g. almost any injective \( f(x) \) -- consider for instance what happens in your case if \( f(x) \) is a Gaussian Process. 3) As far as I understand, while there is no assumption of independence here (c.f. nonlinear ICA), the results are also clearly weaker i.e. for latent vector \( \mathbf{z} \) your results give essentially \( f(\mathbf{z}) = A f'(\mathbf{z'}) + b \) -- contrast this to nonlinear ICA where one would identify the individual coordinates as per \( z_i = h(z_i') \) for some invertible \( h \) -- please correct if I have misunderstood.

Author response:
> The identifiability results are weaker than acknowledged. or at least not sufficiently discussed

Although from Kivva et al. (2022) results we might have weaknesses in terms of the identification results (transitions and emission up to affine transformations), we argue that they have other advantages that can be interesting depending on the application.
- The results from Kivva allow identifiability up to affine transformations of the prior distribution (which is identifiable), and hence we still achieve identifiability of the discrete state distribution up to permutations.
- The results from Kivva also allow weaker assumptions than the injective mapping assumed in Halva et al. (2021), which uses a weakly-injective (and continuous) mapping (e.g. ReLU networks).
- Following the previous point, note that weaker assumptions generally imply weaker identifiability statements. Therefore, our approach does not achieve decoder component-wise identifiability as we are not assuming independence as in ICA.

We follow this discussion point in Q2.

Reviewer comment (Evaluation):
**Experimental evaluation lacking in places** -- The evaluation of the models on the salsa data seems very qualitative and much more rigorous evaluation would be preferred with some quantitative evaluation metrics. (Q3 below)

Author response:
> Experimental evaluation lacking in places

We agree that the evaluation of salsa data is focused on qualitative visualizations, although we provide reconstruction error metrics for reference. We continue this point in Q3.

Reviewer comment (Theory):
"p(z) is identifiable up to affine transformations" -- This is imprecise language, explain whether you mean parameter identification or identification of z or what.

Author response:
> misc:

We thank the reviewer for carefully revising our writing and we will make sure these changes are implemented accordingly. The only thing to note is that we already introduce the new indexing notation in the last sentence of page 3 as follows: 

“Combined with the path indexing function, this establishes an injective mapping $\phi \circ \varphi^{-1}$ to uniquely map a set of states $s_{1:T}$ to the $a_{1:T}$ indices, and we can view $p_{a_1^i}(z_1)$ and $p_{a_t^i}(z_t |z_{t-1})$ as equivalent notations of $p(z_1 | s_1)$ and $p(z_t |z_{t-1}, s_t)$ respectively for $s_{1:T} = \varphi(i)$. This notation shows that the MSM extends finite mixture models to temporal settings as a finite mixture of $K^T$ trajectories composed by (conditional) distributions in $\Pi_{\mathcal{A}}$ and $\mathcal{P}_{\mathcal{A}}$”

We will add a sentence clarifying that the subscript $\theta$ is dropped for simplicity.

Reviewer comment (Writing):
Equation (4) and elsewhere you use \( p_a \) but earlier \( p_{\theta} \), please explain more clearly what the subscript \( a \) means.

Author response:
> misc:

We thank the reviewer for carefully revising our writing and we will make sure these changes are implemented accordingly. The only thing to note is that we already introduce the new indexing notation in the last sentence of page 3 as follows: 

“Combined with the path indexing function, this establishes an injective mapping $\phi \circ \varphi^{-1}$ to uniquely map a set of states $s_{1:T}$ to the $a_{1:T}$ indices, and we can view $p_{a_1^i}(z_1)$ and $p_{a_t^i}(z_t |z_{t-1})$ as equivalent notations of $p(z_1 | s_1)$ and $p(z_t |z_{t-1}, s_t)$ respectively for $s_{1:T} = \varphi(i)$. This notation shows that the MSM extends finite mixture models to temporal settings as a finite mixture of $K^T$ trajectories composed by (conditional) distributions in $\Pi_{\mathcal{A}}$ and $\mathcal{P}_{\mathcal{A}}$”

We will add a sentence clarifying that the subscript $\theta$ is dropped for simplicity.

Reviewer comment (Novelty):
"Hälvä et al. (2021) introduces time-dependence between sources via linear SDSs" , this appears to be incorrect as mentioned above (this mistake is in two places in the paper).

Author response:
> There are lacking citations and incorrect interpretations of previous works.

We thank the reviewer for carefully comparing our work to existing literature. We will make sure previous work is properly acknowledged (including Morioka et al. (2021)) and modify the reference to other studies using general SDSs (which can be covered by Halva et al. 2021) by highlighting the assumptions on temporal dependence, rather than the assumptions on the transition model. 

Nonetheless, we believe our contribution is significant even though other works have studied similar models.
- The SDS assumptions of Halva et al. (2021) are a combination of the ones used for HMM-ICA (Halva et al. 2020), and Hyvärinen et al. (2017) [2], thereby inheriting their limitations.
  - Regarding HMM-ICA (Halva et al. 2020), the discrete states $s_t$ are assumed to follow a first-order stationary Markov chain with a full-rank transition matrix. The number of states are also required to be known. Instead, **our work makes no assumptions on knowing the number of states, or the structure of the dynamic model of $s_t$,** meaning it can be e.g., non-stationary and/or higher-order Markov.
  - Regarding Hyvärinen et al. 2017 [2], while it is true that they do not impose linearity the assumptions are placed on the temporal dependencies between sources. Our strategy works in a different spirit, where we directly establish restrictions on the transition mapping itself, which is more intuitive.
- The identifiability results of Halva et al. (2021), although stronger in terms of achieving disentanglement of the sources, are weaker in terms of identifying the transitions in the latent space. 
  - Their main results (Thms 1 & 2) prove the identifiability of the non-linear mixing and source distribution, but make **no identifiability statements about the transition dynamics**. 
  - The Linear SDS assumed in Halva et al. (2021) is overcomplete ( observation dimensions < latent dimensions), which is problematic in terms of achieving transition identifiability.
- We believe the following claim “Identifiability of SDSs is thus achieved in Halva et al. (2021)” is not accurate. 
  - Identifiability results for HMMs include identifying the discrete transitions and source emission [3] (which would be transitions for SDSs). Halva et al. (2021) provides no result in this regard for SDS. 
  - Existing HMM identifiability results do not transfer to MSMs due to the explained limitation of Allman et al. (2009) technique. 
- Morioka et al. (2021) provides transition identifiability for vector autoregressive models (VAR), where only one transition function is used. On the contrary, Markov Switching Models propose using many transition functions conditioned on the discrete state variable. The extension IIA-HMM introduces an identifiable HMM from [3] thanks to source identification, but does not provide additional transition functions conditioned on the discrete variables.