Paper: Deep Independent Vector Analysis (ICLR-2024)

Reviewer comment (Theory):
I am missing a more theoretically grounded justification for why the approach of combining MISA and iVAE would yield an identifiable solution in the cross-modal setting (especially with the alternating two-step training procedure, see Question 2). So far the approach seems like stitching existing methods together without further introspection. Pointing out to at least a sketch of a proof, e.g., in the appendix would be desirable.

Author response:
Thank you for the suggestion on the sketch of proof. We have restated key assumptions and Theorems in the identifiability theory (Khemakhem et al. 2020), and provided a conceptual sketch of proof showing that the generative model assumed in DeepIVA is identifiable up to a permutation and component-wise transformation in Appendix A. We have also updated Section 2.1 iVAE and DeepIVA parts accordingly in the revised manuscript.

Reviewer comment (Evaluation):
While interesting, it is hard to interpret the soundness of the MRI results. UK Biobank contains a wide range of brain MRI-derived variables, e.g., volumetrical informations of different brain regions of interest (ROIs). Did the authors consider incorporating these into the analysis - perhaps as a proxy for ground-truth sources, since the underlying ROI volumes stay the same regardless of MRI type?

Author response:
We are not clear about what the reviewer is suggesting we could do with the ROIs specifically. We aim to identify latent sources in a data-driven manner, and it is not necessary that linked sources are located at similar brain regions. Thus, a predefined ROI atlas might not be ideal as a proxy for ground-truth sources. Also, ROIs are not always the same for all types of MRI (for example, white matter tracts vs gray matter ROIs). While not seemingly a good depiction of the ground-truth, ROI-based data may be a good alternative to dimensionality reduction. We will consider that in future experiments.

Reviewer comment (Theory):
Maybe I am missing something, but why does DeepIVA have to be trained with a two-step procedure? Since iVAE is identifiable, can’t we first train each separate modality-specific VAE and then align the sources using MISA?

Author response:
Thank you for the thoughtful question. We train DeepIVA in a two-step procedure because of computational efficiency and practical consideration. If we train iVAE and then use MISA to align unimodal sources, it will require solving a combinatorial problem, which can be computationally expensive as the number of sources and datasets scale up. Additionally, if we first train iVAE independently, it is possible that parameters of one iVAE model might be stuck in local minima and the learned sources might not be linked. For example, in Figure 2 first row, we trained two iVAE models independently, and each model can identify sources fairly well, but cross-modal alignment is poor between two sets of independently identified sources. Iteratively training iVAE and MISA can guide the model to a solution not only maximizing for identifiability but also simultaneously maximizing for linkage.

Reviewer comment (Theory):
The generative model assumed in the DeepIVA framework is never explicitly written in the paper. This critical flaw not only diminishes the clarity of the presentation, but also greatly hinders the ability to assess the rigour and soundness of several claims in the paper, particularly regarding identifiability.

Author response:
1. Thank you for the important suggestion. We have explicitly written the generative model assumed in DeepIVA, and provided a sketch of the conceptual proof showing that the generative model is identifiable up to a permutation and component-wise transformation in Appendix A. We have also updated Section 2.1 iVAE and DeepIVA parts accordingly in the revised manuscript.

Reviewer comment (Theory):
The paper lacks a proper identifiability result. Modeling assumptions and assumptions for identifiability are never properly stated. The reviewer’s understanding is that the paper assumes that identifiability for DeepIVA follows trivially from previous works (e.g., iVAE), which is unclear or misleading. While solid empirical work without identifiability proofs can be valuable, identifiability is referred to multiple times in this paper in a way that may be deceptive compared to other works in the nonlinear ICA literature.

Author response:
1. Thank you for the important suggestion. We have explicitly written the generative model assumed in DeepIVA, and provided a sketch of the conceptual proof showing that the generative model is identifiable up to a permutation and component-wise transformation in Appendix A. We have also updated Section 2.1 iVAE and DeepIVA parts accordingly in the revised manuscript.

Reviewer comment (Theory):
The first question is to explicitly write down the full generative model. If the work is intended to claim identifiability for this model, a theorem proving under which assumptions the generative model is identifiable should be provided.

Author response:
1. Thank you for the important suggestion. We have explicitly written the generative model assumed in DeepIVA, and provided a sketch of the conceptual proof showing that the generative model is identifiable up to a permutation and component-wise transformation in Appendix A. We have also updated Section 2.1 iVAE and DeepIVA parts accordingly in the revised manuscript.

Reviewer comment (Theory):
In the manuscript, it sounds like identifiability should follow trivially from previous works, such as the iVAE framework (Khemakhem et al., 2020). Even if this were the case, it would be helpful to restate and discuss the required assumptions. The reviewer finds it hard to understand how to apply the results in (Khemakhem et al., 2020) in the context of this work. In particular, the results in that work lead to a specific kind of identification of the latent components, which is not up to permutation and element-wise nonlinear transformations: an additional linear indeterminacy remains, and the results refer to identifiability of sufficient statistics. It is unclear why the MISA network should be the right model to undo those unresolved transformations in reconstructed components and to match components across datasets, and on what basis identifiability of the resulting two-step procedure would be guaranteed.

Author response:
**Distinction between shared and linked**:

We consider sources following a multivariate distribution with second- or all-order statistical dependence to be “linked”. 

Shared sources, on the other hand, are sources that share an identical underlying variable. For example, let two latent variables $s_1$ and $s_2$ share the same underlying variable $s$, $s_1=s+n_1$, $s_2=s+n_2$. Altogether, $s$, $n_1$ and $n_2$ make up a 3D distribution of 3 independent variables. Projecting from 3D to 2D (a transformation of variables) causes the resulting $s_1$ and $s_2$ to become linearly dependent, since they share the same underlying $s$, and $n_1$ and $n_2$ are independent, meaning $p(s_1,s_2)$ contains only second order statistics. Thus, complicated dependence relationships that transcend second-order statistics cannot be modeled accurately with shared sources. When linear relationships account for all of the dependence in a dataset then shared sources may furnish a good approximation to linked sources.

Intuitively, if two sets of observations $x_1$ and $x_2$ are generated by two different variables $s_1$ and $s_2$, using a shared source with additive noise ($s+n_1$ and $s+n_2$) may not be sufficient to model $s_1$ and $s_2$. Thus, it is more flexible and appropriate to model each modality or dataset with a separate random variable. 

**MISA in the nonlinear case:**

Point-wise nonlinearities do not alter the dependence between variables. The theory of copulas covers that concept extensively (see Sklar’s theorem): any continuous joint PDF can be represented as $p(s_1)p(s_2)c(F(s_1), G(s_2))$, where $c(\cdot, \cdot)$ is the copula density and $F$ and $G$ are the marginal CDFs of $s_1$ and $s_2$, respectively. The copula density $c(\cdot, \cdot)$ captures all of the dependence between $s_1$ and $s_2$, while $p(s_1)$ and $p(s_2)$ can be nonlinearly altered arbitrarily ($F$ and $G$ will compensate the nonlinear transformations automatically). The conclusion is that the point-wise nonlinearities from iVAE do not alter the dependence between linked sources. 

Thus, one could intentionally transform all final iVAE sources into Kotz marginals and then pursue combinatorial optimization in MISA to detect the links. Note, however, that Kotz marginals don’t have a well-defined closed-form, which further motivates the interweaved optimization of the MISA loss (MLE) to gradually induce such marginals on the iVAE latents.

Reviewer comment (Theory):
The relationship of the model proposed in this paper with previous literature on multi-view ICA, both linear [1, 2, 3] and nonlinear [4], is unclear. The reviewer understands that multi-view ICA models the dependence of the components of $\mathbf{s}_i$ differently—e.g., [1], eq. (1), in the linear case; or [4], eqs. (5-7) and Definition 2, in the nonlinear case. In the nonlinear case [4], Def. 2 introduces a technical assumption which rules out trivial cases and constitutes one of the required assumptions for identifiability. The reviewer is confused by the lack of analogous assumption discussion in this work. Additionally, multi-view nonlinear ICA [4] allows for identifiability without an auxiliary variable besides the collection of views, which is something the authors refer to in the Limitations section of the present paper.

Author response:
2. Thank you for pointing out the multi-view ICA literature. Multi-view ICA assumes that the data can be modeled as a linear transformation of a common/shared source with an additive Gaussian noise. This assumption holds if all datasets are generated from a single shared latent variable, such as naturalistic datasets where subjects might share a similar temporal response. However, it doesn’t hold if each dataset is generated from a separate multidimensional latent variable, such as the expression level of structural and functional imaging features across subjects. The key assumption difference between multi-view ICA and DeepIVA is that sources are $\textbf{shared}$ across subjects in multi-view ICA while sources are $\textbf{linked}$ across data modalities in DeepIVA. In the context of multimodal fusion, it is more reasonable to assume that each data modality is generated by modality-specific latent variables, instead of a shared latent variable, especially for data modalities that are inherently heterogeneous (such as imaging and genomics). Thus, we believe that DeepIVA is more capable in multimodal fusion tasks, as it aims to identify a set of linked sources for each modality.

We have added a discussion to distinguish our approach and multi-view ICA in Section 1 Introduction: “Recent studies on multi-view BSS assume that observations from different views originate from a shared source variable and distinct additive noise variables (Richard et al., 2020; 2021; Pandeva & Forre, 2023; Gresele et al., 2020). However, in the context of multimodal fusion, it is more reasonable to assume that each modality is generated by modality-specific latent variables which, in turn, are linked across modalities, rather than a shared set, especially for data modalities that are inherently heterogeneous.”

Reviewer comment (Experiments):
The description of the neuroimaging data is lacking. The reviewer requests a description of the dataset used in the experiments and how they relate to the DeepIVA model. Clarification is needed on what aspect of the neuroimaging data is supposed to be modelled with a statistical dependence across datasets/subjects (i.e., among components of $\mathbf{s}_i$). It should be stated whether the data is task-based neuroimaging or resting state. If task-based, a comparison to [1, 2] or [5] is needed. If resting state, the interpretation of the statistical dependence among components of the $\mathbf{s}_i$ vector should be explained.

Author response:
3. Sorry for the lack of details about the neuroimaging dataset. We use structural MRI data (gray matter segmentation maps from T1-weighted images) and the voxel-wise amplitude of low frequency fluctuations (ALFF) from a resting-state fMRI dataset collected on the same subjects. The statistical dependence between modalities is modeled using subject expression levels. While we assume expression levels change (i.e., are nonstationary) according to age and sex groups, we assume the statistical dependence between data modalities remains the same and, thus, can be interpreted as structural and functional relationships associated with each latent source. 

We have revised Section 2.3 Neuroimaging Data: “We utilize the UK Biobank dataset (Miller et al., 2016) $\mathbf{X}\in \mathbb{R}^{N \times V \times M}$ including two imaging modalities T1-weighted sMRI and resting-state fMRI ($M=2$) from $2907$ subjects ($N=2907$). We preprocess sMRI and fMRI to obtain the gray matter tissue probability segmentation (GM) and amplitude of low frequency fluctuations (ALFF) feature maps, respectively. Each GM or ALFF feature map includes $44318$ voxels ($V=44318$). Here, we use age and sex groups as auxiliary information, assuming that sources within each modality are conditionally independent given the age and sex group. This assumption is based on studies demonstrating the significant impact of age and sex on both brain structure and function (Raz et al., 2004; Good et al., 2001; Ruigrok et al., 2014).”

Reviewer comment (Evaluation):
The reviewer asks how to explain the results in Fig. 5 and why the MCC is so low.

Author response:
4. Regarding the MCC results, please first note that the per-modality per-segment MCC metric evaluates the MCC in the same way as in the nonlinear ICA literature, and these per-modality per-segment MCC values for 5 sources (>0.9) are very close to those reported in Khemakhem et al. 2020. Furthermore, in order to account for cross-modal linkage and cross-segment consistency, we propose a new approach to measure the MCC values (see Section 2.4 for details), and the proposed per-modality MCC, per-segment MCC and aggregated MCC metrics may be lower because the iVAE fails to align linked sources across modalities or consistent sources across segments. 

MCC values are lower when we estimate more latent sources (15 sources) with the same sample size (2800 samples). Our hypothesis is that the sample size was not sufficient to accurately recover more latent sources. We repeated the experiments with a double sample size (5600 samples) and an intermediate problem scale (10 sources). According to the new result, we observe that 1) model performance decreases as the number of sources increases when the sample size stays the same; 2) DeepIVA performance improves as the sample size increases when the number of sources stays the same. 

Please see Figure 4 and Section 3.1 in the revised manuscript: “We perform a systematic evaluation of model performance across different data-generating configurations by varying both the problem scale ($5$, $10$ and $15$ sources) and the sample size ($2800$ and $5600$ samples). The aggregated MD and MCC metrics are shown in Figure 4. Remarkably, DeepIVA outperforms iVAE and MISA in every configuration, showcasing its superior performance across all evaluated scenarios. Within each panel, we observe a consistent drop in model performance as the number of latent sources increases, suggesting that the optimization problem becomes more challenging as the latent dimension increases. Across horizontal panels, the DeepIVA performance improves for configurations with $10$ and $15$ sources when the sample size increases from $2800$ to $5600$, indicating that a larger sample size is necessary to better recover sources in a harder problem.”

Reviewer comment (Novelty):
The contribution is limited. It is essentially a combination of two existing methods (MISA and iVAE) without any theoretical results. Since the key contributions of MISA and iVAE are their identifiability proofs instead of the estimation (e.g., the estimation model of iVAE is basically the vanilla VAE), combining their estimation methods without any theoretical contribution might be incremental.

Author response:
We thank the reviewer for requesting clarification on the motivation and contributions of our proposed method.

Regarding motivation, we have revised the manuscript to explain why we are interested in multimodal fusion in Section 1 Introduction: “Jointly analyzing two imaging modalities can uncover cross-modal relationships that cannot be detected by a single imaging modality, providing new insights into structural and functional interactions in the brain and its disorders (Calhoun & Sui, 2016).” 

Also, another reason we choose MISA is that we want to utilize its flexible subspace modeling to further solve nonlinear independent subspace analysis (ISA) problems. We have added one more reason why we choose MISA to align latent sources in Section 4 Discussion: “We plan to extend our proposed method from nonlinear IVA problems to nonlinear ISA problems, aiming to capture source dependence by leveraging higher-dimensional subspaces.”

Regarding contributions, our study is an initial step towards learning linked and identifiable sources from multimodal data. We empirically demonstrate that DeepIVA can learn linked and identifiable sources by unifying iVAE and MISA, and “the idea of combining iVAE and MISA is new”, as acknowledged by reviewer queK. We would like to point out that our proposed method is not “simply a combination of existing methods”. We modified the iVAE encoder architecture to process data features and auxiliary information separately, as noticed by reviewer dLxh. Additionally, the linear demixing matrix in the original MISA framework was replaced by a nonlinear encoder, and we adjusted the loss term accordingly as described in Section 2.1 Deep Independent Vector Analysis. 

We have updated Figure 1 to illustrate the extra details of our method, including the data input and the encoder layer. We have also updated Section 2.1 Deep Independent Vector Analysis to explain the architecture change: “Additionally, since MISA is not designed to handle auxiliary information, we modify the original encoder architecture to distinguish between data features $\mathbf{x}^{m}$ and auxiliary variables $\mathbf{u}$ such that 1) the iVAE updates model parameters with respect to both $\mathbf{x}^{m}$ and $\mathbf{u}$ at the input layer, and 2) the MISA updates only those pertaining to $\mathbf{x}^{m}$ but not $\mathbf{u}$. The original iVAE model uses a single input layer taking the concatenated $\mathbf{x}^{m}$ and $\mathbf{u}$. In DeepIVA, we split this layer into two: one for data features $\mathbf{x}^{m}$ and another for auxiliary variables $\mathbf{u}$. The parameters with respect to $\mathbf{u}$ will only be updated at the iVAE training step but will remain frozen at the MISA training step. Also, the inputs for the auxiliary variables are set to $0$ during MISA training to ensure no influence from the frozen weights.”

Furthermore, we have restated the identifiability theory in Khemakhem et al., 2020, and provided a conceptual sketch of proof showing that the generative model assumed in DeepIVA is identifiable up to a permutation and component-wise transformation in Appendix A. We have also updated Section 2.1 iVAE and DeepIVA parts accordingly in the revised manuscript.

Reviewer comment (Novelty):
Almost half of the manuscript is dedicated to the introduction of existing works, and the other half focuses on experiments. Meanwhile, only **half a page** describes the proposed method. The brevity of the description might be understandable if the contribution is simply a combination of existing methods. However, if that's not only a combination, a more detailed explanation regarding the motivation and unique contributions of the proposed method is necessary.

Author response:
We thank the reviewer for requesting clarification on the motivation and contributions of our proposed method.

Regarding motivation, we have revised the manuscript to explain why we are interested in multimodal fusion in Section 1 Introduction: “Jointly analyzing two imaging modalities can uncover cross-modal relationships that cannot be detected by a single imaging modality, providing new insights into structural and functional interactions in the brain and its disorders (Calhoun & Sui, 2016).” 

Also, another reason we choose MISA is that we want to utilize its flexible subspace modeling to further solve nonlinear independent subspace analysis (ISA) problems. We have added one more reason why we choose MISA to align latent sources in Section 4 Discussion: “We plan to extend our proposed method from nonlinear IVA problems to nonlinear ISA problems, aiming to capture source dependence by leveraging higher-dimensional subspaces.”

Regarding contributions, our study is an initial step towards learning linked and identifiable sources from multimodal data. We empirically demonstrate that DeepIVA can learn linked and identifiable sources by unifying iVAE and MISA, and “the idea of combining iVAE and MISA is new”, as acknowledged by reviewer queK. We would like to point out that our proposed method is not “simply a combination of existing methods”. We modified the iVAE encoder architecture to process data features and auxiliary information separately, as noticed by reviewer dLxh. Additionally, the linear demixing matrix in the original MISA framework was replaced by a nonlinear encoder, and we adjusted the loss term accordingly as described in Section 2.1 Deep Independent Vector Analysis. 

We have updated Figure 1 to illustrate the extra details of our method, including the data input and the encoder layer. We have also updated Section 2.1 Deep Independent Vector Analysis to explain the architecture change: “Additionally, since MISA is not designed to handle auxiliary information, we modify the original encoder architecture to distinguish between data features $\mathbf{x}^{m}$ and auxiliary variables $\mathbf{u}$ such that 1) the iVAE updates model parameters with respect to both $\mathbf{x}^{m}$ and $\mathbf{u}$ at the input layer, and 2) the MISA updates only those pertaining to $\mathbf{x}^{m}$ but not $\mathbf{u}$. The original iVAE model uses a single input layer taking the concatenated $\mathbf{x}^{m}$ and $\mathbf{u}$. In DeepIVA, we split this layer into two: one for data features $\mathbf{x}^{m}$ and another for auxiliary variables $\mathbf{u}$. The parameters with respect to $\mathbf{u}$ will only be updated at the iVAE training step but will remain frozen at the MISA training step. Also, the inputs for the auxiliary variables are set to $0$ during MISA training to ensure no influence from the frozen weights.”

Furthermore, we have restated the identifiability theory in Khemakhem et al., 2020, and provided a conceptual sketch of proof showing that the generative model assumed in DeepIVA is identifiable up to a permutation and component-wise transformation in Appendix A. We have also updated Section 2.1 iVAE and DeepIVA parts accordingly in the revised manuscript.