Paper: Non-negative Probabilistic Factorization (ICLR-2024)

Reviewer comment (Presentation):
Equations (2), (3), (4) and Figure 1 are fairly standard for people familiar with NMF. However, the authors dedicate considerable space to these concepts and even replicate them later, such as in Equations (8) and (9).

Author response:
**[Novelty]** &emsp; We politely disagree with the reviewer and believe that while equation (2) describes the standard formulation of NMF, Eqs. 3 and 4 provide a new model that includes a per-sample latent component matrix H[i] sampled from learned prior distributions - unlike previous probabilistic\bayesian extensions of NMF, where the prior distributions are assumed to be pre-learned and the component matrix is shared between all samples. We will add to the paper a more detailed discussion on this type of related work to highlight the inherent differences. 

Moreover, Eqs. 8 and 9 reformalize Eqs. 3 and 4 using auxiliary variables to provide a more efficient way of computing the model’s likelihood.

Reviewer comment (Novelty):
It appears that the proposed VarNMF is a simplified version of the model/framework designed in Tan, V. Y., & Févotte, C. (2012). Automatic relevance determination in nonnegative matrix factorization with the/spl beta/-divergence. IEEE transactions on pattern analysis and machine intelligence, 35(7), 1592-1605. The authors of the current submission should not be unaware of this closely related work.

Author response:
We would like to stress again the fundamental difference between our work and the different probabilistic/Bayesian versions of NMF, including the one suggested in [TF12].

We emphasize that the novelty is *not* in the algorithm (which is a fairly standard graduate-level exercise EM in our case) but in the probabilistic model itself. Thus while our learning and inference algorithms are different from those of previous works, it is due to the differences in the mathematical models. Therefore we will not get into the optimization algorithms employed by the different papers.

To highlight the differences in the *mathematical models*, we attach below a description of the model in our work versus the standard NMF [1], Bayesian NMF [2], and [TF12] suggested by the reviewer. We hope this will clarify any misunderstanding. Additionally, as mentioned, we will add to the paper a more detailed discussion on this type of related work to highlight the inherent differences.

First, we note that [1] and [TF12] refer to W as the basis\component KxM matrix and to H as the activation\weights NxK matrix (where N=#samples, M=#features and K=#components). As the component and weight matrices are symmetrical in [1, TF12] but not in our work, we will switch the names of W,H when describing [1, TF12]. In [2], the notations are completely different and we will match them to our notation for an easier read.

**The plate notations of the different models are provided in https://ibb.co/J3zWh85.**

These translate into the following generative models, where

1. For each random variable $X$, $p_X(\theta^X)$ is the distribution of $X$ in the relevant model with some parameters $\theta^X$.
2. We index samples\ data-points with $i=1,…,N$, features with $j=1,…,M$, and components with $k=1,…,K$. 
3. A random variable $X$ is indexed by sample $i$ using square parentheses $X[i]$, by $j$ and $k$ using subscript $X_{k}$\ $X_{j}$\ $X_{k,j}$, and by both as $X[i]\_{k}$\ $X[i]\_{j}$\ $X[i]\_{k,j}$ (as mentioned, a more explicit notation description will be added to the paper).

**NMF:** $W,H$ are parameters

$\forall\_{i,j}, V[i]\_{j}\sim p_V(\sum_k W[i]\_{k} \cdot H\_{k,j})$

**Bayesian-NMF:** $\theta^W, \theta^H$ are hyperparameters

$\forall\_{i,k}, W[i]\_{k} \sim p\_W(\theta^W\_{i,k})$

$\forall\_{k,j}, H\_{k,j} \sim p_H(\theta^H\_{k,j})$

 $\forall\_{i,j}, V[i]\_{j}\sim p_V(\sum_k W[i]\_{k} \cdot H\_{k,j})$

**[TN12]:** $a,b$ are hyperparameters

$\forall\_{k}, \lambda\_{k} \sim p\_\lambda(a,b)$

$\forall\_{i,k}, W[i]_{k} \sim p\_W(\lambda\_{k})$

$\forall\_{k,j}, H\_{k,j} \sim p\_H(\lambda\_{k})$

 $\forall\_{i,j}, V[i]\_{j}\sim p\_V(\sum_k W[i]\_{k} \cdot H\_{k,j})$

**VarNMF:** $W,\theta_H$ are parameters

$\forall\_{i,k,j}, H[i]\_{k,j} \sim p_H(\theta^H\_{k,j})$

 $\forall\_{i,j}, V[i]\_{j}\sim p_V(\sum_k W[i]\_{k} \cdot H[i]\_{k,j})$

Note the difference in the indexing of $H$ in VarNMF compared to all other models - it is now indexed by ******sample****** $i$ and not only by component $k$ and feature $j$. In the plate models this is reflected by the fact that in VarNMF, the red (sample) plate includes $H$. This is not the case in all other models. 

This difference is the main point of our work - we have a latent component matrix $H[i]$ **for each sample $i$**, in contrast to a single (latent or not) component matrix $H$ in all other models. This change is also reflected in the sampling of $V$, which is identical for all previous models (all samples share the matrix $H$), but different in ours (each sample $i$ is sampled using a different matrix $H[i]$).

The relevance of defining a component matrix $H[i]$ for each sample $i$ is that now we can ask about the distribution of these matrices in the population - i.e., about the distribution $p_H$ that is defined over samples (specifically - estimate its parameters $\theta^H$ from data). This is not done in the other models mentioned above - either because there is no distribution on $H$ (NMF), or because the distribution  $p_H$ reflects prior knowledge\ regularization purposes (e.g., non-negativity\ sparsity constraints in Bayesian-NMF, cost on higher dimensionality in [TF12]) and not variance between samples\ data-points.   

Specifically, the prior distributions $p(W[i]\_{k}|\lambda\_k)$,  $p(H\_{k,j}|\lambda\_k)$ in [TF12] makes $W[i]\_k$  and $H\_{k,j}$ to be “[…] tied together by a common variance-like parameter $\lambda\_k$ […]. When a particular $\lambda\_k$ is small, that particular column of W and row of H are not relevant and vice versa.” [TF12] - i.e., the distribution $p\_H$ does not reflect variance in the values of $H$ between samples. This is simply a prior distribution that drives the values of $W,H$ down to regularize the number of components $K$.

With this explanation, we feel like we demonstrated the fundamental differences between our work and previous works, including [TF12]. We hope this clarifies any misunderstanding.

Reviewer comment (Writing):
The mathematical notations in the paper are somewhat messy. For example, the dimension of $V$ is first mentioned on page 3, but $H_k \in \mathbb{R}^M_{\ge 0}$ suddenly appears on page 2. On page 4, the notation $V = (V[1], \ldots, V[N]) \in \mathbb{R}^{N\times M}_{\ge 0}$ is written in a confusing style. Additionally, some simple equations such as (8), (9), and (12) are repeated (although notations are slightly different) and occupy entire lines.

Author response:
**[Notation]** &emsp; We are sorry that the notation is not clear. We are adding a notation section to provide a clear reference for the notation we use. We hope that this will alleviate the confusion.

Reviewer comment (Evaluation):
Another limitation of the method is the heavy computational cost of the algorithm, even if the authors propose approaches to cope with this issue.

Author response:
**[Computational costs]**   Again, we agree with the reviewer's comment on the computational costs and we are exploring different computational approaches (e.g., variational inference) to reduce the computational costs. We decided to focus here on the basic idea, as the paper is already dense.

Reviewer comment (Theory):
The method proposed by the authors is specific to the Poisson distribution for the observational probability (and to the assumptions made on the dependence structure of the sources).

Author response:
**[Observational probability]**   Indeed, the model is specific to Poisson distribution. There are many applications where the Poisson distribution is the natural model of technical variability, including most sequencing-based assays. More complex distributions used in the literature, such as Negative Binomial distributions essentially fold the Poisson noise with additional levels of uncertainty that we claim are often better modeled as source variation (wrt to your first question). With that said, it is of interest to explore the feasibility of other noise models. A move to variational inference, for example, will provide flexibility with respect to the observational distribution.

Reviewer comment (Theory):
How the method can be adapted to consider other observational probability such as a negative binomial distribution (instead of the Poisson)? The negative binomial distribution would be particularly valuable for the biological application suggested by the authors as it enables the modeling of overdispersion, a phenomenon known to occur in such biological settings.

Author response:
**[Observational probability]**   Indeed, the model is specific to Poisson distribution. There are many applications where the Poisson distribution is the natural model of technical variability, including most sequencing-based assays. More complex distributions used in the literature, such as Negative Binomial distributions essentially fold the Poisson noise with additional levels of uncertainty that we claim are often better modeled as source variation (wrt to your first question). With that said, it is of interest to explore the feasibility of other noise models. A move to variational inference, for example, will provide flexibility with respect to the observational distribution.

Reviewer comment (Theory):
In the same spirit, would it be possible to change the model to have dependence between features of source $k$?

Author response:
**[Independence of features]**   We definitely agree with the reviewer’s comment regarding the limitation of the independence assumption (see our last paragraph) and are actively exploring different models for capturing dependencies between genes (ranging from network of dependencies to lower-dimensional latent representations). This also relates to your second question.

Reviewer comment (Novelty):
The related work has not been discussed thoroughly. The paper only shallowly discusses related work on Poisson Factorization and misses out on Poisson Factor Analysis. Probabilistic Matrix Factorization and Bayesian Matrix Factorization are not discussed in detail.

Author response:
**[Related work and comparisons]**   We thank the reviewer for the kind summary. However, we want to point out the fundamental difference between our work and Poisson Factorization, Bayesian MF, and other methods mentioned by the reviewer.

Poisson Factorization, as used for example in recommendation systems (e.g., [1]), introduces k latent features (preferences) of the user and parallel k features (attributes) of the item. The product of those (followed by Poisson noise) determines the observation. 

As a plate model, this can be represented the same as the NMF model (Figure 1C top), where W and H are latent (sampled from priors). Importantly, the items’ attributes H can vary between items but cannot vary among users – they are assumed to be shared between all users. In contrast, in our model, the component matrix H has a different value per-user i (sample i in our setting), H[i]. In terms of the plate model shown in Figure 1C (bottom), the red plate over samples includes the variable H which is indexed by both i (samples) and j (genes). 

To the best of our knowledge, the various Poisson Factorization, Probabilistic MF, Bayesian MF, and Hierarchical Poisson Factorization models all share the same inherent structure, where the component matrix is shared between samples [1,2,3,4]. Therefore, they cannot capture variability in sources signal between samples. Thus, adding additional comparisons will not touch the main point which is the usefulness of capturing variability in sources signal between samples and the feasibility of estimating it from mixed samples.

That said, following the comments of this reviewer and the others, we will revise the related work section to further highlight this difference. Please let us know if you have more questions concerning this issue.

Reviewer comment (Novelty):
Given that the authors consider the 'mean signals' of the variable components in their interpretation and analysis, it is not clear if one could achieve the same by using a Bayesian Matrix Factorization and an appropriate prior distribution. Given the lack of contextualization, it is not clear how novel the proposition is.

Author response:
**[Related work and comparisons]**   We thank the reviewer for the kind summary. However, we want to point out the fundamental difference between our work and Poisson Factorization, Bayesian MF, and other methods mentioned by the reviewer.

Poisson Factorization, as used for example in recommendation systems (e.g., [1]), introduces k latent features (preferences) of the user and parallel k features (attributes) of the item. The product of those (followed by Poisson noise) determines the observation. 

As a plate model, this can be represented the same as the NMF model (Figure 1C top), where W and H are latent (sampled from priors). Importantly, the items’ attributes H can vary between items but cannot vary among users – they are assumed to be shared between all users. In contrast, in our model, the component matrix H has a different value per-user i (sample i in our setting), H[i]. In terms of the plate model shown in Figure 1C (bottom), the red plate over samples includes the variable H which is indexed by both i (samples) and j (genes). 

To the best of our knowledge, the various Poisson Factorization, Probabilistic MF, Bayesian MF, and Hierarchical Poisson Factorization models all share the same inherent structure, where the component matrix is shared between samples [1,2,3,4]. Therefore, they cannot capture variability in sources signal between samples. Thus, adding additional comparisons will not touch the main point which is the usefulness of capturing variability in sources signal between samples and the feasibility of estimating it from mixed samples.

That said, following the comments of this reviewer and the others, we will revise the related work section to further highlight this difference. Please let us know if you have more questions concerning this issue.

Reviewer comment (Experiments):
The paper lacks in experimental contextualization. That is, the paper does not properly compare VarNMF to the state-of-the-art of direct competitors from Poisson Factorization, Poisson Factor Analysis, or Bayesian Matrix Factorization. Comparing with Hierarchical Poisson Factorization (HPA) as a baseline might be interesting. That means, benefits and limitations of VarNMF compared to the state-of-the-art remain unclear.

Author response:
**[Related work and comparisons]**   We thank the reviewer for the kind summary. However, we want to point out the fundamental difference between our work and Poisson Factorization, Bayesian MF, and other methods mentioned by the reviewer.

Poisson Factorization, as used for example in recommendation systems (e.g., [1]), introduces k latent features (preferences) of the user and parallel k features (attributes) of the item. The product of those (followed by Poisson noise) determines the observation. 

As a plate model, this can be represented the same as the NMF model (Figure 1C top), where W and H are latent (sampled from priors). Importantly, the items’ attributes H can vary between items but cannot vary among users – they are assumed to be shared between all users. In contrast, in our model, the component matrix H has a different value per-user i (sample i in our setting), H[i]. In terms of the plate model shown in Figure 1C (bottom), the red plate over samples includes the variable H which is indexed by both i (samples) and j (genes). 

To the best of our knowledge, the various Poisson Factorization, Probabilistic MF, Bayesian MF, and Hierarchical Poisson Factorization models all share the same inherent structure, where the component matrix is shared between samples [1,2,3,4]. Therefore, they cannot capture variability in sources signal between samples. Thus, adding additional comparisons will not touch the main point which is the usefulness of capturing variability in sources signal between samples and the feasibility of estimating it from mixed samples.

That said, following the comments of this reviewer and the others, we will revise the related work section to further highlight this difference. Please let us know if you have more questions concerning this issue.

Reviewer comment (Experiments):
The experimental section lacks in breadth. Since the focus lies on bioinformatics case-studies, it is unclear how VarNMF performs in other domains. Considering the interests of our ICLR community, readers probably expect a broader evaluation. Given the related work, it would be highly beneficial to include experiments on recommender systems or computer vision datasets, such as Movielens, Netflix Prize (which all are ideally suitable for the Poisson Factorization), or Olivetti Faces.

Author response:
**[Different datasets]**   Our work focuses on real-world applications. Specifically, we are mostly interested in the computational biology domain that offers a wide range of possible applications. While the suggested applications are of interest, we decided to analyze problems in this domain only.

Reviewer comment (Experiments):
The empirical evaluation lacks in depth. The robustness of VarNMF under noise has not been evaluated. An empirical evaluation on convergence is not included. The paper does not compare the runtime of VarNMF to competing methods. The performance characteristics of VarNMF on a wide range of datasets is unclear.

Author response:
**[Robustness to noise]**   We agree with the reviewer that robustness is an important property and we performed various evaluations. We decided to focus our presentation in the paper on the one we believed is most crucial – robustness to increased level of source variability. We will add to the appendix results regarding levels of observation noise and number of samples.

Reviewer comment (Presentation):
The title of this paper is quite general and does not match the specific contribution of this paper.

Author response:
**[Title]**   We oscillated between having a detailed title and a shorter one. We can shift back to “Non-negative probabilistic factorization with source variability”.

Reviewer comment (Writing):
What is the purpose of \hat{K}-NMF?

Author response:
**[$\tilde{K}$-NMF]**   As mentioned in Section 4.2, the number of parameters in VarNMF is larger than in NMF. To show that the improvement in performances in VarNMF is not due to this increase, we also test it against NMF with an increased number of sources - $\tilde{K}$, where $\tilde{K}$ is the minimal number of sources to compensate for the difference in number of parameters.

Reviewer comment (Theory):
I am concerned about the high computational complexity of the proposed model (the number of parameters increases with the number of data points).

Author response:
**[High computational complexity]**   We believe that this comment, as well as the first question, is due to a confusion between the model parameters and the latent variables in the model. The individual H[i] of each sample i is a random variable that we integrate over. The parameters of the model $\theta_H$ define the prior distribution over H, from which H[i] are assumed to be sampled i.i.d. These distinctions are highlighted in the plate model of Figure 1C. Specifically, the number of parameters in VarNMF is NK+2MK vs NK+MK in NMF.

Does this answer your concern?

Reviewer comment (Experiments):
The manuscript does not include extensive or rigorous experiments with real data. In section 4.3, it is stated that the experiment was done once with the randomly split dataset.

Author response:
**[Experiments in real data]**   Our evaluation on real data aims to provide understanding of the implication of the learned models beyond performance (that was evaluated on synthetic data). Thus, our focus was on the biological relevance of the learned components. We can of course provide results for multiple train/test splits.

Reviewer comment (Reproducibility):
Regarding the training/testing experiments in sections 4.2 and 4.3. Each data point has its H[i] matrix. How can we define H[*] for a test point *?

Author response:
**[Test component matrix]**   For a training sample i, given the learned weights W[i] and prior distribution over H, we estimate H[i] using the mean of the per-sample posterior distribution from Eq.18 (see illustration in Figure 2C). For a test sample \*, as described in Appendix A.7, the weights W[\*] are estimated using the EM procedure while keeping the prior distributions over H constant (similar to the deconvolution problem). Given the new weights W[\*], estimating H[\*] is done in the same way as for a training sample. Please let us know if this is still not clear.

Reviewer comment (Experiments):
Have you compared the proposed NMF method with other Bayesian NMF models on the dataset used in the experiments?

Author response:
**[Comparison to Bayesian NMF]**   While we thank the reviewer for the suggestion of additional baselines, we note that similar to NMF, using Bayesian NMF models will result in an estimation of a component matrix H that is shared between all samples. Although the extracted H has the potential to result in a more accurate estimation for the mean signal of the sources, it cannot capture the variability in sources signal between samples. Thus, adding additional comparisons will not touch the main point which is the usefulness of capturing variability in sources signal between samples and the feasibility of estimating it from mixed samples.