From: ICLR.cc/2021/Conference/Program_Chairs
title: Final Decision
decision: Accept (Poster)
comment: The authors present a hierarchical factorization of the Poisson matrix and explain why sparcity in the encoder is important for interpretability. The reviewers appreciated the contribution of the paper and highlighted the advantage of such an approach for users. The authors have improved their initial version by adding more detail on inferences and experiments.  The decision is to accept the paper.

From: ICLR.cc/2021/Conference/Paper3647/Authors
title: Summary of major changes
comment: We greatly enjoyed this open peer review experience and would like to thank the reviewers for helping us to improve our manuscript.  We have touched on each of the individual changes in various comments, however, here is a summary of the major changes that we have made to the manuscript:

- More information on inference
- Direct comparison to standard HPF, on the synthetic datasets (Fig 3)
- A new comorbidity factorization figure (Fig 4)
- More-detailed exposition of interpretation within the comorbidity example
- Less-rushed exposition of the main model equation (Eq 2)
- Emphasis on how we are using the encoder as a proxy for subsequent Bayesian inferences (Eq 3)
- Better motivation behind the choices of priors, noting rationale behind hyperparameter presets used in the manuscript.
- Training runs for generating our synthetic results are now given in a notebook in the Supplemental Materials

We hope that these changes will be satisfactory. Thanks!

From: ICLR.cc/2021/Conference/Paper3647/AnonReviewer5
title: Strong paper in development
comment: Dear authors, 

I am responding for all of your separate comments here, just to keep things in one place. Overall, I am delighted for the very detailed responses you are providing, and all of your responses are reasonable.

Regarding inference, I fully subscribe to the probabilistic programming ideal of focusing on model specification and not on inference details (unless writing a paper on inference as such), but my main concern was about whether your empirical experiments are sufficiently transparent for the reader and can be relied on. For this is it is important to provide sufficient details both on what you did and how well it worked. The new Inference section motivates the choices much better and is satisfactory, and the WAIC-based comparison of the link functions shows that the claims on evaluation methods that could be used were real.

Regarding the model and its intuition, I still think that the idea is nice and clearly worth publishing. My main concern here was that of a bit lost opportunity to describe the connections in a way that maximally many readers understand what you are aiming at. The revised version is again better, but the presentation remains quite compact (naturally also due to space constraints) and I am not sure how much it helps readers who are already not thinking in these terms. 

Overall, I think that the revised paper has improved and I am still leaning towards acceptance, but the paper still feels a bit rushed in presentation. I am increasing my score by one point to account for the improved discussion of the missing details.

From: ICLR.cc/2021/Conference/Paper3647/Authors
title: Re: Use the review process as you wish 
comment: Thanks for the encouragement.

We have posted a revised version of our manuscript  to address the critiques mentioned. In particular, we added more exposition around the interpretation of  the real-data example. We also added a direct comparison to HPF. Finally, our new version incorporates additional background behind inference.

From: ICLR.cc/2021/Conference/Paper3647/AnonReviewer2
title: Use the review process as you wish
comment: I understand that you are discovering the review discussion. Use it as you wish. Do not feeling pressurized to do it one way or the other. I for one, I'm simply genuinely interested in understanding better your view, the points that you are trying to put forward, and the evidence that you are bringing. Better exposition makes strong manuscripts.

From: ICLR.cc/2021/Conference/Paper3647/Authors
title: A metaphor
comment: Additionally, I thought I would comment on how our misinterpretation of the review process is a metaphor for how matrix factorization methods are misinterpreted. The review process resembles that of an ordinary journal in many ways. For this reason, we had the apriori bias that it would proceed along the same dynamics -- a deadline for us to dump a comprehensive set of revisions and a detailed rebuttal. Instead, if we put aside our biases and read the instructions in more detail, we would have seen that there is to be interaction between us and  the reviewers.

So, we made fundamentally the same mistake that people make in misinterpreting existing factorization methods. All matrix factorization methods bear strong resemblance to PCA. In PCA, the inferred loading matrix is orthogonal. For this reason, the transpose of this matrix provides an inverse transformation - one never needs to think about whether a given transformation is data -> representation or representation -> prediction. Given one, the other is implied. However, the central message on our manuscript is that this is not the case for factorization methods in general.

From: ICLR.cc/2021/Conference/Paper3647/Authors
title: Re: Emphasis on interpretability, but the improvement on this aspect is not well demonstated 
comment: We thank the reviewer for their concern. While we work on improving the exposition behind the two main examples we give in the text (Synthetic data  + comorbidity), so that our  point on interpretability is clear, we wanted to comment on the hyperparameters.

While our method has the appearance of having many hyperparameters, we have set it up so that this is not the case. For example, some the parameters $\eta_i$ and $\xi_u$ are scaling parameters that are computed directly from the data. The reason we do this is so that we can set  our priors in the model so that they generalize without much in the way of hyperparameter optimization, so that the priors are weakly informative in that they regularize the problem but do not influence the solution much when ample data exists.

The only real hyperparameters in our model control the scaling of the horseshoe distributions, though in our presentation of the model we have preset the values for  these parameters (see Eq 3), which will be Eq 4 in our revision. In the horseshoe prior, the scaling variables control the expectation for the amount of apriori sparsity in the solution. In Piironen and Vehtari (2017), they show that the expected number of nonzero components scales to the square root  of the size of the parameter vector. **We note that  this is just what is apriori expected, and there is actually a wide range that is admissible.** Based on their analysis, we pre-set the scale of the Horseshoe on the variables u to $1/\sqrt{UI}$, so that we have invariance to problem size. All of this is theoretical so far.

What we didn't show in the main text was that we tuned the method to different combinations of $U$ and $I$ to exhibit the behavior seen in analysis of synthetic data in Fig 2. The behaviors we wanted were

1. Random unstructured noise variables are excluded
2. Covarying variables are included

Our presets for all of the potential hyperparameters exhibited the desired behavior robustly across many choices of $U$ and $I$, from small to large.  In our revision we'll comment a bit more about this. In essence, we have set it up so that there are zero hyperparameters in our model, though one might want to make some modification for less or more sparsity as desired, by multiplying the $1/\sqrt{UI}$ term by a constant.

From: ICLR.cc/2021/Conference/Paper3647/Authors
title: Our new description of inference
comment: We haven't yet posted the revision, while we are making other edits, but here is the inference scheme:

The model of Eq. 2 is a generalized linear factor model that we have mathematically related to probabilistic autoencoder. When augmenting HPF with explicit encoder inference, as we have done, one obtains a probabilistic autoencoder.  This fact suggests that other work in the literature can serve as a guide for training, especially work done on using the horseshoe prior in Bayesian neural networks (Ghosh & Doshi-Velez, 2017a; Ghosh et al., 2018; Louizos et al., 2017).

In particular, Ghosh et al. (2018) investigated structured variational approximations of inference of Bayesian neural networks that use the horseshoe prior and found them to have similar predictive power as mean-field variational approximations.  The disadvantage of structured approximations is the extra computational cost of inferring covariance matrices.  For these reasons, we focus on mean-field black-box variational inference, using Ghosh et al. (2018) as a guide, noting consistency of their scheme with other works that have investigated variational inference on problems using the horseshoe prior (Wand et al., 2011; Louizos et al., 2017).

As  in  Ghosh  &  Doshi-Velez  (2017a);  Ghosh  et  al.  (2018);  Chang  et  al.  (2019),  for  numerical stability,  we  reparameterize  the  Cauchy  distributions  in  terms  of  the  auxiliary  inverse  Gamma representation (Makalic & Schmidt, 2016),

(Equation)

We perform approximate Bayesian inference using fully-factorized mean-field Automatic Differenti-ation Variational Inference (ADVI) (Kucukelbir et al., 2017). For all matrix elements, we utilizedsoftplus-transformed Gaussians, and coupled these to inverse-Gamma distributions to for scale param-eters, as investigated in Wand et al. (2011).

From: ICLR.cc/2021/Conference/Paper3647/AnonReviewer2
title: Emphasis on interpretability, but the improvement on this aspect is not well demonstated
comment: I thank the author for their reply.

I understand that the main benefit may be on interpretability, but this benefit is not demonstrated in a convincing way. Theoretical arguments, such as the need for sparsity, are not enough, as the end judge of interpretability is a human. Reading the authors' reply, I get the impression that the contributed model is more stable, or easier to tune, with regards to "details" such as hyper parameters. If this is the case, it should be demonstrated. The authors claim that shortcomings in interpretability of the HPF model is what motivated them to contribute a new method. They should show the reader what they saw, to back this motivation.

From: ICLR.cc/2021/Conference/Paper3647/Authors
title: R1 - inference
comment: - **My biggest issue with the paper concerns inference. The description is limited, only referring to a specific algorithm (ADVI) without specifying all details (mean-field vs full-rank approximation), and there is no discussion or analysis on how well it works.**

As a Bayesian hierarchical model, there are many methods to perform inference. Because we do not claim to be using the best method, and were short on space, we did not devote much space to our exact method. Additionally, we did not want to detract from the main message of sparse encoding.

In our revision in preparation, we are providing more transparency as to the details of the scheme. In summary, we are relying on published literature for providing backing behind our inference scheme. In particular, we are adapting the mean-field variational black box scheme used in literature on horseshoe Bayesian autoencoders (Ghosh 2019, and newly cited Louizos 2020). In our manuscript, we provided Eq. 8 (will be Eq 9 in revision) as a specific reparameterization for improving computational stability, as done in the literature.

-  **The authors do say that the algorithm converged fast, but do not show this in any experiment. More importantly, the convergence does not yet guarantee the approximation is good and there are well-known cases for which ADVI does not really work that well.  Expanding both the discussion and empirical demonstration of this would be critical. Now you say "one may use..." and "one can access..." with references to specific techniques for evaluating the quality, which gives the impression you have not actually done that.**

The cited manuscripts Ghosh 2019 and Louizos 2019 discuss inference and we would like to maintain a focus on interpretability. However, in the Supplemental Materials, we will now provide notebooks showing inference performed for the synthetic data examples. Actually, our implementation is already public on github, along with links to Colab notebooks that reproduce Fig. 2. We will not link to our repository yet to preserve anonymity. Post-deanonymization, we will directly link to empirical results. For now, please refer to our revised Supplement that we will post sometime this weekend.

- **As I presume you implemented the model in Stan (no point in using ADVI if not; there are better stochastic VI methods around that are also easier to implement), would you be able to compare the inference results against HMC at least in some small-scale problem?**

In actuality we implemented the method in Tensorflow-Probability (TFP), due to the extra flexibility that it affords. As we allude in the Introduction, we use the enclosed method as a piece of a larger modeling infrastructure that is applied to modeling problems that require flexibility and interpretability. TFP does provide methods for HMC/Nuts, though the implementation is different than Stan. There is some difficulty to us implementing HMC in our code-base however. In essence we designed the entire model source base to use batched data, for stochastic minibatch-based inference on the variational objective. We are attempting to make relevant changes so that we can use HMC, however, this requires extensive changes and we cannot promise that we will have this done by Monday/Tuesday.


- **Very limited coverage of inference, which is an important aspect even if carried out by an external software. Both theoretical and empirical evidence is missing, even though methods for evaluating the approximation quality are referred to.**

Please see above.


- **Figures 3 and 4 are pretty, but the spherical representation does not seem to add anything here and only results in the plots taking too much space while being slightly more difficult to read. More generally, the comorbidity example is a bit superficial and could have been developed a bit further.**

Thank you so much for the compliment, we also like the figures. We have been aware that they consume a lot of whitespace. With great sense of remorse, we have redone these figures so that they take up less space now (though they are now aesthetically vanilla). We think this sacrifice is worth it because it is giving us extra space to expand on inference and improve on overall exposition. We also are expanding on the comorbidity example.


- **How was ADVI applied? How did you check the approximation is good?**

Please see above.

- **Did you apply WAIC/PSIS-LOO or just say that it could be done?**

We have an implementation of WAIC that we use in the Supplemental Materials for comparing the choice of different link functions f/g. We are conceptualizing a follow-up paper that will be more-focused on learning those functions consistently and on how they impact the generating process.

From: ICLR.cc/2021/Conference/Paper3647/Authors
title: R1 - Intuition and motivation
comment: - **The model itself is a bit counter-intuitive, explaining a generally desirable property (latent variables following a reasonably chosen prior distribution, free from additional computational constraints) as a limitation and proceeds to replace it with a simplified mapping from inputs, but as the mapping itself is well justified in terms of sparsity the overall construct still makes sense.**

We see constraints in statistical problems as generally desirable as they remove ambiguity and add regularization. However, the intent of our method is to only have minimal impact on the generating process implied by the decoder portion of the model (which corresponds to the standard matrix factorization method). Other than the usage of a updated sparsity model, our generating process is the same that is used in standard HPF.

In all pure matrix factorization methods, some mapping $Y \to \pi(\theta|Y, ...)$ from data to representation exists, however, is not explicitly specified or necessarily well-posed. In other words, for a given decoding, an encoding may or may not exist that is sparse in an useful way -- regardless of decoder sparsity. The constraint removes this ambiguity completely. Furthermore, in the original HPF, a prior is still placed on the representation which is itself encouraged to be sparse. Hence, in terms of a priori restriction on the representation space, we believe that our method is at least equivalent to that of HPF.

- **The presentation angle is somewhat narrow and some connections are missed (e.g. the authors do not explain this as amortizing the inference for the latent variables, but explain the model in terms of encoders/decoders), but this is not a major issue.**

Thank you for the suggestion on how we should highlight the fact that our method amortizes inference for the latent variables. We are making this point more prominent (previously it was mentioned in passing under the caption of Fig 1). We have added the following text to our revision in preparation:

The distributions of the parameters of the encoder are learned self-consistently with other model parameters.
In the process, one is training not only the generative model, but also the subsequent Bayesian inference of mapping data to representation by learning the statistics of the posterior distribution,

\begin{equation}
\theta_u \vert \mathbf{y}_u \sim  \iint \pi(\theta_u \vert \mathbf{B},\varphi, \mathbf{y}_u)d\mathbf{B}d\varphi,
\end{equation}

where the generative process has been marginalized.
In short, the model of Eq. 2 uses the marginal posterior distribution of the encoding matrix $\mathbf{A}$ to reparameterize this Bayesian inference. 
Doing so makes it easier to apply the model to new data in order to compute representations.
It also allows us to impose desirable constraints on the representations themselves.


From our perspective, it is the mathematical connection between matrix factorization methods and autoencoders that is key to thinking of our overall method. The generative (decoder) process of a linear autoencoder is exactly a matrix factorization. Hence, by adding an encoding machinery to factorization (representation inference), a matrix factorization model becomes an autoencoder. 

The first advantage of thinking in terms of encoder/decoder structures is that principled extensions to the method, while retaining the full intrinsic interpretability, become evident. However, these extensions are not the main message behind our manuscript, and we see how they may be confusing the message, so we have reduced prominence of the GAM aspect in the revision. In a forthcoming work we will expand on theoretical aspects of extension of the the method to non-linearity. As the reviewer noted, "The paper proposes a PMF variant that replaces the free-form representation for latent factors from explicit mapping from inputs, to improve interpretability." We are devoting more attention to this main message in our revision.

From: ICLR.cc/2021/Conference/Paper3647/Authors
title: To R1/R2/R3: Thanks for the careful reviews and apologizes for these late posts 
comment: First, thank you for the careful critique of the work. Also, apologies for posting a reply so late in the review period. We are still getting used to this type of review process and did not realize that it is meant to be a more-interactive experience than that is typical of journals. While we are still preparing edits to our manuscript, we thought it would be good to respond to your reviews.

From: ICLR.cc/2021/Conference/Paper3647/Authors
title: Response to R2 - Practical value, bigger picture
comment: - **The manuscript is well written: precise and well articulated. It develops well the theoretical point that sparsity in encoding and decoding is important. However, the practical value of the contribution is not strongly demonstrated. In the real-life application, on comorbidity data, the sparsity is demonstrated as expected, but the benefit compared to other approaches is difficult to gauge, whether it is to a non-sparse approach, or an approach based on sparsifying priors. The benefit of the GAM decoding is not clear.**

In our revision, we are expanding on the comorbidity application to better explain why sparsity in the encoder says more than sparsity in the decoder, on a real application. This is important to us because our focus is on highlighting the benefit in terms of interpretability.

We incorporated the GAM to provide a principled way of extending the generative model to nonlinearity. In our Supplement we do some model comparison of such nonlinear models on synthetic data. We wish to expand more on the GAM aspect in a subsequent manuscript, where we will explore how the interplay between f and g influences the generative process.

- **While there are good theoretical arguments for the model, they only partly convince in practical terms: a full analytics pipeline has made aspects to it, and the arguments might not be as important as they seem in practice. It could help to perform more empirical comparison, and to study more the contribution in the context of full analyses. The empirical demonstrations show that the model exhibit the properties that it was designed for: sparsity in the decoding. What are practical consequences of these properties in real-life applications?**

We hope that by improving the exposition of the comorbidity application that the consequences of sparse encoding will be more-evident from a first-reading. In short, without sparse encoding, one cannot say that a particular representation coordinate is formed from any given subset of features. When judging interpretability, particularly in high-dimensional problems, sparsity is a desirable property because it allows one to focus on a few of many features at a time as a coherent concept. In our application, we can say that a given coordinate is determined by a specific subset of medical billing codes that co-occur often in the dataset. We know explicitly what the relative importance of each of the billing codes in the subset are on each representation coordinate, by reading entries off the encoding matrix. So for example, in our manuscript, we highlight that there is a particular representation coordinate that pertains broadly to respiratory disorders.

- **In the bigger picture, it is unclear to see how this contribution positions itself in terms of practical benefits in the vast literature on latent factors with distangling approaches (distangling autoencoders, various matrix factorizations including NMF with different losses).**

We appreciate this criticism and we are editing our manuscript to make it clear that we are ameliorating a gap between how these pre-existing methods are often interpreted, and what the models actually say. In short, through a modification (an imposition of sparse encoding), one can actually interpret these methods in the natural way that is often done. One can say that a a given set of features coherently loads together into a representation coordinate.

From: ICLR.cc/2021/Conference/Paper3647/Authors
title: Response to R2 - On training
comment: We would like to thank you for the interesting perspectives on our paper that we had not fully appreciated. Our focus is on interpretability - in particular addressing a gap between how similar factorization methods are interpreted versus what the models actually say. This gap appears superficially subtle but we believe it has large consequence. For this reason, we would like to refrain from straying too much from this central message. That said, we will address your comments in our revision that we will post this weekend. In the meantime, please see below for our responses.

- **The main practical benefit compared to sparsifying priors (as in Gopalan et al. (2014)) is that the contributed method can be solved with automatic differentiation variational inference and hence stochastic solvers, which makes it in theory easier to scale, though this improvement is not demonstrated empirically in the manuscript.**

Interestingly, it was issues with the scaling of HPF that lead us to first create our method. For HPF, the posterior distribution of an UxK representation matrix is learned at training. When we initially implemented HPF in custom Tensorflow code, we found that we could not scale it to suit our problem (to fit into GPU memory) where U was on the order of tens of millions and K is sometimes on the order of tens to hundreds.

We also did not see a clean way of performing batch inference on this problem. In standard HPF, the representation matrix is itself a model parameter. In minibatch optimization, one avoids storing the entire dataset  in GPU memory. However, one still needs to store the parameters  for the model (including the representation for HPF) because they are what is updated in each iteration of inference. In our revision we are expanding a bit on this theoretical discussion. We should note that the package hpfrec, that we use in our comparisons, has implemented a nonstandard minibatch algorithm for standard HPF. 

Regardless, it soon became clear to us that interpretability of HPF was most lacking. We would like the focus of our manuscript to be on interpretability. Even if HPF is faster, through various tunings and hacks for implementing minibatch training, it doesn't provide what our method provides  in terms of interpretability. That said, we will expand more on why having a constrained encoder transform leads to more memory-efficient training.

It is not however an apples to apples comparison when contrasting the computational cost of our method against that of standard HPF. This is mainly because we are using an updated sparsity model whereas HPF uses gamma distributions. For this reason, HPF has the advantage of possessing exact variational updates whereas we use stochastic gradient descent. Additionally, hpfrec, which we use for comparisons, is implemented in CPU code and our implementation is GPU. For these reasons, we hope that the reviewer will be sympathetic to our perspective that a direct benchmark comparison of the two algorithms is of limited use.

From: ICLR.cc/2021/Conference/Paper3647/Authors
title: Response to R3
comment: - **For me the weakness of the approach is that the solution proposed appears very ad hoc with no probabilistic basis. Particularly in matrix factorisation problems where there is often no easy way to establish ground truth: this often leads to setting hyperparameters arbitrarily. To be fair the authors point to ways of assessing predictive power which might help in this domain. A less ad hoc model might allow a more principled approach to choosing hyperparameters.**

Thank you for your critique of the method. We think we can help address your concern by relating our method to other matrix factorization approaches like PCA and SVD. It is true that the unsupervised problem lacks a ground truth. In our view, the objective of methods like PCA and SVD is to find a useful parameterization of the data in fewer dimensions, that  retains most of  the data's variability. PCA and SVD do so under a Gaussian noise model. HPF and our variant of HPF do so under a Poisson model.

The question then is whether such a  parameterization is useful - the answer to this question is problem-dependent and the general solution outside of the scope of our manuscript. However, in our revision we are expanding our exposition of the comorbidity application to better-describe how the output of our method can be used downstream in analysis. In our case, the method is useful because it first reduces the dimensionality of the data and second makes coarse-graining of the data, by grouping like datapoints together, more tractable. All this is done while maintaining interpretability of the overall model in terms of the original data features, at all times.

As to hyperparameters, the parameters within the model can influence the result. However, we have carefully tuned the parameters using standard Bayesian considerations. For instance, the scaling on the background process is set to a somewhat large multiple of the average value for each feature in the dataset. This  type of prior is known as a weakly-informative prior in the Bayesian literature. We set the scaling of the regularization so that the variance of  the priors is invariant with data size. This setting yielded the results demonstrated in Fig 2 -- in particular we are able to replicate the same results on matrices of different sizes, using the scalings provided in our manuscript. In that respect, there are few hyperparameters that need to be tuned.

From: ICLR.cc/2021/Conference/Paper3647/AnonReviewer5
title: Interesting HPF variant with slightly superficial presentation
review: Summary:
The paper proposes a PMF variant that replaces the free-form representation for latent factors from explicit mapping from inputs, to improve interpretability. A reasonable probabilistic formulation with carefully selected priors is provided, but inference is carried out using generic tools, and the method is illustrated on artificial data and a simple comorbidity application.

Reasons for score:
The proposed model is interesting and well motivated, and the technical details regarding the choice of priors for encouraging sparsity are good and match current recommendations. The model itself is a bit counter-intuitive, explaining a generally desirable property (latent variables following a reasonably chosen prior distribution, free from additional computational constraints) as a limitation and proceeds to replace it with a simplified mapping from inputs, but as the mapping itself is well justified in terms of sparsity the overall construct still makes sense. The presentation angle is somewhat narrow and some connections are missed (e.g. the authors do not explain this as amortizing the inference for the latent variables, but explain the model in terms of encoders/decoders), but this is not a major issue.

My biggest issue with the paper concerns inference. The description is limited, only referring to a specific algorithm (ADVI) without specifying all details (mean-field vs full-rank approximation), and there is no discussion or analysis on how well it works. The authors do say that the algorithm converged fast, but do not show this in any experiment. More importantly, the convergence does not yet guarantee the approximation is good and there are well-known cases for which ADVI does not really work that well. Expanding both the discussion and empirical demonstration of this would be critical. Now you say "one may use..." and "one can access..." with references to specific techniques for evaluating the quality, which gives the impression you have not actually done that.  As I presume you implemented the model in Stan (no point in using ADVI if not; there are better stochastic VI methods around that are also easier to implement), would you be able to compare the inference results against HMC at least in some small-scale problem?

Pros:
1. The idea is insightful and matches well the application needs.
2. The priors match current literature on suggestions for sparsity-inducing priors.

Cons:
1. Very limited coverage of inference, which is an important aspect even if carried out by an external software. Both theoretical and empirical evidence is missing, even though methods for evaluating the approximation quality are referred to.
2. Figures 3 and 4 are pretty, but the spherical representation does not seem to add anything here and only results in the plots taking too much space while being slightly more difficult to read. More generally, the comorbidity example is a bit superficial and could have been developed a bit further. 

Questions:
1. How was ADVI applied?
2. How did you check the approximation is good? Did you apply WAIC/PSIS-LOO or just say that it could be done?

Modifications after discussion:
Increased score by one since the revised paper clarifies the missing details on inference and also improves the motivation.
rating: 7: Good paper, accept
confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct

From: ICLR.cc/2021/Conference/Paper3647/AnonReviewer2
title: Good theory, not completely convincing in practice
review: This manuscript revisits the hierarchical Poisson matrix factorization (HPF) promoting sparsity in both the representation and the decoding function with a Horseshoe+ prior. Sparsity on both sides is put forward for interpretation purposes, and to create a column-clustering property. In addition, the proposed approach caters for non-linear decoding using a GAM model.

The main practical benefit compared to sparsifying priors (as in Gopalan et al. (2014)) is that the contributed method can be solved with automatic differentiation variational inference and hence stochastic solvers, which makes it in theory easier to scale, though this improvement is not demonstrated empirically in the manuscript.

The manuscript is well written: precise and well articulated. It develops well the theoretical point that sparsity in encoding and decoding is important. However, the practical value of the contribution is not strongly demonstrated. In the real-life application, on comorbidity data, the sparsity is demonstrated as expected, but the benefit compared to other approaches is difficult to gauge, whether it is to a non-sparse approach, or an approach based on sparsifying priors. The benefit of the GAM decoding is not clear.

While there are good theoretical arguments for the model, they only partly convince in practical terms: a full analytics pipeline has made aspects to it, and the arguments might not be as important as they seem in practice. It could help to perform more empirical comparison, and to study more the contribution in the context of full analyses. The empirical demonstrations show that the model exhibit the properties that it was designed for: sparsity in the decoding. What are practical consequences of these properties in real-life applications?

In the bigger picture, it is unclear to see how this contribution positions itself in terms of practical benefits in the vast literature on latent factors with distangling approaches (distangling autoencoders, various matrix factorizations including NMF with different losses).


rating: 6: Marginally above acceptance threshold
confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct

From: ICLR.cc/2021/Conference/Paper3647/AnonReviewer3
title: Sparse Encodings for counting matrix factorisation.
review: ** Description

This paper provides a new approach for finding a sparse encoding of count data matrices and hence automatically achieve feature selection.

** Pros

The proposed technique is clearly efficient and practical.  It identifies a failing in traditional hierarchical Poisson matrix
factorisation (HPF) and proposes a solution.  This approach is tested on real world datasets where its usefulness is demonstrated.

** Cons

For me the weakness of the approach is that the solution proposed appears very ad hoc with no probabilistic basis.  Particularly in
matrix factorisation problems where there is often no easy way to establish ground truth: this often leads to setting hyperparameters arbitrarily.  To be fair the authors point to ways of assessing predictive power which might help in this domain.  A less ad hoc model might allow a more principled approach to choosing hyperparameters.

rating: 6: Marginally above acceptance threshold
confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct