Paper: Constraining Non-Negative Matrix Factorization to Improve Signature Learning (ICLR-2024)

Reviewer comment (Novelty):
the proposed SMF could be viewed as a combination of NMF and SEM, so its novelty is limited.

Author response:
**The proposed SMF could be viewed as a combination of NMF and SEM, so its novelty is limited.**

We thank the reviewer for bringing out this point. We will improve the explanation of how our method relates to NMF and SEM in the revised version of the manuscript.

SMF is not a straightforward combination of NMF and SEM. It can be thought of as a new model that relies on NMF and constrains the space of solutions by using ideas inspired by SEM. SEM itself is not directly plugged into the model. We proposed a “self-expressive term” that differs from the original term used by SEM. One of our insights was including the low dimensional representation into the self-expressive term and observing that this results in representations that can preserve better the proximities from original sub-spaces. We then derived the multiplicative rules of our model and showed that SMF can take advantage of both NMF and SEM strategies.

Reviewer comment (Novelty):
It is somewhat incremental improvement from SEM. The results are comparable, not significantly better.

Author response:
We thank the reviewer for bringing out this point. Although we agree that some results are comparable for recommendation tasks, and we do mention this in the paper, we think that this is not necessarily a weakness, since the main goal of the paper is to show that we can obtain more meaningful representations. Thus, our main results are shown in the Signature Analysis section, whereas the recommendation tasks illustrate that better representations can lead to better or comparable predictive power. Nevertheless, it is interesting to notice that in the overall picture of the recommendation tasks, SMF has a significantly lower RMSE than SEM while also maintaining a higher precision than NMF.

Reviewer comment (Evaluation):
How to interpret the meaningful relationships in the low dimensional representations need more elaboration.

Author response:
We thank the reviewer for this suggestion. We will clarify the interpretation analysis and what we mean by “meaningful relationships” in the revised version of the manuscript. 

When we say that our model captures meaningful relationships between objects, we mean that the similarity between two signatures reflects similarities in terms of latent processes underlying the data. Since we assume that the latent features of our model encode relevant processes, we expect that the learned signatures of two objects will be similar if they share latent processes, and dissimilar, otherwise. To test this hypothesis, we leverage additional knowledge about the objects, which was not utilized during the learning process. This enables us to cluster objects that are expected to share latent processes. For instance, drugs within the same ATC category are likely to share molecular mechanisms. If our hypothesis holds true, we expect that drugs within the same ATC category will exhibit more similar signatures than those from different categories.

Reviewer comment (Evaluation):
Some metrics are not well explained in the paper. What is the formula to compute Z-score?

Author response:
Thank you for pointing this out. We will include the formula and explanation in the revised version of the manuscript.

We use the Z-score from the two-sample z-test to measure the difference between signature similarities of objects that are from the same group and objects that are from different groups. Given the average signature similarity of objects that are in the same group ($\mu_in$), the average signature similarity of objects that are in different groups ($\mu_out$), and the corresponding standard deviations (\sigma_{in} and \sigma_{out}), the z-score formula is:

$Z = \frac{ \mu_{in} - \mu_{out}}{\sqrt{\frac{\sigma_{in}^2}{n_{in}} + \frac{\sigma_{out}^2}{n_{out}}}}$,

where $n_{in}$ and $n_{out}$ are the number of object pairs intra- and inter-groups, respectively. 

We can interpret the z-score as a normalized distance that measures how different two distributions are by adjusting the difference between the means according to their standard deviation.

Reviewer comment (Evaluation):
Are the low dimensional data representations stable? How would it change if small amount of data are added or deleted from association matrix?

Author response:
As evidenced by our experiments, the generated signatures consistently yield stable results across different runs. To address the missing data concern, we ran experiments similar to the ones detailed in Section 4.3 for signature analysis, but we specifically trained the model using only 90% of the known interactions in the SIDER dataset. Notably, even with this reduced dataset, the difference in results compared to using the full matrices is minimal.

Here we reported the means of the Z-Score difference between the means for both models NMF and SMF:

| Models | ATC $1^{st} Level$ | ATC $2^{nd} Level$ | ATC $3^{rd} Level$ |
| --------- |  :--------------: | :--------------: | :--------------: |
| NMF | $-0.1013$ | $0.3934$ | $0.7411$ |
| SMF | $2.9746$ | $7.0157$ | $8.5293$ |

We plan to run similar experiments for the other datasets and include them in the revised version of the manuscript.

Reviewer comment (Evaluation):
In figure 2, Precision at top-K decreases in the proposed method, especially for movielens dataset. Is there any explanation?

Author response:
For that particular case, the first top-K recommendations already contain a high amount of true positives, which results in high precision. As the value of K increases, the amount of true positives does not increase accordingly, as most of them were already captured in the highest value for K. Note that the same trend can be observed for SEM when examining the precision score for K equal to 20 and upwards.

Reviewer comment (Experiments):
NMF suffers from local minima, and this needs to be studied extensively, even empirically.

Author response:
We acknowledge that the optimization process described in this paper suffers from local minima, similar to many other machine learning frameworks in the literature, and we do not believe this is a significant problem for the task of learning meaningful representations. Our reported results obtained from different runs of our framework with different initializations were shown to be stable, indicating that SMF is able to consistently propose similar solutions. We will modify the paper to clarify that the proposed optimization process does not offer any robustness to local minima.

Reviewer comment (Experiments):
The number of experiments is too small to get some significant guarantees, and a much bigger pool of benchmarks is needed.

Author response:
To address this concern, we selected an additional dataset, called ModCloth [2], where users assign ratings to clothes. Due to the time constraints in the rebuttal phase, we conducted experiments only on one additional dataset. We also had to skip the exhaustive hyperparameter tuning and perform a subsetting of the dataset to manage computational time. In the revised version of the manuscript, we intend to include the full set of experiments on the complete or a less subsetted, version of this dataset.

After randomly subsetting the data, we obtain a data matrix $C \in \mathbb{R}^{n_c \times m_c}$ for $ n_c = 5419$ clothing items and $m_c = 32.089$ users with a density of $\approx 0.05 \%$. Each clothing item belongs to $1$ of $66$ different categories that are used for the signature analysis. The results shown in the table below correspond to one run of NMF, SEM and SMF. Similarly to the other datasets, SMF clearly outperforms the other methods and achieves better category separation in contrast to NMF.



| Models | RMSE    | Correlation | AUROC  | AUPRC   | Z-score |
| ------ | :-----: | :---------: | :----: | :-----: | :-----: |
| NMF    | $1.5847$ | $0.0801$ | $0.6871$ |$1.64\mathrm{e}{-4}$ | $62.34$ |
| SEM    | $4.3112$ | $0.0423$ | $0.6230$ |$2.50\mathrm{e}{-4}$ | (-) |
| SMF    | $1.3291$ | $0.1012$ | $0.7515$ |$2.84\mathrm{e}{-4}$ | $238.35$ |


[2] Misra, Rishabh, Mengting Wan, and Julian McAuley. "Decomposing fit semantics for product size recommendation in metric spaces." Proceedings of the 12th ACM Conference on Recommender Systems. 2018.

Reviewer comment (Experiments):
Scalability is another issue not examined, which is strongly tied to runtimes and convergence steps needed.

Author response:
Here we address the issue of the scalability of our method, in the following table illustrates the mean time of each iteration and the mean number of steps necessary for convergence for the Movielens and the Cloth datasets. We will modify the paper accordingly to report these characteristics.

**Time per iteration in seconds:**

| Models | Movielens    | ModCloth |
| ------ | :----------: | :---: |
| NMF    | $0.01531$      | $1.4287$|
| SEM    | $0.04338$      | $7.3948$|
| SMF    | $0.02797$      | $7.4693$|

**Iterations for convergence:**

| Models | Movielens    | ModCloth |
| ------ | :----------: | :---: |
| NMF    | $1868.20$      | $467$   |
| SEM    | $244.20$       | $292$   |
| SMF    | $1427.733$     | $1032$  |