Paper: ImplicitSLIM and How it Improves Embedding-based Collaborative Filtering (ICLR-2024)

Reviewer comment (Theory):
The paper introduces various settings for ImplicitSLIM, but it would be beneficial to analyze and summarize the computational complexity of different variants respectively to provide a clearer understanding of their efficiency.

Author response:
We fully agree with this comment (see also Q2 of Reviewer uknn). In the updated version of the paper, we have added Figure 3 in Appendix E.1 to visualize the tradeoff between computation time, embedding dimensionality, and ranking scores. In particular, they show how much ImplicitSLIM is faster and more memory efficient compared to EASE, and also show how ImplicitSLIM boosts matrix factorization in terms of convergence time as well as in terms of performance.

Reviewer comment (Experiments):
The paper could benefit from more detailed experimental studies on the influence of hyperparameters. Given the presence of multiple regularization and optimization items in the method, it would be more illustrative to have the model performance w.r.t. different parameter settings.

Author response:
The primary hyperparameter that, in our opinion, indeed deserves extensive experiments is the embedding dimensionality; we have presented comparative studies of different embedding dimensions in Figures 1 and 3 and Table 4. As for the other hyperparameters, there are no significant tradeoffs there, and we just look for their optimal values with Bayesian hyperparameter search.

Reviewer comment (Evaluation):
Although ImplicitSLIM has shown competitive performance against traditional linear encoders, it shows limitted improvement on deeper models like UltraGCN and RecVAE.

Author response:
We note that the application of ImplicitSLIM to all deep models in the experimental section statistically significantly accelerates their performance (except for one very specific case, H+Vamp(Gated) trained on the Netflix dataset and evaluated with Recall@k, in all other cases the speedup is significant). 

In addition, we note that the application of ImplicitSLIM to these models improves convergence, with minor computational and memory overhead thanks to relatively low embedding dimensionality in deep models; for example, RecVAE+ImplicitSLIM achieves the best score of RecVAE almost twice faster than RecVAE itself.

Therefore, we believe that applying ImplicitSLIM to deep recommendation models such as RecVAE and H+Vamp may bring advantages other than just improved metrics, and our experiments show that ImplicitSLIM is beneficial for all models where we have been able to apply it.

Reviewer comment (Experiments):
not a standalone approach which makes training less straightforward

Author response:
Yes, we agree with this statement in the general case: ImplicitSLIM is indeed a modification aimed to improve existing recommender systems rather than a separate model.

However, in order to mitigate this issue as much as possible, we have performed extensive experiments applying ImplicitSLIM to the simplest suitable models, MF and PLRec. Moreover, one of experimental setups (called “ImplicitSLIM init” in the paper) assumes that the item embeddings matrix for matrix factorization is initialized by ImplicitSLIM output, and there are no other computations during the training phase (user embeddings for held-out users are computed during the test phase given item embeddings). This learning procedure is indeed not performed in an end-to-end fashion.

In Section 4.2, we also make an attempt to make the training of ImplicitSLIM more straightforward; there, we show that a model regularized with the SLIM regularizer approximately matches a model regularized with ImplicitSLIM.

Reviewer comment (Novelty):
applicable to embedding-based models only

Author response:
Yes, we agree that our approach deals only with embedding-based models and is not applicable, e.g., to recommendations based on nearest neighbors. However, most currently used recommender systems employ embedding-based models, and we believe that this does not significantly hinder the applicability and importance of our results.

Reviewer comment (Reproducibility):
the source code is not provided

Author response:
By the conference guidelines for anonymity, we indeed have not published the source code openly but it has been attached in the supplementary files here in the submission.

Reviewer comment (Theory):
In Section 3 Proposed Approach, the authors should explain more on the formular choices when developing ImplicitSLIM. For example, why do we use LLE, but use the neighbourhood of NN(i) = {1,2,…,I} \ {i} to make it ‘global’ (Section 3.1)? Why do we drop the sum-to-one constraint (Section 3.2) (the authors did mention they have no good reasons, but why)? In my opinion, Section 3 is very important and the authors should provide deeper explanations and discussions about this. Otherwise it’s very hard to convince the readers.

Author response:
Thank you for this remark – we agree that this could be explained better. First, the chosen neighborhood function and dropping the sum-to-one constraint are both necessary to perform the trick in Section 3.2 that makes it possible to avoid inverting huge matrices. 

Considering EASE as a special case of the first step of LLE gives us additional motivation to use this neighborhood function, as we mention in Section 3.1; in general, the similarity of EASE and LLE motivates us to use LLE as a model for item/user representations learning.

The sum-to-one constraint at the first step implicitly ensures the sum-to-zero constraint at the second step of the original LLE, but in this work we propose a different second step, which is defined as an unconstrained optimization problem, so we have a technical reason to drop the sum-to-one constraint. We have updated Sections 3.1 and 3.2 to explain this more clearly.

Reviewer comment (Experiments):
In the first paragraph after Figure 2, the authors mentioned that “In addition, ALS applied to MF regularized by …. about 5x faster”, I may missed it but where do we find the ‘5x faster’ comparison?

Author response:
Thank you for noticing this! In the updated version of the paper, we have added Fig. 3 in Appendix E.1 to visualize the tradeoff between computation time, embedding dimensionality, and ranking scores. In particular, it shows how much ImplicitSLIM is faster and more memory-efficient compared to EASE, and also shows how ImplicitSLIM boosts matrix factorization in terms of convergence time as well as in terms of performance.

Reviewer comment (Experiments):
In Figure 1, why ImplicitSLIM init + SLIM reg and SLIM-LLE init cannot have results with > 500 embedding dimension? Could we please add an appendix on the ‘high computational costs’?

Author response:
Both of these setups, unlike the others, require huge matrix inversions. Moreover, SLIM-LLE init setup requires the computation of eigenvectors for the smallest eigenvalues, and the ImplicitSLIM init + SLIM reg setup could not be solved with ALS, so we had to employ gradient descent for this problem. As a result, computational costs for these two settings are much higher, especially when the dimensionality of embeddings is large; hence we have evaluated these setups with embedding dimensions equal to 500 and lower. In the updated version of the paper, we have expanded Appendix C to explain it.

Reviewer comment (Evaluation):
In Table 1, the performance results are not really impressive. For example, in the Appendix E.1, the authors mentioned that “Moreover, ImplicitSLIM is also faster than EASE… could replace EASE in some cases due to lower computational time and comparable performance.”. Please explain which cases?

Author response:
Please see the answer to Q2 above: we have added Figure 3 in Appendix E.1 (and a corresponding explanation) to visualize how ImplicitSLIM outperforms EASE in terms of wall clock time and memory at the cost of a slight performance drop.

Reviewer comment (Presentation):
References should be sorted as the current version is hard to follow (minor)

Author response:
We apologize but we didn’t understand this remark: the references are sorted alphabetically by the first author’s last name; we have checked last year’s ICLR papers and saw the references sorted in the same way…