Paper: Safe Collaborative Filtering (ICLR-2024)

Reviewer comment (Writing):
Some concepts may need to be explained in more detail in the main paper (e.g.- the reformulation of the CVaR minimization objective, smoothing techniques for quantile regression, NR algorithm etc.)

Author response:
Next, we also appreciate your suggestion to provide more detailed explanations of specific concepts in our main paper. Owing to page limitations, we directed these explanations to the Appendix in our Supplementary Material. In the revised version, we will clarify these concepts further, as you have suggested. We believe this revision will enhance the paper's readability and appeal to a broader audience. We greatly appreciate your valuable input.

Reviewer comment (Experiments):
As far as I understand, the main motivation of this paper is tail performance. (in Abstract) However, I found that the main contribution of this paper focuses on parallel computation and Scalability. (in the manuscript.) What if we just put some high weight on the tail users? or What if we just compute gradient only with bottom-$\alpha$% losses?

Author response:
> As far as I understand, the main motivation of this paper is tail performance (in Abstract). However, I found that the main contribution of this paper focuses on parallel computation and scalability (in the manuscript).

The main focus of this work is tail performance optimization, with a particular emphasis on scalability. We believe that recommender systems are particularly critical applications of machine learning where scalability is paramount. Therefore, instead of applying existing CVaR optimization methods (e.g., CVaR-MF), we develop an approach to achieve scalability in large-scale recommender systems.

> What if we just compute gradient only with bottom-$\alpha$% losses?

We interpreted your comment as focusing on the bottom-$\alpha$% of users rather than individual user-item interactions. 
The suggested approach is essentially consistent with CVaR-MF, which focuses on the bottom-$\alpha$% users. Hence, **we believe this hypothesis has been thoroughly examined in our manuscript.**

> What if we just put some high weight on the tail users?

There are various weighting strategies for this approach. In this light, CVaR-MF can be viewed as a naive implementation that equally weights tail users while disregarding others. Our algorithm can also be viewed as one that assigns smooth weights to users,　represented by dual variables $\mathbf{z}$. This leads to an ALS-type algorithm with user reweighting (Section 3.3), which achieves excellent scalability.
Therefore, **we believe our manuscript comprehensively examines both of the approaches you've suggested.**

Reviewer comment (Experiments):
The main experimental result focuses on that SAFER2 shows comparable results when $\alpha=1$ and superior results when $\alpha=0.3$. When $\alpha=1$, there are many recent methods outperforming Multi-VAE. e.g., LightGCN, SGCL, and in my opinion, the pair-wise and list-wise loss functions outperform the point-wise loss. When $\alpha=0.3$, there are many recent methods enhancing the performance of tail users.

Author response:
> When $\alpha=1.0$, there are many recent methods outperforming Multi-VAE. e.g., LightGCN, SGCL.

Our evaluation operates within the strong generalization setting. However, **the suggested GNN-based methods cannot be evaluated in this setting.** These models cannot make predictions for new users. In fact, we could find few references that compare such methods in a strong generalization setting; Mao et al. (2021) have attempted to compare their GCN-type method in this protocol with some modifications to their method (see Section 4.8 of Mao et al. (2021)).

Finally, we would like to emphasize that we do not intend to claim that SAFER2 is a state-of-the-art method regarding the average ranking quality; in fact, we have reported the superior average performance of Mult-VAE on ML-20M in Table 1. We are confident our experiments sufficiently support our claim that SAFER2 improves tail performance while maintaining scalability and average ranking quality competitive with iALS, which is the fastest method with excellent ranking quality.

> When $\alpha=1.0$, in my opinion, the pair-wise and list-wise loss functions outperform the point-wise loss.

To our knowledge, **pairwise loss functions are not considered competitive in this setup, and they are not commonly evaluated as baselines**; for example, see Section 7 of Steck et al. (2020), Section 4.4 of Liang et al. (2018), and Table 3 of Rendle et al. (2022). Also noteworthy in this context is the conclusion of Rendle et al. (2022), *"It is striking that iALS achieves competitive or better performance than models learned with ranking losses (LambdaNet, WARP, softmax) which reflect the top-n recommendation task more closely."*  Furthermore, **we have already compared listwise loss functions.** The multinomial likelihood of Mult-VAE is essentially a listwise loss (a.k.a. softmax cross-entropy); see Eq. (2) of Liang et al. (2018).
> When $\alpha=0.3$, there are many recent methods enhancing the performance of tail users.

Thank you for your feedback. We would appreciate it if you could provide references for this.

Reviewer comment (Experiments):
If you trained two SAFER2 models with $\alpha=1$ and $\alpha=0.3$, that is somewhat unfair for the baselines (e.g., VAE-CF).

Author response:
> Did you train two SAFER2 models with $\alpha=1.0$ and $\alpha=0.3$? If so, that is somewhat unfair for the baselines (e.g., VAE-CF).

**We set $\alpha=0.3$ for both CVaR-MF and SAFER2 throughout all experiments.**
Although we have already discussed this in Section 5.1 and Table 3 in the supplementary material, we acknowledge your suggestion and will improve the clarity of this point in the revised manuscript.

Reviewer comment (Presentation):
Equation numbers are needed for each equation when finding the location of the equation by ordering.

Author response:
> Equation numbers are needed for each equation when finding the location of the equation by ordering.

We will ensure the equation numbers are displayed as you suggested, and we appreciate your suggestion.