Paper: Debiased Collaborative Filtering with Kernel-Based Causal Balancing (ICLR-2024)

Reviewer comment (Theory):
The optimization formulation for balancing may be computationally intractable if the number of items and/or the number of users are large.

Author response:
We thank the reviewer for raising an interesting concern.

- First, **the balancing weights $w_{u, i}$ are parameterized using a deep neural network model $g(x_{u, i}; \phi_w)$**, which ensures the model scalability and the model parameters $\phi_w$ could be much less than the number of user-item pairs when the number of items and/or the number of users is large. 

- Second, **we can write the primal optimization problem to the Lagrange dual problem**, in which the number of parameters is the same as the number of constraints $O(J)$, and is independent with the user numbers and item numbers.

- Third, for the optimization problem, **we can uniformly sample a batch in $\mathcal{D} = \[u_1, u_2, ..., u_m\] \times \[i_1, i_2, ..., i_n\]$ to compute $\tau^{(j)}$ for $j = 1, 2, ..., J$**, instead of using all $m \cdot n$ user-item pairs that concerns the computational efficiency.

Reviewer comment (Experiments):
It is unclear how to incorporate the proposed method into the modern deep learning (DL) based recommender system and to test its effectiveness in a field setting.

Author response:
We thank the reviewer for the useful comments. **The proposed methods are model-agnostic**, which means the balancing weight model $g(x_{u, i}; \phi_w)$ and the prediction model $f(x_{u, i}; \theta)$ can adopt the modern deep neural networks. To show this, **we add the experiment on a large-scale industrial dataset Product** using **neural collaborative filtering (NCF) [1]** as the base model to empirically show the performance of the proposed methods in Appendix B, Table 2. The results are shown below.
||AUC|NDCG@20|F1@20|
|:--:|:--:|:--:|:--:|
|NCF| 0.823±0.001|0.575±0.002|0.166±0.002|
|+CVIB|0.820±0.002|0.544±0.004|0.162±0.002|
||
|+IPS|0.822±0.003|0.579±0.006|0.169±0.003|
|+SNIPS|0.833±0.002|0.586±0.002|0.178±0.002|
|+ASIPS|0.832±0.002|0.583±0.003|0.178±0.002|
|+IPS-V2|0.835±0.001|0.588±0.002|0.181±0.001|
|+CBIPS|0.832±0.001|0.584±0.003|0.178±0.003|
|+WKBIPS|0.833±0.002| 0.588±0.003|0.181±0.003|
|+AKBIPS|**0.836*±0.001**| **0.592*±0.002**|**0.183*±0.002**|
||
|+DR |0.758±0.004|0.526±0.003|0.146±0.003|
|+DR-JL|0.832±0.001|0.581±0.003|0.178±0.002|
|+MRDR-JL|0.833±0.001|0.585±0.002|0.179±0.001|
|+DR-BIAS|0.834±0.002|0.585±0.003|0.178±0.002| 
|+DR-MSE|0.834±0.002|0.587±0.002|0.180±0.001|
|+MR|0.837±0.001|0.588±0.002|0.181±0.002|
|+TDR-JL|0.834±0.002|0.582±0.002|0.179±0.003|
|+SDR|0.835±0.002|0.587±0.003|0.179±0.002|
|+DR-V2|0.837±0.002|0.586±0.004|0.182±0.002|
|+CBDR|0.834±0.002 |0.586±0.003|0.179±0.003|
|+WCBDR|0.837±0.002| 0.588±0.003|0.180±0.002|
|+ACBDR|**0.840*±0.002**|**0.590*±0.003**|**0.183±0.002**| 
|| 

The experiment results show that the proposed kernel balancing methods achieve overall performance improvement compared to the baseline methods.

Reviewer comment (Reproducibility):
How to update the weights for the problems where the number of items and/or the number of users are large?

Author response:
We thank the reviewer for raising an interesting concern.

- First, **the balancing weights $w_{u, i}$ are parameterized using a deep neural network model $g(x_{u, i}; \phi_w)$**, which ensures the model scalability and the model parameters $\phi_w$ could be much less than the number of user-item pairs when the number of items and/or the number of users is large. 

- Second, **we can write the primal optimization problem to the Lagrange dual problem**, in which the number of parameters is the same as the number of constraints $O(J)$, and is independent with the user numbers and item numbers.

- Third, for the optimization problem, **we can uniformly sample a batch in $\mathcal{D} = \[u_1, u_2, ..., u_m\] \times \[i_1, i_2, ..., i_n\]$ to compute $\tau^{(j)}$ for $j = 1, 2, ..., J$**, instead of using all $m \cdot n$ user-item pairs that concerns the computational efficiency.

Reviewer comment (Novelty):
It is advisable to include a discussion of the relationships or distinctions between this work and prior research in the Related Work section.

Author response:
**Response:** Thank you for your kind advice. As suggested by the reviewer, **we add a discussion of the relationships and distinctions between our work and prior research** in the Related Work section **for both “Debiased Recommendation” and “Covariate Balancing in Causal Inference” parts.**

Reviewer comment (Theory):
In terms of the Adaptive Kernel Balancing method, I am confused why balancing the kernel functions with maximal $|\alpha_{s,t}|$ can contribute the most to the $e_{u,i}$. It requires more explanation to improve the readability.

Author response:
**Response:** Taking CBIPS as an example, in the last equation on Page 5, we use the universal property and the representer theorem to show the bias of CBIPS estimator is 

$$\operatorname{Bias}^2(\mathcal{L}_{\mathrm{CBIPS}}(\theta))$$

 $$= \[\frac{1}{|\mathcal{D}|}\sum_{(u,i) \in \mathcal{D}} (o_{u, i} w_{u,i}-1)(\sum_{(s, t) \in \mathcal{D}}\alpha_{s, t}K(x, x_{s, t})) \]^2$$
 $$= \[\frac{1}{|\mathcal{D}|}\sum_{(s, t) \in \mathcal{D}}\alpha_{s, t}\sum_{(u,i) \in \mathcal{D}} (o_{u, i} w_{u,i}-1)(K(x, x_{s, t})) \]^2$$
 $$= \[\frac{1}{|\mathcal{D}|}\sum_{(s, t) \in \mathcal{D}}\alpha_{s, t}\sum_{(u,i) \in \mathcal{D}} (o_{u, i} w_{u,i}K(x, x_{s, t}) - K(x, x_{s, t})) \]^2.$$
If we balance $K(x, x_{s, t})$, we have $$\frac{1}{|\mathcal{D}|}\sum_{  (u,i) \in \mathcal{D} }  o_{u, i}w_{u,i} K(x, x_{s, t})  =   \frac{1}{|\mathcal{D}|}\sum_{ (u,i) \in \mathcal{D} }  K(x, x_{s, t}),$$ that is,
$$\frac{1}{|\mathcal{D}|}\sum_{(u,i) \in \mathcal{D}} (o_{u, i} w_{u,i}K(x, x_{s, t}) - K(x, x_{s, t})) = 0.$$

Therefore, selecting the $K(x, x_{s, t})$ with maximal $|\alpha_{s, t}|$ to balance can effectively and rapidly reduce $\operatorname{Bias}(\mathcal{L}_{\mathrm{CBIPS}}(\theta))$.

Reviewer comment (Experiments):
The authors adopt the method of balancing the kernel functions with maximal $|\alpha_{s,t}|$ to improve efficiency, but there is no corresponding experiment to validate this.

Author response:
**Response:** Thanks for pointing out this issue and we really apologize for the **typo** here. **We intend to say "effectiveness" instead of "efficiency".** Nevertheless, we still **add the experiments to analyze the trade-off between the effectiveness and the efficiency of CBDR and ACBDR** in Appendix B, Table 3. The results are shown below.

| Number of balancing functions | Metrics | CBDR-Gaussian | ACBDR-Gaussian |
|:--:|:--:|:--:|:--:|
|$J=3$| AUC     | 0.681±0.003   | 0.688±0.003    |
|| NDCG@5  | 0.650±0.003   | 0.657±0.003    |
|| F1@5    | 0.323±0.002   | 0.326±0.002    |
|| Time(s) | 389.28±17.51  | 664.99±12.37   |
||
|$J=5$| AUC     | 0.683±0.002   | 0.694±0.002    |
|| NDCG@5  | 0.652±0.003   | 0.664±0.002    |
|| F1@5    | 0.325±0.002   | 0.332±0.002    |
|| Time(s) | 394.51±16.61  | 678.13±15.43   |
||
|$J=10$| AUC     | 0.682±0.002   | 0.694±0.002    |
|| NDCG@5  | 0.650±0.003   | 0.663±0.002    |
|| F1@5    | 0.324±0.003   | 0.330±0.003    |
|| Time(s) | 399.91±8.58   | 719.63±23.79   |
||
|$J=20$                           | AUC     | 0.683±0.002   | 0.695±0.002    |
|| NDCG@5  | 0.651±0.002   | 0.664±0.003    |
|| F1@5    | 0.325±0.002   | 0.332±0.003    |
|| Time(s) | 389.11±22.39  | 727.29±26.76   |
||
|$J=50$ | AUC     | 0.684±0.002   | 0.695±0.002    |
|| NDCG@5  | 0.652±0.003   | 0.664±0.002    |
|| F1@5    | 0.324±0.003   | 0.333±0.002    |
|| Time(s) | 407.89±11.67  | 722.43±25.55   |
||

The ACBDR method stably outperforms CBDR with varying number of balancing functions, which shows the **effectiveness** of the proposed adaptively balancing method. **For efficiency, ACBDR takes about "doubled running time" to converge than KBDR,** since it needs to sort all the coefficients of kernel functions, which is time-consuming when the number of user-item pairs is large. When $J > 5$, the performance of the AKBDR does not improve as the growth of $J$, but the running time increases.

Reviewer comment (Writing):
The definition of $o_{u,i}$ is not clarified clearly.

Author response:
**Response:** We thank the reviewer for pointing out this issue, and **we have added the formal definition of $o_{u, i}$ in Section 3.1, and carefully polished the manuscript in our revised version to avoid typos and improve readability.**

Reviewer comment (Writing):
This paper lacks a sufficient motivation for the importance of carefully selecting the function class for balancing. The authors do discuss the computational cost associated with choosing numerous balancing functions in Section 3.2. However, the original paper [1] suggests that even selecting a single function for balancing can yield performance improvements. It is essential to elucidate why the choice of multiple balancing functions is necessary and why the specific type of function is critical. In my view, the motivation may be derived from Corollary 1 to some extent, which asserts that certain types of balancing functions introduce bias (correct me if I am wrong). Nonetheless, it would be valuable to provide a more insightful explanation in the introduction section. I would like to see an illustrative example, which could enhance comprehension of the motivation.

Author response:
We thank the reviewer for the careful and insightful thoughts, and **have revised the Introduction Section** in our revised manuscript.

-	First, we **agree** that “the original paper [1] suggests that even selecting a single function for balancing can yield performance improvements”, and **our experimental results also prove this.**

-	However, on the one hand, **it is not realistic to balance all possible functions for a specific model** using only finite samples.

-	On the other hand, **balancing only a single function is not sufficient for IPS and DR methods to achieve unbiased learning** (see Corollary 1 for the formal theoretical results).

-	Thus, **it is necessary to discuss which functions should be more favored to be balanced for the IPS and DR estimators**, resulting in **smaller estimation biases of the ideal loss** and enhanced performance of unbiased learning.

Reviewer comment (Theory):
The optimization problems in Sections 4.2 and 4.4 are perplexing. The rationale behind minimizing the sum of \( g(\cdot) \log g(\cdot) \) as opposed to a standard cross-entropy loss is unclear and confusing. The purpose of constraining the sum of \( o_{u,i}g(\cdot) \) to be one and the meaning of the third constraint require more detailed explanation and motivation. The authors should clarify the significance and reasoning behind each statement in the proposed optimization problem.

Author response:
We thank the reviewer for pointing out this issue, and **have added detailed explanations regarding the optimization problems in Sections 4.2 and 4.4.** Both of the optimization problems consist of the following three features.

-	First, **the objective function** is the **empirical entropy of the balancing weights**, by the principle of maximum entropy, it **reaches the maximum value when the balancing weights are uniform**, thus **effectively avoiding high variance due to extremely small propensities.**

-	Second, **the balancing constraints** are imposed to equalize the selected covariate functions between the observed and missing samples.

-	Third, **two normalization constraints are imposed**, which implies that the weights sum to the normalization constant of one, and the **nonnegativity of the balancing weights**, making the empirical entropy as the objective function well-defined.

-	For the optimization problem in Section 4.4, we **further introduce a slack variable $\xi_j$** for each balancing function and **a pre-specified threshold $C$**, since **achieving strict balancing on all balancing functions is usually infeasible** as the number of balancing functions increases.

Reviewer comment (Theory):
Several theorems and corollaries, such as Theorem 1, Corollary 1, and Lemma 2, lack supporting proofs. In particular, the proof of Corollary 1, which appears directly relevant to the paper's motivation, is crucial.

Author response:
We really apologize for the lack of supporting proofs, and **have added detailed proofs for Theorem 1, Corollary 1, and Lemma 2 in Appendix A.**

Reviewer comment (Writing):
The fourth paragraph in Section 1 references "the first question" and "the second question", but it is unclear where these questions are presented in the paper.

Author response:
The reviewer has raised an important point, and **we have carefully revised the presentation here in Section 1** to improve the clarity.

-	First, we propose universal kernel-based balancing methods to balance functions on the reproducing kernel Hilbert space (RKHS), which **adaptively selects the functions that most need to be balanced to reduce the estimation biases of the IPS and DR estimators**. 

-	Moreover, we propose **a novel entropy-based optimization problem to effectively balance the selected functions.**

-	Finally, we perform **theoretical analysis** showing that the learned kernel balanced propensities can effectively **reduce the generalization bound**.

Reviewer comment (Writing):
Given that kernel functions are not frequently used for debiasing Recommender Systems, it is recommended that the authors provide more specific definitions and formulations regarding the kernel functions in Section 4.3 to enhance clarity.

Author response:
Thank you for your kind advice. As suggested by the reviewer, we **add a formal definition of kernel function** at the beginning of Section 4.3, as well as **the explicit form of Gaussian kernel and exponential kernel as examples.**

Reviewer comment (Writing):
There are some typos in this paper. 
- Section 1: "it is the first paper provides ...", "Ours theoretical analysis shows...". 
- Section 4.4: "which Random chooses...".

Author response:
We thank the reviewer for pointing out this issue, and **we have carefully polished the whole manuscript in our revised version to avoid typos and improve readability.**

Reviewer comment (Experiments):
The performance on "Coat" reported in Table 1 is significantly lower than that in [1]. It would be helpful to understand whether this variance is attributed to different experimental settings or other factors.

Author response:
We thank the reviewer for pointing out this issue. We suppose the main reason is that **there exist several different experimental settings and code-base** in the field of debiased recommendation. Specifically, **we find the results in [1] are similar to the results in [2-4],** which use the following code-base: https://github.com/DongHande/AutoDebias. Instead, **we follow the experiment setting and code-base used in [5-7]**: https://github.com/RyanWangZf/CVIB-Rec, which leads to **similar results** compared to the published papers [5-7].

Reviewer comment (Writing):
The abbreviation "CB" for causal balancing is indeed similar to "Covariate Balancing" and may cause confusion. A more distinct abbreviation should be considered to prevent any potential misunderstandings.

Author response:
Thanks for the comment and we fully agree with the reviewer. In our revised version, **we replaced causal balancing (CB) with kernel balancing (KB) and revised all figures and tables**, we think "KB" is a more specific and relevant abbreviation.