Paper: Cross-domain Recommendation from Implicit Feedback (ICLR-2024)

Reviewer comment (Theory):
It is important to balance denoising and preservation of unique user behaviors. This weakness pertains to the treatment of implicit feedback in the denoising strategy and the Dynamic Noise Reduction (DNR) process within the NARF framework. The concern is that these processes might overly standardize user behavior, potentially leading to the loss of unique and valuable user behavior patterns inherent in implicit feedback. In the NARF framework, implicit feedback is subjected to a denoising strategy to mitigate the impact of noisy interactions. Implicit feedback, by nature, is derived from user actions such as clicks, views, or purchases, and it encapsulates a wide array of user behavior patterns. These patterns are crucial as they provide unique insights into user preferences and behavior. However, the denoising strategy, as described in the paper, might not sufficiently differentiate between noise and genuine user behavior patterns that are less common or more nuanced. This is particularly evident in the sections of the paper where the authors discuss the implementation of the DNR process. The DNR process aims to dynamically reduce noise by focusing on reliable user-item interactions. However, the criteria for determining reliability are primarily based on loss values computed during training. This approach raises the concern that unique user behavior patterns, which might initially result in higher loss values due to their rarity or complexity, could be mistakenly treated as noise and filtered out. This is especially problematic if these unique patterns are crucial for understanding specific user preferences or behavior in niche domains. Furthermore, the paper does not provide a detailed discussion on how the NARF framework, and specifically the DNR process, handles the trade-off between denoising and preserving the richness of implicit feedback. The lack of this discussion leaves readers questioning whether the framework can adequately capture the diversity of user behavior inherent in implicit feedback, particularly in scenarios with diverse and biased datasets. In summary, while the NARF framework aims to improve recommendation performance through denoising strategies, there is a need for a more explicit discussion and concrete examples in the paper to address concerns regarding the potential loss of inherent user behavior patterns in the implicit preferences. This is crucial for ensuring that the benefits of denoising do not come at the expense of overlooking valuable insights that implicit feedback can provide.

Author response:
**Response 1:** It's important to note that in practical scenarios, we often lack precise information about the amount of noise present in a dataset. Our approach, therefore, relies on using the validation set to determine the appropriate level of denoising. This process involves deciding how many reliable user patterns should be retained during training to ensure the integrity of the data.

Based on this approach, we then evaluate the performance of our method on the test set. In our experiments, we observed a significant improvement when comparing our denosing method against approaches that do not incorporate such a mechanism. This marked enhancement in performance, as evidenced by our empirical results, strongly indicates the effectiveness of our method.

By focusing on retaining useful and reliable user patterns while eliminating noise, our approach demonstrates its capability to enhance the quality and reliability of the data used in the training process, leading to more accurate and robust outcomes.

**Response 2:** We appreciate your insightful concern regarding the balance between denoising and preserving unique user behaviors in implicit feedback scenarios. Let us clarify our approach:

Our framework's primary objective is to enhance recommendation performance by addressing the inherent noisiness of implicit datasets. This noise often leads to suboptimal recommendation results. Through our experiments, we have demonstrated that effectively removing this noise significantly boosts performance.

Addressing your query about maintaining this delicate balance, our strategy involves utilizing the validation set to fine-tune the denoising factor. This factor is crucial as it dictates the equilibrium between eliminating noise and retaining the richness and diversity of implicit feedback. The optimal denoising factor identified through the validation set is expected to yield favorable results on the test set. Our experimental findings affirm the validity of this approach.

However, upon reflection and considering your suggestion, I realize that our paper could benefit from a more detailed discussion on the workings of the NARF framework, particularly in the context of this balance. We acknowledge this as a way to improve the readability of our paper.

Reviewer comment (Experiments):
The cross-domain datasets used in the experiments are all sourced from Amazon, covering domains like movies, music, and books. These domains are relatively similar, and the use of more diverse and biased datasets could provide a more comprehensive evaluation of the proposed method.

Author response:
**Response 3:** **There might be a misunderstanding.** In our original paper, we not only utilized the Amazon synthetic dataset but also collected a substantial real-world dataset, referred to as Task 3. This dataset is particularly important as it reflects the complexities and challenges encountered in real-world scenarios, especially where the proportion of noisy data is difficult to ascertain. The motivation for collecting this dataset stems from a critical observation: existing Cross-Domain Recommendation from Implicit Feedback (CDRIF) algorithms predominantly rely on synthetic data. We identified a potential bias in results derived solely from such data. To address this, we gathered real-world data and applied our method to establish a more realistic and robust benchmark. We will later open source this dataset for scientific research purposes to encourage more work in this field.

Reviewer comment (Writing):
It is imperative to maintain a consistent use of terminology when expressing the substitution of DNR with CTD in the ablation study, ensuring clarity and precision in communication. For instance, in Table 4, the term 'CTD' is utilized, whereas in the main model description sections, such as in Figure 3, the term 'DNR' is employed. This discrepancy necessitates attention to ensure consistency and avoid potential confusion for readers.

Author response:
**Response 4:** We appreciate your pointing out the inconsistency in terminology between 'CTD' and 'DNR'. We will ensure that the terms are used consistently to avoid confusion.

Reviewer comment (Experiments):
The evaluation is very weak. There are only two baselines, EMCDR (Man et al., 2017) and PTUPCDR (Zhu et al., 2022). For cross-domain recommendation (CDR) methods (applicable for both explicit and implicit feedback), there are so many advanced learning techniques, just name a few bellows: graph neural networks based CDR, transfer learning based CDR, attention mechanism based CDR, adversarial learning based CDR.

Author response:
We understand your concern regarding the baseline methods selection. In our study, we focused on methods most relevant to our specific research question. However, we acknowledge the omission of certain important works in implicit feedback recommendation systems. We agree that including these methods could provide a more comprehensive understanding. To address this, we are actively expanding our experimental scope to include these important methods. We are committed to ensuring a thorough and robust evaluation of our framework.

**We will promptly report the updated results once these additional experiments are completed**.

Reviewer comment (Novelty):
Many related works are ignored. Also, see the above weak W2. In detail, transfer learning is a main research thread to address the cross-domain recommendation (for both explicit and implicit feedback). For example, the CoNet method (Collaborative Cross Networks for Cross-Domain Recommendation) can address the cross-domain recommendation from implicit feedback as investigated in this ICLR submission. In its task setting on the Cheetah Mobile dataset, it recommends apps (the target domain) by exploiting knowledge from news reading history (the source domain). Obviously, the reading logs are implicit feedback and the installations of apps are also implicit feedback. As a result, the claim in the introduction “to the best of our knowledge, prior to our study, no existing CDR research has offered algorithms to handle implicit feedback scenarios” does not hold true.

Author response:
We appreciate your pointing out the omission of relevant works, such as the CoNet method. We will revise these and other relevant studies, ensuring a comprehensive overview of the state-of-the-art in CDR, particularly focusing on implicit feedback scenarios.

Reviewer comment (Experiments):
The newly introduced dataset is not detailed. I read the Appendix A.2 for the dataset description on PubMed and DBLP. I checked the web pages on PubMed and DBLP, but I still do not know: How to align the authors on these two domains (name disambiguation)? Where to get the topic identification of an author? Why use the topic as the items and what is the reason to recommend a topic to an author? Why not use the {venue, paper/references, potential co-author et al} as the items instead? Is any data sample to be shown? By the way, will this dataset be released to use for research purposes? What is the cleaning, filtering, and preprocessing detail for this newly introduced dataset?

Author response:
Thank you for highlighting the need for more detailed information about our newly introduced dataset. **This is a new dataset collected by ourselves.** The DBLP dataset and PubMed dataset are both from the Microsoft Academic Graph (MAG), which can be downloaded from this link: https://www.aminer.cn/oag-2-1. We selected this dataset as it provides multiple data source identifiers (DBLP and PubMed IDs, etc.), disambiguated authors, institutions and research topics (with unique identifiers). That explains how we aligned the two datasets and obtained the pre-processed author and topic information. The research topics are extracted via a hierarchical concept extraction tagging algorithm developed by the Microsoft research team (Shen, Z., Ma, H., & Wang, K. (2018). A web-scale system for scientific knowledge exploration. arXiv preprint arXiv:1805.12216.). we assume that an author's topic collection is the union of all topics from papers he/she has authored. The specific procedures for extracting this subset are: 
1. Download the MAG dataset, 
2. Download the V12 DBLP dataset (https://www.aminer.org/citation) and retrieve the paper IDs (which are already aligned with the MAG dataset).
3. Fetch the records with PubMed ID information from the MAG dataset as the PubMed dataset (source domain), then fetch the records from the MAG dataset as the DBLP dataset (target domain). 

In our paper, we primarily recommend using the topic as the subject for our experimental analysis. This choice was made after careful consideration, as we believe that **topic** offers a more significant and valuable context for demonstrating the effectiveness of our algorithm. While venue and co-author are also viable options, we opted for the topic based on its potential to yield more impactful and relevant insights in our study. We also plan to release the dataset for research purposes, subject to obtaining the necessary permissions.

Reviewer comment (Theory):
In Eq. (14), for the sigma summation, $m$ should be $n$ which denotes the number of items instead of the number of users.

Author response:
You are correct; this is a typographical error, and we will correct it.

Reviewer comment (Writing):
Below Eq. (14), what is $f_j$? And what is the relation between $f_j$ and $p_j$

Author response:
You are correct; this is a typographical error, and we will correct it.

Reviewer comment (Writing):
Checking the syntax issue for this sentence, “the evaluation metrics are recall@k and ndcg@k are computed following the all-ranking protocol”, which has two “are”s.

Author response:
We appreciate your pointing out the syntax error and will rectify it.

Reviewer comment (Experiments):
In Tables 2 and 6, how about the noise level equals zero?

Author response:
We will include results for the case where the noise level equals zero in the updated paper and report the results here when done.

Reviewer comment (Experiments):
There are so many implicit feedback benchmarks like Criteo (CTR clicks) and Microsoft MIND (news clicking). Why not ignore evaluations on such large, real-world datasets? By the way, the evaluated Amazon review dataset is NOT an implicit feedback dataset.

Author response:
We **know** that there are so many implicit feedback benchmarks, but it is not easy to find implicit feedback datasets for cross-domain scenarios. In fact, our paper has introduced a dataset we collected ourselves as the experiment results shown in Table 2.

Reviewer comment (Novelty):
Technical contribution is limited. Both implicit feedback recommendation and cross-domain recommendation are well studied topics, and the proposed denoising framework is also stacked by well studied methods, e.g., negative sampling from learning to rank, AD/CTD. Moreover, the authors claim this is the first work on cross-domain recommendation with implicit feedback, however, it's not convincing. For example, "Cross-domain Recommendation Without Sharing User Relevant Data" and "User-specific Adaptive Fine-tuning for Cross-domain Recommendations" are both proposed to work on the implicit feedback data.

Author response:
We appreciate your concerns about the novelty of our work. However, we assert that our contribution is notably distinct from the studies you have mentioned. The referenced research primarily focuses on Cross-Domain Recommender (CDR) algorithm design and primarily **involves experiments with implicit datasets**. However, their approach, which typically incorporates random negative sampling to address implicit feedback scenarios, fails to fully address **the unique challenges inherent in implicit feedback**. In our view, this area presents significant opportunities for advancement.

In contrast, our work specifically addresses Cross-Domain Recommendation from Implicit Feedback (CDRIF). We introduce an innovative framework tailored to this particular issue. This framework, designed to be compatible with existing techniques, has demonstrated its effectiveness through rigorous experiments. Its model-agnostic and plug-and-play nature facilitates easy implementation and integration into current systems, a significant advantage over existing methods.

Addressing the critique of our technical contribution, it is important to note the distinctiveness of our framework. Developing a versatile framework compatible with a range of existing techniques, especially for CDRIF, is a non-trivial task. Our work pioneers in this area, marking a significant technical advancement. **Therefore, dismissing our technical contribution in designing this innovative framework from scratch would be unjust**.

Our framework's uniqueness is further highlighted by its ability to address two critical challenges in recommendation systems:

1. **Cold-Start Problem in Implicit-Feedback Systems**: Traditional implicit-feedback systems struggle with this issue. Our framework, as evidenced in our experiments, effectively leverages knowledge transfer from a source domain to mitigate the cold-start problem in the target domain.
    
2. **Handling Implicit Feedback in CDR Methods**: Current CDR methods inadequately address implicit feedback, often resorting to a combination of traditional CDR methods and negative sampling. Our framework, however, goes beyond this by systematically investigating cross-domain recommendations under implicit feedback scenarios and proposing a comprehensive solution rather than just methodological adjustments.

In summary, our paper is the **first** to thoroughly explore CDR under implicit feedback, proposing a robust and adaptable framework for CDRIF. This framework not only outperforms existing baselines with **a large margin** but also offers ease of implementation and deployment in practical settings. **Therefore, again, dismissing our contribution to designing a novel, general framework from scratch would overlook the substantial advancements our work presents in this field**.

Reviewer comment (Experiments):
Baseline methods are not well selected. There are numbers of sampling related research in learning to rank and applied to implicit feedback recommendation methods, however, few are included in comparison or discussions, e.g., Learning Recommenders for Implicit Feedback with Importance Resampling, etc.

Author response:
We understand your concern regarding the baseline methods selection. In our study, we focused on methods most relevant to our specific research question. However, we acknowledge the omission of certain important works in implicit feedback recommendation systems. We agree that including these methods could provide a more comprehensive understanding. To address this, we are actively expanding our experimental scope to include these important methods. We are committed to ensuring a thorough and robust evaluation of our framework.

**We will promptly report the updated results once these additional experiments are completed**.

Reviewer comment (Presentation):
The presentation of this work needs improvement. In particular, the notations used in this paper are hard to follow and can be simplified. Moreover, the framework section is hard to follow. It's unclear why the calibration factor can act as a denoising factor. What's the difference between this proposed idea and other active learning ideas?

Author response:
We appreciate your feedback on the clarity of our presentation. We plan to revise the paper to simplify notations and provide a clearer explanation of our framework.

Reviewer comment (Novelty):
What's the main technical contribution for this work if this is not the first one tackling cross-domain recommendation with implicit feedback?

Author response:
We appreciate your concerns about the novelty of our work. However, we assert that our contribution is notably distinct from the studies you have mentioned. The referenced research primarily focuses on Cross-Domain Recommender (CDR) algorithm design and primarily **involves experiments with implicit datasets**. However, their approach, which typically incorporates random negative sampling to address implicit feedback scenarios, fails to fully address **the unique challenges inherent in implicit feedback**. In our view, this area presents significant opportunities for advancement.

In contrast, our work specifically addresses Cross-Domain Recommendation from Implicit Feedback (CDRIF). We introduce an innovative framework tailored to this particular issue. This framework, designed to be compatible with existing techniques, has demonstrated its effectiveness through rigorous experiments. Its model-agnostic and plug-and-play nature facilitates easy implementation and integration into current systems, a significant advantage over existing methods.

Addressing the critique of our technical contribution, it is important to note the distinctiveness of our framework. Developing a versatile framework compatible with a range of existing techniques, especially for CDRIF, is a non-trivial task. Our work pioneers in this area, marking a significant technical advancement. **Therefore, dismissing our technical contribution in designing this innovative framework from scratch would be unjust**.

Our framework's uniqueness is further highlighted by its ability to address two critical challenges in recommendation systems:

1. **Cold-Start Problem in Implicit-Feedback Systems**: Traditional implicit-feedback systems struggle with this issue. Our framework, as evidenced in our experiments, effectively leverages knowledge transfer from a source domain to mitigate the cold-start problem in the target domain.
    
2. **Handling Implicit Feedback in CDR Methods**: Current CDR methods inadequately address implicit feedback, often resorting to a combination of traditional CDR methods and negative sampling. Our framework, however, goes beyond this by systematically investigating cross-domain recommendations under implicit feedback scenarios and proposing a comprehensive solution rather than just methodological adjustments.

In summary, our paper is the **first** to thoroughly explore CDR under implicit feedback, proposing a robust and adaptable framework for CDRIF. This framework not only outperforms existing baselines with **a large margin** but also offers ease of implementation and deployment in practical settings. **Therefore, again, dismissing our contribution to designing a novel, general framework from scratch would overlook the substantial advancements our work presents in this field**.

Reviewer comment (Theory):
Why the learnable calibration parameter can help denoising? What's the difference between this idea and the active learning idea?

Author response:
**It seems there may be some confusion regarding our calibration parameter.** I would like to clarify that this parameter in our model is not learnable through training. Instead, it is a predefined factor in the NARF model, which has demonstrated its effectiveness in our experiments, particularly in enhancing recommendation accuracy. The primary purpose is to calibrate the training data to better reflect users’ actual preferences. Specifically, it involves determining the calibration factor for each user-item pair, with a higher value indicating closer alignment with the user’s genuine preferences. Based on these factors, we can prioritize training the recommender system with data that carry higher factors. 

Our approach differs from traditional active learning, particularly in how we handle outliers and noise. Traditional active learning is sensitive to outliers and noise [r1]. To mitigate this vulnerability, our framework incorporates adaptive denoising strategies. These strategies are designed to effectively filter out noise and reduce the influence of outliers. By integrating these denoising approaches, we enhance the robustness of our recommendation system, leading to improved performance and more reliable recommendations. This integration of adaptive denoising represents a key advancement over traditional active learning methods, addressing a critical challenge and boosting the overall efficacy of our recommendation framework.

Reviewer comment (Experiments):
What's the performance comparison with state-of-the-art negative sampling method in learning to rank?

Author response:
We appreciate the emphasis on comparing our work with advanced negative sampling methods. While our current study did not encompass this specific analysis, we recognize this as a limitation and will consider incorporating it. 

However, it is important to note that our primary focus is not on negative sampling methods. The prevalent approaches in CDRIF predominantly rely on random negative sampling. Our framework builds upon this foundation. We have found that even when employing the most basic sampling method, such as random negative sampling, CDR algorithms that utilize our framework still manage to achieve commendable results.

This observation underscores the inherent strength and versatility of our framework, suggesting that its effectiveness is not solely dependent on advanced negative sampling techniques. Nevertheless, a comparative analysis with negative sampling methods could provide additional insights, and we plan to explore this in our subsequent work.