Paper: Exploring the Upper Limits of Text-Based Collaborative Filtering Using Large Language Models: Discoveries and Insights (ICLR-2024)

Reviewer comment (Experiments):
Only evaluates two basic recommender architectures. More complex models may behave differently.

Author response:
We sincerely appreciate your suggestion. In our paper, we acknowledged a limitation of our study as an initial exploration where we only focused on two classical architectures (see Section 4.1). This choice was made because we discovered that even in these well-established architectures, many assumptions do not hold. For instance, even when employing the classical architecture, TCF with LLM still lacks transferability, and items do not possess universality. Given these challenges that persist within classical architectures, we argue that discussing additional recommendation models would not make much sense.

Furthermore, the SASRec model, based on the Transformer architecture, remains the state-of-the-art (SOTA) in recommender systems. The Transformer architecture is not only considered SOTA in the recommender systems community but also in the NLP, CV, and broader AI communities. To date, there is no consensus within the recommender systems community or other AI communities on models that clearly surpass the Transformer.

While many subsequent sequential recommendation models in the recommender systems community compare themselves to SASRec and claim to outperform it, a closer examination of recent papers reveals that SASRec consistently remains the SOTA model, albeit slightly inferior to the proposed model in each respective paper. We believe that the recommender systems community currently faces certain challenges in its development, with one of the most significant issues being the lack of a public benchmark for evaluating the SOTA models. The majority of SOTA models are evaluated based on the authors' own data splits, which raises the possibility that many SOTA models may not truly be the best performers. For example, although BERT4Rec claims to outperform SASRec significantly, subsequent studies have found that BERT4Rec is not as effective as initially claimed. Similar findings can be observed in the comparison between NCF and MF, even though NCF is a more complex model than MF.

In fact, even if some models can surpass SASRec, we believe it would be more of an incremental improvement, as the entire AI community has not made a groundbreaking breakthrough after the basic Transformer architecture. Both GPT and ChatGPT are still based on the Transformer architecture, and certain variations of the Transformer would not affect the core findings of our paper. 

The replicability of recommender systems has garnered significant attention in the community in recent years, as many SOTA models have been achieved through unfair means. For example, [2,3,4,5,6,7,8] have highlighted this issue, and a recent study by the authors of GRU4Rec [6] pointed out that the majority of literature with advanced sequential recommendation models did not adhere to the original design when comparing with the basic GRU4Rec, resulting in significantly worse performance compared to the original paper. The pioneers,  Rendle and Koren [2,3],  have also expressed concerns about a significant number of SOTA papers in the recommender systems field over the past decade as they found that once evaluating these SOTA models under fair settings, they were no longer SOTA. Honestly speaking, the reproducibility of research papers in the field of recommender systems have faced catastrophic challenges in  recent  years.

We argue that the observations made on Transformer or SASRec have more general applicability than some SOTA models claimed by the authors. Likewise, the findings for the DSSM model apply to most CTR models as the performance gap between two-tower DSSM and single-tower CTR models in public benchmarks is typically less than 2%, which does not affect our main findings.

[2] Neural collaborative filtering vs. matrix factorization revisited. By Rendle etc. Recsys2020

[3] on the difficulty of evaluating baselines: A study on recommender systems. By Rendle and Koren. 2019

[4]Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation 

Approaches. Recsys2019 Best paper

[5] Reproducibility Analysis of Recommender Systems relying on Visual Features: traps, pitfalls, and countermeasures. Recsys2023

[6] The effect of third party implementations on reproducibility

[7] Our Model Achieves Excellent Performance on MovieLens: What Does it Mean?

[8] Everyone’s a Winner! On Hyperparameter Tuning of Recommendation Models. Recsys2023


We hope that our sincere response can alleviate the concerns you may have.

Reviewer comment (Reproducibility):
Limited hyperparameter tuning details, so optimal configurations are unclear.

Author response:
We're thinking you might have missed a few places. On page 5, we said “**As for hyper-parameters, we first perform a grid search for standard IDCF as a reference, ..., We report details in Appendix B**” Very detailed hyper-parameter tuning are reported in Appendix B.

Reviewer comment (Experiments):
Focuses only on random splits, not temporal evaluation protocols.

Author response:
In page 5, we said the data is split based on temporal orders. See below statement from the original paper: 

“The **latest** user-item interaction was used for evaluation, while the **second-to-last** interaction was used for hyper-parameter searching, and all other interactions were used for training”

Reviewer comment (Experiments):
Transfer learning study limited to simple zero-shot approach. More advanced techniques could be explored.

Author response:
Zero-shot learning is a significant challenge for the transfer learning research, not only in the domain of Recsys but also in CV and NLP. Evaluating the foundation model's zero-shot prediction ability is crucial, as it sheds light on the progress of our community in terms of transferable or foundation recommendation models. We argue that assessing its performance in this standard zero-shot setting (as in our paper) serves as an essential indicator of our community's current advancements. An important example is CLIP, which highlights its performance under the zero-shot setting.

While we acknowledge the presence of other techniques, such as fine-tuning, adapter tuning, and prompt tuning, it is important to highlight that these parameter tuning approaches necessitate additional computational resources and training examples compared to zero-shot prediction. Zero-shot prediction is a more challenging task in comparison and has more practical values. Although fine-tuning, adapter tuning, and prompt tuning have been acknowledged for their ability to achieve improved performance, the primary objective of this paper is to investigate whether TCF with LLM as the item encoder can accomplish a universal "one-for-all" model. Therefore, evaluating its performance under the fundamental zero-shot setting holds significant importance. Zero-shot prediction vs. finetuning (adapter-tuning or prompt-tuning) is just like the relationship of BERT vs. ChatGPT. No doubt, ChatGPT with zero--shot prediction ability is more appealing and has significant more practical values. It took the NLP community 4 years to move from BERT to ChatGPT.

Please note that our TCF paradigm, as discussed in this paper, is not compatible with some advanced techniques, such as instruction tuning or chain of thought. These techniques are not suitable within the scope of the TCF paradigm because our recommendation backbone is not a LLM. These techniques are mainly developed for LLM. These techniques can be utilized in other recommendation paradigms, such as using LLM as the recommendation backbone, as demonstrated in [1], or based on ChatGPT.  However, we did not focus on such settings in this paper, as LLM-based recommendation models have undergone numerous variations in the past 6 months, among which TCF with LLM as item encoder is an classical and important paradigm.

[1] Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)

Reviewer comment (Experiments):
For transfer learning, have you tried multi-task or explicit domain adaptation techniques?

Author response:
Thanks for this great suggestions. Unfortunately, there is currently no large-scale public dataset available that contains both rich textual data and multiple labels for supporting multi-task learning. While we agree that multi-task learning could potentially lead to more universal representations, achieving zero-shot performance, as discussed in our paper, would require more fundamental advancements in either models or evaluation methodologies.

Regarding explicit domain adaptation, do you mean fine-tuning or retraining the pre-trained model on the target dataset, this approach would lead to comparable or better performance compared to IDRec (see Table 3). However, this is widely recognized as fine-tuning on the target dataset will usually perform no worse than the its training-from-scratch version. However, this is not our focus as performing fine-tuning or other parameter-tuning methods (e.g. adapter tuning or prompt tuning) still requires additional training examples and significant computational resources. The goal of our paper is to explore the possibility of achieving a universal model that can be directly used for downstream tasks without further tuning on new data, similar to. ChatGPT or CLIP etc. This challenge is a crucial one in the fields of NLP, computer vision, and recommender systems. If, in the future, we can develop a universal model for recommender systems, it would significantly reduce the efforts required for individual task-specific recommendation models.

We sincerely hope that our genuine response will meet your approval. If you find our response to address your main concerns, we kindly request that you consider revising the score for our paper. Completing such a paper has costed us $200,000 for over a year. We hope to get some encouragement and recognition from the community.

Reviewer comment (Theory):
It is a very empirical result and very limited effort is made on the theoretical analysis front.

Author response:
> It is a very empirical result and very limited effort is made on the theoretical analysis front.

Thank you for raising this question. We acknowledge that this work lacks some theoretical analysis, as we find that some issues may not be well theoretically analyzed.  Instead, we have conducted extensive empirical studies, which are resource-intensive. For example, in Section 6, we conducted fine-tuning of the LLM with a model size of tens of billions of parameters. This process necessitated the use of dozens of A100 80G GPUs and significant time investments for each experiment. As a result, the compute costs for these experiments exceeded $200,000. We realize such expensive experiments are not commonly seen in the recommender system field. We find that  top-tier conferences like ICLR, NIPS and CVPR have published quite a few pure empirical research papers, such as "chain of thought"[1] and literature [2,3,4,5,6,7]. We fully agree with you with some theoretical analysis will make the paper stronger.

1) Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS2022

2) Rethinking ImageNet Pre-training. By Kaiming He etc. CVPR 2019

3) Bag of Tricks for Image Classification with Convolutional Neural Networks. CVPR2019

4) An Empirical Study of Training Self-Supervised Vision Transformers. CVPR2020

5) Emergent Abilities of Large Language Models. Transactions on Machine Learning Research2022

6) Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal on Machine Learning Research2019

7) AN EMPIRICAL STUDY OF EXAMPLE FORGETTING DURING DEEP NEURAL NETWORK LEARNING. ICLR2019

We genuinely appreciate your valuable feedback and greatly appreciate your support. We humbly request your further support when discussing this paper with other reviewers and the AC.

Reviewer comment (Experiments):
The experimental datasets use the title as the item feature, and there may be more information that can be utilized but has not been used, leading to the potential of the tested method being underestimated.

Author response:
This is a very good question, and indeed, as you suggested, using more information could potentially improve the performance of TCF. For example, in the case of news recommendations, we could utilize both the title and the full text content. We are now conducting related experiments and will report these results later.


We did not do this in the original submission primarily due to the computational challenges involved in using full-length texts. Many experiments with full-length texts would require ultra-high computational resources, such as the fine-tuning experiments and experiments on LLMs with billions of parameters.  

We have just completed the experiment where we have taken into account both the abstract and the title of a new article. Please find the updated results presented below.

Reviewer comment (Novelty):
The paper is mainly experimental and does not propose new solutions.

Author response:
We greatly appreciate your comment. As you mentioned, this paper does not aim to propose new deep learning techniques. Instead, its objective is to shed light on the current advancements within the recommender systems community, specifically focusing on LLM4Rec or TCF‘s scaling effect, and to explore the possibility of establishing a foundation recommendation model akin to BERT or ChatGPT in the field of natural language processing (NLP).

We recognize that recommender systems still have a considerable journey ahead to catch up with NLP and computer vision (CV) in terms of large-scale foundation models. Presently, the recommender systems community lacks readily available one-for-all  pre-trained models (e.g. in HuggingFace)  that can be applied to various downstream tasks, similar to NLP's BERT. This gap signifies that developing a general-purpose model in the realm of recommender systems might pose more significant challenges compared to NLP and CV.


Second this paper demonstrates some surprising findings, for example, even with a 175B LLM, the results of TCF are only comparable to those IDRec by using simple itemID embedding features. This finding is unexpected because numerous papers claim to have achieved state-of-the-art (SOTA) performance with their multimodal or text-based recommendation models. However, upon reviewing the related literature, we discovered that these papers often rely on unfair comparison settings, such as using different backbones, loss functions, and samplers for comparison. Such comparisons make it challenging to attribute improvements solely to LLMs or other components.

We feel that empirical studies are vital for the recommender systems community. We have noticed that ICLR receives a significant number of NLP and CV papers each year that focus on empirical studies. By contrast, there are relatively few empirical studies in the field of recommender systems published at ICLR.  Additionally, while this paper is an empirical study, it involves extensive and expensive experiments. For example, fine-tuning the billion-parameter LLM requires dozens of A100 GPUs, resulting in compute costs exceeding $200,000. We hope that our extensive experiments will provide the community with some useful insights. In fact, the fields of NLP and CV have seen numerous empirical study papers, and it would be unfortunate if the recommender systems community hindered such research. Therefore, we sincerely hope to receive your continued support.

We listed several high-impact papers which are purely empirical and published in top-tier venues e.g. CVPR and ICLR.

1) Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS2022

2) Rethinking ImageNet Pre-training. By Kaiming He etc. CVPR 2019

3) Neural Collaborative Filtering vs. Matrix Factorization Revisited. Recsys2020

4) An Empirical Study of Training Self-Supervised Vision Transformers. CVPR2020

5) Emergent Abilities of Large Language Models. Transactions on Machine Learning Research

6) Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal on Machine Learning Research

7) AN EMPIRICAL STUDY OF EXAMPLE FORGETTING DURING DEEP NEURAL NETWORK LEARNING. ICLR2019

8) Bag of Tricks for Image Classification with Convolutional Neural Networks. CVPR2019

Finally, please allow us to express our gratitude for your support and great advice.

Reviewer comment (Novelty):
This paper only show the empirical experimental results and present relevant discussions. It does not introduce some novel deep learning techniques.

Author response:
We appreciate your concern. We have realized that there are quite a few empirical study papers in every year‘s ICLR proceedings, as well as a significant number of related paper in CVPR and ACL, e.g.[1,2,3,4,5,13,14].  Many of them have very high impact on their research communities. Empirical study papers, unlike typical algorithm-focused papers, usually do not necessarily require the introduction of new deep learning techniques. However, they often delve deeper into important or common research paradigms or deep learning models, providing valuable insights and guidance to the community. These papers can stimulate further reflection and inspire future research. We believe that high-impact research should initiate community discussions and reflections, highlighting existing shortcomings and challenges. Papers, such as empirical studies, rethinking studies, datasets, and benchmarks, often focus on studying existing deep learning techniques rather than introducing new ones. Another example is that the call for paper part also mentioned one research direction is "visualization or interpretation of learned representations". Our paper can be seen as an interpretation of learned representation of LLM for the recommender system problem.


**We feel that introducing new deep learning techniques is not the sole criterion for a good research paper in the field of AI. Instead, we believe that the most significant papers are those that uncover important yet unknown facts, inspire and motivate the community, and stimulate new discussions.  If these important findings have not been revealed before, they should be regarded as novel.  Identifying problems can sometimes be as important as solving them. In the recommender systems community, we believe it is also important to encourage diverse research.**

Reviewer comment (Reproducibility):
Some details of the experimental settings are not clear. In the experiments, the authors also fine-tune the LLMs with the data in recommendation domains. However, it is not clear what kind of data are used in LLM fine-tuning, and how to fine-tune the LLMs.

Author response:
In section 6, we conduct experiments involving fine-tuning of LLMs. We utilize the training dataset mentioned in Table1  to perform the fine-tuning process. The key distinction in these experiments compared to other frozen LLM experiments is that the LLMs, when used as item encoders, undergo optimization during the training process, whereas frozen LLMs maintain their LLM parameters unchanged. Also see description on page 7 around footnote 8 and figure 3.

Reviewer comment (Experiments):
For sequential recommendation, there are some SOTA models that have more complex structure than SASRec and DSSM. Why not using these SOTA sequential recommendation models with more complex structures as backbone models to study the TCF?

Author response:
DSSM and SASRec (with Transformer as the backbone) are regarded as among the most representative models in the field of recommender system. DSSM can represent the performance of the majority of CTR models, as the main difference lies in the single-tower versus dual-tower architecture, which has minimal impact on performance and does not affect the core findings of our paper. The difference between single towers and twin towers usually fluctuates within 2%.

While some papers claim that their new models outperform SASRec, a closer look at their future work reveals that SASRec consistently remains the top baseline, usually only below the model proposed by the authors. It is important to note that some SOTA improvements are not necessarily due to the claimed advancements proposed by the authors. Often, they arise from unfair comparisons, such as using different loss functions, samplers, or suboptimal hyperparameters. We have extensively evaluated numerous SOTA models proposed that claimed better than SASRec, but unfortunately, none of them (e.g., BERT4Rec, P5[10], SSE-PT[11], etc) have really surpassed SASRec's performance by properly tuning the hyper-parameters.  There are too many SOTA papers that do not clearly show their evaluation details (we realized that many papers compare baselines at randomly chosen embedding sizes (e.g. 100)), which is not convincing as a SOTA claim should be based on carefully tuned optimal embeddings and hyper-parameters. Some improvements will disappear if we enlarge the embedding size or model size of baseline models. Consequently, these claimed SOTAs may only represent the authors' own SOTA rather than being widely accepted as community-wide SOTAs.)




In recent years, the reproducibility of research papers in the field of recommender systems have faced catastrophic challenges. Numerous subsequent papers have pointed out that the credibility of SOTA (State-of-the-Art) models in recommender systems is highly questionable, see [6,7,8,9]. The paper [9] highlights that there are hardly any correct third-party implementations of GRU4Rec (all of these well-known implementation is much weaker than the authors' official implementation), and Rendle and Koren [6,7] has also expressed serious doubts about a significant number of so-called SOTA papers in the recommender systems field over the past decade. Therefore, we believe that studying the problem using the most widely recognized baselines is more convincing and representative than relying solely on self-proclaimed SOTA models.

In addition, even assume there are some more advanced models, we believe they are only incremental improvements because so far there are no groundbreaking networks after Transformer, neither in Recys nor in NLP and CV. These new variants or incremental improvement of DSSM and Transformer intuitively will not affect the core findings in this paper. We agree with our reviewers, evaluating more models will be interesting, but it is very difficult to present too many results and discussions due to a limited 9 page space.  We argue that a good paper is the one inspires more research but not the one solving all problems or considering all aspects. In addition, we have shown some unexpected or surprising results on the two classical models and if we cannot solve them on the standard models, it seems not very meaningful to study many other variants at this stage.  To sum up, Transformer or SASRec is still a SOTA model in terms of network architecture in the recsys community and the broader AI community.  These incremental improvements or variant models will not have a significant impact on our findings. In fact, many empirical study work in NLP and CV are also evaluated on the Transformer architecture as so far Transformer still represents the classical and SOTA network backbone in these fields (as well as in Recsys).

Reviewer comment (Experiments):
According to my understanding, the TCF methods studied in this paper use the LLM as the item encoder and use traditional sequential recommendation models to model users' behaviours. Is it possible to directly use LLMs to model the user behaviours?

Author response:
This is a very good question. This paper we focus on LLM as item encoder, but we also evaluate using LLM as the backbone to model user behavior as reported in Appendix D figure 7. Unfortunately, we did not find using LLM as user backbone shows satisfied results. This architecture is very very costly in terms of compute and memory compared to using LLM as item encoder. Assume that we have 100 items, and each item has 100 words, the sequence is up to 10, 000, very challenging to train such big models and perform online inference.


We believe that you are an open-minded and receptive reviewer, and we are committed to addressing your questions to the best of our abilities. We kindly request your support if we can resolve your main concern. Although our work is an empirical study, we have made highly novel discoveries that are  important to the community. To conduct the relevant experiments, we have invested nearly $200,000, which is uncommon for research papers in the recommender systems community. We hope to receive the support of the reviewers. **The fields of NLP and CV have seen numerous empirical study papers, and it would be unfortunate for the recommender systems community to stifle such research. In fact, in recent years, particularly during the deep learning era, the recommender systems community has lagged behind NLP and CV communities. Our community relies on you!**

Thank you once again for your diligent review of our paper. We genuinely hope that our sincere response has met your approval. If you find our response satisfactory in addressing your main concerns, we kindly request that you consider raising the score for our paper. Completing this paper has required a significant investment of $200,000 over the span of a year. We earnestly hope to receive encouragement and recognition from the community.