Paper: Factual and Personalized Recommendation Language Modeling with Reinforcement Learning (ICLR-2024)

Reviewer comment (Evaluation):
The experiments conducted primarily assess the model concerning the targeted reward functions, showing that incorporating RL improves performance on these specific metrics—a finding that is somewhat predictable. It would be advisable to include human evaluations when assessing baseline methods to more convincingly demonstrate practical effectiveness.

Author response:
__> advisable to include human evaluations…__

Please check our common response.

Reviewer comment (Evaluation):
The paper's technical contribution seems limited and under-assessed. The crux of the proposed method's novelty resides in the design of LLM-based reward functions. However, the validation of these reward functions is lacking, and no clear insights are given on how these rewards enhance recommendations. This omission leaves the paper's contributions indistinct.

Author response:
__> insights are given on how these rewards enhance recommendations__

Thank you for your insightful question. We would like to clarify that this work specifically addresses the generation of factual, compelling, and personalized endorsement statements (or “pitches”) of a specific item. The broader question of how these endorsement texts impact real users’ engagement and how a conversational recommender should continue the interaction is left for future research.

Reviewer comment (Experiments):
The experimental validations are not comprehensive. The reliance on a single dataset, along with a comparison with only three baselines—two of which are merely variants, and the other, the foundational model also used for data generation. There is a need for broader evaluations against additional baselines, datasets, and models to affirm the method's effectiveness.

Author response:
__> The reliance on a single dataset...__

Please check our common response.

Reviewer comment (Evaluation):
There is no human evaluation on comparison among P4LM, P4LM-S, SFT, SFT-Text, PaLM-L. How do we know that P4LM is actually better than others in terms of real human feedbacks? Also, PaLM-L’s samples are not provided in appendix.

Author response:
We acknowledge the importance of human evaluation in assessing the effectiveness of P4LM compared to other models. To this end, we are conducting comprehensive human evaluations comparing P4LM, P4LM-S, SFT, SFT-Text, and PaLM-L. These evaluations will provide crucial insights into real human feedback on our model's performance. We will update the paper with these results as soon as they are available. For more details on this process, please refer to our common response.

Reviewer comment (Experiments):
The authors only experiment on one dataset. I understand the complexity of the whole procedure, but it this paper would be much stronger if the proposed method could be validated another dataset.

Author response:
We agree with your assessment regarding the need for validating our method on more than one dataset. To address this, we are currently training P4LM on the Amazon product dataset, which will significantly broaden our evaluation scope. The results from this expanded dataset will be included in our paper as soon as they are available, strengthening our findings.

Reviewer comment (Experiments):
I didn't search very carefully, but the author didn't compare the proposed method with any baselines from other papers. Or if the problem is completely new (I am not sure), then this would not be an issue.

Author response:
To the best of our knowledge, our work is the first to tackle the specific problem of generating endorsement texts with defined characteristics in a recommendation setting. Therefore, direct comparisons with existing baselines in other papers may not be straightforward. However, we are open to including additional relevant baselines if suggested.

Reviewer comment (Experiments):
What recommender system is used for extracting the embeddings? How does recomender system performance affect this task?

Author response:
In our study, we employed matrix factorization-based collaborative filtering to generate user-and-item behavioral embeddings whose dot product predicts the rating of the recommendation. While our problem assumes the embedding vectors are provided as data input and P4LM is general enough to take different forms of embedding vectors as input, we acknowledge that more informative latent embeddings from a more advanced recommender system could potentially enhance the quality of generated endorsement texts. Improving the quality of the behavioral embedding space is left for future work.

Reviewer comment (Novelty):
The paper does not explain why existing methods are insufficient or what are the specific challenges and opportunities in this domain.

Author response:
To the best of our knowledge, we are the first to address the problem of generating personalized recommendation endorsement texts, which has significant implications as more online recommenders adopt LLMs to interact with users. The key challenges we address include the absence of datasets for personalized endorsements and the standardized evaluation metrics to quantify the quality of endorsements. Our contributions are: identifying key characteristics that contribute to good endorsements, demonstrating quantification methods for these characteristics, using these metrics to evaluate and RL-finetune LLMs, and substantiating our findings with human evaluations.

Reviewer comment (Novelty):
The paper provides a comprehensive literature review of related work, especially on conversational recommender systems, language models, and reinforcement learning. But it only cites those papers without discussing their strengths and limitations or comparing them with the proposed approach.

Author response:
We acknowledge that the related work section could be more comprehensive in comparing our approach with existing literature. We will revise this section to include qualitative comparisons and better position our work in the context of existing research.

Reviewer comment (Theory):
Paper does not discuss the assumptions and limitations of the approach.

Author response:
We will include a discussion of the assumptions (behavioral embeddings of user representations in recommender systems are given) and limitations (to specific items endorsements) of our approach in the revised version of the paper.

Reviewer comment (Reproducibility):
The paper does not describe the implementation details and hyperparameters of the proposed approach, such as the size of the models, and the reinforcement learning algorithms. For example: "where η1, η2, η3, η4 ≥ 0 are importance weights for the component rewards, and are treated as hyper-parameter"

Author response:
We will update Appendix B to provide detailed information about the hyperparameters. As for the reinforcement learning algorithm used in our study, we have discussed it in length in the Appendix. We kindly ask the reviewer to check Appendix D.

Reviewer comment (Evaluation):
The evaluation is not rigorous for an applied paper. One dataset is not enough to draw conclusions.

Author response:
We agree that evaluating our approach on a single dataset is limiting. As mentioned in our common response, we are extending our experiments to the Amazon product dataset.

Reviewer comment (Evaluation):
The paper does not present any qualitative analysis or examples of the generated recommendation texts by the proposed approach. It only shows quantitative scores based on model-based metrics, which may not reflect the true quality and diversity of the texts.

Author response:
Generated texts and their qualitative analyses are included in Appendix A of our paper. Additionally, Table 3 presents human evaluation results for the model trained with a single reward. We are conducting further human evaluations to compare P4LM against baselines more rigorously.

Reviewer comment (Experiments):
What are the advantages and disadvantages of using adapter layers to augment the language model?

Author response:
We appreciate your question about the use of adapter layers. These layers are a practical choice that allows us to avoid the introduction of excessive trainable parameters. With the adapter layers, we can incorporate exogenous information such as behavioral embeddings from RS. A disadvantage is that it is less intuitive how one should train the new parameters together with the standard Transformer parameters. We have discussed this in Section 5; please check the paragraph titled “warm-start training for adapter-augmented LMs”.

Reviewer comment (Experiments):
How does the P4LM model deal with cold-start problems, where there is not enough user or item information available?

Author response:
P4LM can utilize available user or item information to generate relevant endorsement texts. For new items, as long as their descriptions are provided (as embeddings or as texts), P4LM can create appealing endorsements. For new users, as long as the behavioral embedding is given, P4LM can still generalize to produce personalized endorsements, otherwise the model focuses less on personalization.

Reviewer comment (Experiments):
How does the P4LM model compare with other conversational recommender systems that use different language models or architectures?

Author response:
Our work does not address the item recommendation aspect of conversational recommender systems. Instead, we focus on enhancing LLMs for generating endorsement texts of particular items. Our primary comparison is with PaLM2 baselines to assess improvements in Precision, Appeal, Personalization, and Preference Relevance.

Reviewer comment (Evaluation):
Can authors talk more about trade-offs between different reward models, such as precision, appeal, personalization, and preference relevance?

Author response:
Your question about the trade-offs between different reward models is pertinent. The full scope of these trade-offs can only be understood through extensive human evaluation studies. Future research could explore how to balance these weights to achieve desired endorsement outcomes with varying properties.

Reviewer comment (Evaluation):
Although the paper incorporates a plethora of metrics, the efficacy of these indicators in truly measuring the corresponding performance needs to be substantiated (see Questions for more details).

Author response:
We acknowledge that our evaluation can be further substantiated, and to address this, we are expanding our experiments to include the Amazon product dataset and conducting comprehensive human evaluations.

Reviewer comment (Experiments):
The baseline used by the author is too concise. It is a consensus in the LLM field that models that have undergone reinforcement learning have better rewards than SFT and pre-trained models. More recommendation-related baselines should be mentioned.

Author response:
Our work uniquely addresses the generation of factual and personalized recommendation endorsement texts, which is distinct from the traditional item selection focus in recommendation literature. Our primary objective is novel, which is to enhance the quality of recommendation endorsements of specific items, not to recommend items to users. We are open to including comparisons with any relevant baselines that you suggest.

Reviewer comment (Experiments):
The article only collected data and trained on a closed-source model. The comparison of training on open-source models should be discussed.

Author response:
We agree that an extensive study involving various LLMs, both open-sourced and closed-source, would be valuable. However, our current focus is on demonstrating the effectiveness of our approach using the PaLM2 family of LLMs. Our aim is to establish the importance of our identified pillars for recommendation endorsements (such as Precision, Appeal, etc.) and to demonstrate improvements over baseline methods. We are supplementing our model-based evaluations with human evaluations and additional dataset analysis (currently in progress) to provide more comprehensive evidence of our approach's effectiveness.

Reviewer comment (Experiments):
The author mentioned multiple ways to use PaLM to construct synthetic data in the article, but the rationality of this method has not been verified. (see Questions for more details)

Author response:
The absence of datasets containing personalized recommendation endorsement texts presented a significant technical challenge. We leveraged recent advances in RLAIF to overcome this issue, using PaLM to generate synthetic data for training our reward models.

Reviewer comment (Experiments):
The author mentions in the text the utilization of PaLM to construct data for training the reward models of Personalization and Preference Relevance. One question arises: despite the strong capabilities of LLM, there seems to be no conclusive evidence to suggest their proficiency in accomplishing this task effectively. Can the author provide corresponding evidence through real-world data or human evaluation?

Author response:
In response to your suggestion, we are conducting human evaluations and utilizing the Amazon product dataset to train P4LM.

Reviewer comment (Experiments):
Due to the wealth of global knowledge and generalization capabilities possessed by LLM, could P4LM, after undergoing direct training in the movie domain, be applicable to other domains?

Author response:
Your question about the cross-domain applicability of P4LM is intriguing. While there may be cases where preferences in one domain inform another, the ideal approach would be to train P4LM on a diverse set of domains to enhance its capability in generating personalized endorsement texts.