Paper: SUBER: An RL Environment with Simulated Human Behavior for Recommender Systems (ICLR-2024)

Reviewer comment (Experiments):
The paper does not discuss the limitation of SUBER. In my opinion, I think the input feature is quite simple, and miss numerical and sequential informations.

Author response:
The feature space of our environment contains both numerical and sequential information, i.e., for each user we have a list of all movies previously watched together with their rating in sequential order. 

Since we wanted to demonstrate that a simple RL model can learn to recommend items in our environment, we use a simple state space that only records the last rating that the user gave to a movie. We chose this simpler scenario to show the learnability of our environment, and that the trained policy has some interesting features that we can interpret as human (cf. Appendix E.1, Figure 7). We have added more experiments on other RL algorithms in the revised manuscript (Section 4.3).

It is also worth noting that the environment can be adapted easily and can support any type of feature.

Reviewer comment (Experiments):
The paper does not compare SUBER and other RL-based simulators, such as VirtualTaobao, RecoGym. Thus it is quite hard to evaluate the significance of SUBER in the RL4RS area.

Author response:
The lack of comparisons with alternative environments is mainly because of the synthetic nature of our experimental setting. Thus, it is impractical to draw direct comparisons with other environments. Our environment is more closely related to Recogym [1] and Recsim [2] in this regard. We have provided a more extensive answer on this topic in the general response.

Reviewer comment (Novelty):
The paper does not cite two recent papers about RL4RS simulators, "RL4RS: A Real-World Dataset for Reinforcement Learning based Recommender System" and KuaiSim: A Comprehensive Simulator for Recommender Systems.

Author response:
Thank you for pointing out these two contemporary works, we have updated our related work section to include these in the revised manuscript.

Reviewer comment (Experiments):
As an RL environment, I would suggest that the author evaluate more RL algorithms besides A2C, such as SA2C (Supervised Advantage Actor-Critic for Recommender Systems), HAC (Exploration and Regularization of the Latent Action Space in Recommendation) and off-policy top-k (Top-K Off-Policy Correction for a REINFORCE Recommender System).

Author response:
We have extended our analysis beyond the initial experiments with A2C, and added evaluations for additional RL algorithms (e.g., PPO, TRPO, DQN).

Regarding the specific methods the reviewer mentioned (SA2C, HAC, top-k), we face practical limitations. These methods currently lack publicly available, gym-compatible interfaces, making direct comparison within our environment challenging. We acknowledge their importance and hope to include such methods in future work as they become more accessible.

However, we would like to emphasize that our primary contribution lies in providing a versatile recommender system environment designed for experimenting with various RL and RecSys algorithms. The intent is to offer a platform for future explorations rather than conducting exhaustive evaluations of all existing algorithms.

Reviewer comment (Novelty):
The proposed framework is to tackle key challenges in RL4Rec, such as data accessibility, the uncertainty of the user model, and the assessment of models. However, the originality of this research is ambiguous to me. It appears to predominantly integrate components of Reinforcement Learning (RL) with Large Language Models (LLMs). Furthermore, due to the absence of a thorough comparison with existing RL simulators and state-of-the-art techniques including RL4Rec and LLM-integrated RecSys, it's challenging to position the precise significance of the contributions claimed in this study.

Author response:
All previous related work has either incorporated LLMs as the RecSys model itself, or used other strategies (e.g., GAN, transformers, statistical models) to simulate realistic user behavior. LLMs have been shown to excel in simulating user behavior through natural language [1,2].

The main novelty of our work is that we tackle the challenges in RL4Rec by introducing LLMs into the environment on which the RL RecSys algorithm is trained. This approach allows us to create a synthetic environment which simulates user behavior in a more natural and realistic manner. Furthermore, our environment provides practitioners the ability to train and evaluate their RL RecSys agents online.

We have added a table and updated the related work in the revised manuscript to provide a more in-depth comparison with existing research.

Reviewer comment (Experiments):
The prompts used in the pre-processing module and the user description generation step require hand-crafted templates, which may limit the generalizability of the method in other scenarios.

Author response:
We have constructed a general prompt that works for, but is not limited to, both movie and book recommendation, where we change only specific wording such as “movie” with “books”.
In terms of generalizability, this is not an issue. The prompting component can be freely customized to use different types of items in different settings. The environment is designed with modularity in mind, and therefore, adapting different components for different scopes is feasible. In fact, the prompt used in all of our experiments is not fixed and can be changed at will.

Reviewer comment (Evaluation):
This method may require more computational resources than other methods due to the usage of LLM. More analysis and evaluations should be done.

Author response:
We performed benchmarks to understand the throughput (interactions/second) that our environment can achieve (see Table 18 in Appendix E.3 for a single GPU). It is also worth noting that the environment can be accelerated using multiple GPUs in parallel. In order to achieve the interactions/second mentioned in Table 18, we also engineered a specific prompt structure, which takes advantage of caching and significantly speeds up (4x) the performance of the environment (cf. paragraph 2 section 4.1).

Reviewer comment (Evaluation):
The paper claims that it addresses the challenge of model evaluation, is it referring to the evaluation metrics mentioned in Section 4.2 and Table 1-2? How do these metrics outperform the existing evaluation methods?

Author response:
Table 4 in Section 4.2 and Table 13 in Appendix D.2 (before Table 1-2) are not tools for evaluating RL models; rather, they serve to qualitatively assess our environment's capability to mimic human concepts (e.g., genre preference, item collection), via natural language analysis.

The challenge of model evaluation in our paper specifically pertains to the inherent complexities of evaluating an RL model in RL4Rec, where one trained model is to be tested on live, online data rather than relying on pre-recorded, logged data. 

We address this challenge by developing our synthetic environment on which RL models can be tested online.

> How do these metrics outperform the existing evaluation methods?

These metrics are unique to our environment and are employed internally for the evaluation of the environment and its different modular components, rather than the trained RL RecSys model. Hence, they are not meant for direct comparison with the existing evaluation metrics used by other related work.

Reviewer comment (Evaluation):
As addressed in the weakness section, I think the comparisons between this method and other methods on metrics such as MAP/R^2/Personalization are essential to verify the effectiveness of the method. Could the authors provide these results?

Author response:
In the updated manuscript, we tested various RL algorithms trained in our proposed environment on common recommendation metrics, such as MAP@10, MMR@10, and personalization.

Reviewer comment (Experiments):
I’m not sure about the purpose of generating the user descriptions. The paper mentioned that the Age / Job / Hobbies of the users are randomly sampled from external distributions, how does this random information affect the outcome? And what’s the motivation for doing so? An ablation study to compare the results with/without this information would be helpful.

Author response:
Generating the user description is important because LLMs work with textual data. The LLM is at the core of our environment and is used to simulate users. The RL model interacts with the environment by interacting with these simulated users that are part of the environment.

The user description is generated by the LLM. Note that the dataset generation is done a priori and then stored in memory, it is not done during the RL training. Some basic additional information is provided to the LLM to generate the user description, such as age/profession/hobbies. The generated user description will strongly depend on this information, as it is included in the prompt when generating the user. The main goal of this process is to generate users that are different from each other; if the users are generated randomly by the LLM, they will suffer from some inherent biases of the language model. We found that users generated by the LLM without providing additional information were very similar to each other, and therefore, not diverse enough for an environment. In contrast, LLMs generating users using this additional information do not suffer from this, and the generated users represented a diverse group of humans (cf. Appendix A.1 Figure 4).

Reviewer comment (Experiments):
It’s known that different prompt methods could affect the response of LLMs. How does the prompt template affect the outcomes in this framework? I suggest the authors try several different prompt templates in the pre-processing module and the user description generation step, then report the range of the results.

Author response:
We are aware that prompting is very important and different templates can influence the outcome of the LLM. Therefore, we compared various different prompt templates (Table 4 in Section 4.2 and Table 13 in Appendix D.2). We experimented with few shots prompting, shifting the range of the rating scale as well as custom system prompts. It is worth mentioning that during the development of our environment we tested a multitude of different prompt strategies, however, we only present the most relevant results in our work.
Additionally, we also worked on the prompt structure to enhance the performance of our environment (Section 4.1). The prompts are constructed such that we can leverage the key-value cache of LLMs to speed up (4x) the interactions/second of our environment and reduce the time needed to train an RL model on it.

Reviewer comment (Evaluation):
In Fig 4a, I observed a significant performance drop somewhere between 1.6M - 1.7M steps, why does this happen? It seems not due to the randomness because this pattern is consistent across all embedding dimensions. Or more generally, I found the pattern of these three lines seems to be extremely similar, I’m surprised by this because I suppose these are three independent experiments with different dimension settings. Is there any particular reason for the similarity between these three lines?

Author response:
This phenomenon was due to the same seed used across the experiments for reproducibility and consistency. We have added more experiments with multiple seeds and computed the confidence intervals for all of our testing. As shown in Appendix E.1 Figure 6 in the revised manuscript, we no longer have such a significant performance drop. We have updated the revised manuscript with the new results.

Reviewer comment (Experiments):
The manuscript could benefit from more robust experimental support. The absence of comparisons with other simulation algorithms may impact the persuasiveness of the paper.

Author response:
The lack of comparisons with other environments is mainly due to the synthetic nature of our environment. As a result, comparing our environment with others is not feasible. From this perspective, we are more similar to Recogym [1] and Recsim [2]. We have provided more details in the general response.

Reviewer comment (Experiments):
The paper appears to lack some key experiments that validate the effectiveness and advantages of RL training in the proposed environment. While the paper presents ablation studies on different components of the environment, it does not compare the RL-based recommender system with other baselines or state-of-the-art methods on real user data or benchmark datasets. It would be advantageous if the authors could include such experimental validations in future research.

Author response:
We have added additional experiments with different RL models in our environment, in particular A2C, PPO, DQN, and TRPO. In Section 4.3, we demonstrated that A2C can effectively learn in our environment by achieving a high average reward, MAP@10, and MRR@10 score. Additionally, we also demonstrate random qualitative examples. Appendix E further illustrates how A2C is able to recommend movies according to the user's genre preferences.

Reviewer comment (Evaluation):
The use of LLMs to generate synthetic users is an innovative approach that leverages the powerful capabilities of LLMs to simulate human behavior and preferences. However, the paper does not evaluate the quality and diversity of the user generation, nor does it compare it with real user data. This could potentially lead to biases and inaccuracies in the simulation. It would be beneficial if the authors could delve deeper into this aspect in future research.

Author response:
We agree with the reviewer that the use of LLMs could potentially lead to bias. To mitigate this, we reduce the bias by explicitly mentioning in the system prompt that the model should not be biased and should answer the prompts realistically (Appendix A.3.1). Furthermore, we sampled characteristics of users like professions, hobbies, age, and interests. This process ensures the LLM creates unbiased synthetic users (e.g., not biased on age or interests) (Appendix A.1). Hobbies and professions are randomly sampled from a list, while age and interests are sampled from real human distributions (Appendix A.1). These choices are included in the prompt, so when the LLM generates the user description, it is conditioned on the previously sampled characteristics. This allows for a more diverse distribution of users, rather than being biased towards a particular group of people. We show the effectiveness of our approach in Figure 4 (Appendix A.1).

Reviewer comment (Evaluation):
Can the author provide a distribution chart for scores predicted by LLMs and actual scores? The numbers reported in the table do not provide an intuitive image. On the other hand, it is more important to focus not on the overall score distribution, but on the differences in scoring for each item (it is possible for the overall score distribution to be the same, but with significant differences in individual item scores).

Author response:
We have added a graph of the score probability distribution in Figure 5 (Appendix D.1) of the revised manuscript.

Reviewer comment (Presentation):
What is the difference between the last two rows in Table 1? Is it a type error?

Author response:
Not an error, these are two different strategies of prompting, in one case “1-10” refers to the digits between “1” and “10” in the prompt, while “one-ten” refers to ratings in the prompt which were written in words (“one”, “two”, “three”, ..., “ten”) (cf. 4th paragraph section 4.1). We have updated the table captions to better reflect this in the revised manuscript.

Reviewer comment (Evaluation):
Comparing rows 3-8 in Table 1 with the 9th row, it can be observed that the model is very good at distinguishing between High and Low Ratings on a scale of 0-9. However, when the score scale changes to 1-10, there is a significant decrease in performance. Does this indicate that the model can only identify very poor items? (0 score)

Author response:
We found this to be an interesting problem. We believe this is related to the ambiguity of the tokenization, however, we were unable to find any research discussing this issue. In general, the 0-9 scale performs much better than the 1-10 scale in almost all of our experiments, mainly because the token “10”, which represents the best score, can be tokenized in 2 ways, either the token “10” directly or the token “1” and then the token “0” (cf. 4th paragraph section 4.1).
In these experiments, the 0-9 scale outperforms the 1-10 scale significantly. This is because, in the 0-9 scale, when the language model assigns a high rating, it directly corresponds to the token “9”. In contrast, in the 1-10 scale, the model occasionally generates the rating “10” by producing the digits “1” and “0” separately, whereas, in some cases, the model generates the EOS-token after computing the token “1”, leading the model to obtain a high loss since it predicted score (1) compared to the true score (10).

Reviewer comment (Experiments):
I suggest the author showcases the performance of several traditional models and conventional RL methods, under your reward metric (Figure .4a)

Author response:
In the revised manuscript, we added more experiments on training RL methods like PPO, A2C, DQN and TRPO. The experiments can be found in Section 4.3.

Reviewer comment (Evaluation):
How do the authors assess the accuracy of their simulated environment? The paper primarily showcases the training curve of the RL model within this environment but lacks a comparative analysis against other environment simulation methods. The absence of online experiments further challenges the validity of the simulated environment.

Author response:
The absence of any comparisons with other environments is primarily due to the synthetic nature of our environment. Unfortunately, comparing our environment with others is unfeasible. In this regard, our environment bears more similarity to Recogym [1] and Recsim [2]. We have provided more details on this matter in the general response.

Reviewer comment (Novelty):
The authors should expound on their rationale for using LLMs to simulate users, clarify why this approach is effective, and outline the advantages it offers over alternative environment simulation methods.

Author response:
The main motivation for using LLMs is their ability to simulate human behavior well, as demonstrated by previous work [3, 4]. Therefore, such an environment is able to capture human dynamics and interests. We have updated the introduction of the revised manuscript to emphasize this aspect. Furthermore, LLMs undergo training on extensive datasets that include information related to various subjects. This enhances their capability to grasp context, and therefore, respond in a realistic manner.

Reviewer comment (Evaluation):
The authors claim that this dynamic environment can serve as a model evaluation tool for recommender systems, but it lacks empirical evidence to support this claim. A clear methodology for measuring the accuracy of the proposed evaluation method is needed.

Author response:
We conduct additional experiments using different RL methods within our proposed environment (Section 4.3). This environment is based on user ratings - in other words, when a movie is recommended to a user, the RL model receives a reward corresponding to the user's rating for that movie. Consequently, the average reward accumulated by the RL model serves as a key metric for evaluating its performance in the environment.

As shown in the experimental results described in Section 4.3, we can measure the effectiveness of the RL model by monitoring its average reward. A higher average reward indicates that the model has successfully selected movies that match interests of users. In addition to average reward, we have added additional metrics such as MAP@10, MMR@10, and personalization to provide a comprehensive evaluation of different RL models.

Reviewer comment (Reproducibility):
The paper mentions the limitation of context length for providing a list of all possible items to an LLM. Further details are required regarding how the author addressed this particular issue.

Author response:
To address the limitation of a fixed context length, we have implemented the Item Retrieval component. For a given user, when constructing a prompt, the Item Retrieval component selects only a subset of items previously seen by the user. We mention this aspect and the role of the Item Retrieval component in Section 3.2.

Reviewer comment (Experiments):
I suggest the authors evaluate the simulated environment from more aspects. To establish the accuracy of the simulated environment, consider conducting comparative experiments. Compare the performance of your proposed environment with existing methods for simulating user interactions. Additionally, performing online experiments where applicable, could help validate the authenticity of your simulated environment.

Author response:
To the best of our knowledge, there is no practical and accurate method for comparing synthetic environments. We address this topic in more detail in the general response. If the reviewer holds a different perspective on this matter, we welcome the opportunity to engage in further discussion.

Reviewer comment (Novelty):
I suggest the authors provide a more in-depth explanation of why LLMs are chosen to simulate users. Elaborate on the effectiveness of this approach by highlighting its advantages over alternative simulation methods. This could include discussing how LLMs can capture complex user behavior or adapt to changing patterns more effectively.

Author response:
We have added more qualitative explanations why LLMs are good for simulating users in the revised manuscript. We base ourselves on the work by Park et al. [3] and Argyle et al. [4] which show that LLMs are good simulators of human behavior.

Reviewer comment (Experiments):
To substantiate the claim that your dynamic environment serves as a model evaluation tool, conduct experiments that demonstrate its utility in evaluating recommender systems. Present a clear experimental setup and results that support this assertion.

Author response:
We added more experiments in Section 4.3 to compare different RL methods trained on SUBER.

As shown in the experimental results described in Section 4.3, we can measure the effectiveness of the RL model by monitoring its average reward. A higher average reward indicates that the model has successfully selected movies that match interests of users. In addition to average reward, we have added additional RecSys specific metrics such as MAP@10, MMR@10, and personalization to provide a comprehensive evaluation of different RL models.

Reviewer comment (Evaluation):
Enhance the reliability and reproducibility of your experimental results by including t-tests or error bars.

Author response:
We have conducted more experiments and added error bars and confidence intervals in the experiment section of the revised manuscript.

Reviewer comment (Reproducibility):
Explain in detail how you addressed the limitation of limited context length. What techniques or strategies did you employ to mitigate this constraint when using LLMs in your environment?

Author response:
We implemented three types of item retrieval: Recency Retrieval, in other words we retrieve the most recently seen items, Feature Similarity Retrieval based on the movie features, and T5 Similarity Retrieval based on the movie description. See Section 3.2 and Section 4.1 for more details. Furthermore, we compare these retrieval approaches against each other in Section 4.2.