Paper: Evidential Conservative Q-Learning for Dynamic Recommendations (ICLR-2024)

Reviewer comment (Evaluation):
I'm concerned about the effect-boosting effect of the proposed method on different datasets. Since the improvement is not large, did the authors perform the hypothesis testing to prove the significance of improvements? Also, the effect enhancement of the proposed method is much smaller in the largest dataset Yahoo! than in other datasets. Is there any specific research on this phenomenon?

Author response:
Thank you for this comment. We have provided the standard deviation of the proposed ECQL model along with the most competitive baselines from each category in Table 7 of Appendix E.2. The small standard deviation clearly the performance advantage over other methods. We further conduct a significance test to compare ECQL with the second best performing baseline CL4SRec. We obtain a p-value of 0.04, which confirms the performance advantage of ECQL over CL4SRec is statistically significant. To better understand the impact on the size of the datasets, we have conducted additional experiments on two larger datasets as suggested by the reviewer. As can be seen from the answer to Q2 in the general response, our model achieves much better P@5 and nDCG@5 as compared with two most recent RL based baselines.

Reviewer comment (Presentation):
Some diagrams can be demonstrated in a more expressive way. For example, Figure 2 could insert some images to illustrate the composition of the modules in the network.

Author response:
Thanks for the suggestion. We updated Figure 2 in the revised paper to better illustrate the modules composition.

Reviewer comment (Writing):
Some formulas are not written very rigorously, for example: all mathematical formulas are not followed by a punctuation mark. In addition, some expressions are not sufficiently formal, such as 'score' and 'rating'.

Author response:
Following the reviewer's suggestion, we have added punctuation marks to all the equations. The mathematical definitions of score and rating are given in Equation 2 and the following text, respectively. We are happy to make them more formal if the reviewer has more specific suggestions.

Reviewer comment (Novelty):
The motivation of this work is not convincing. It is claimed that epsilon-greedy may not be able to learn the optimal policy that captures effective user preferences and achieves the maximum expected reward over the long term. This claim is not supported by any evidences.

Author response:
**Q1. It is claimed that $\epsilon$-greedy may not be able to learn the optimal policy that captures effective user preferences and achieves the maximum expected reward over the long term. This claim is not supported by any evidence.**

Since $\epsilon$-greedy performs completely random exploration, it is very likely that the explored items may contain the ones that the user is already familiar with or those that completely deviate from users' interest. In contrast, as mentioned above, the vacuity-guided exploration encourages the RL agent to recommend items that the model has the least knowledge. After collecting the user feedback, the RL agent can most effectively gain knowledge of the user preference to make better recommendations in the long run. Besides $\epsilon$-greedy, we also compared with another commonly used exploration strategy, soft-actor-critic (SAC), which leverages entropy to perform exploration. However, as we made clear in the paper, a high entropy may imply either high vacuity (lack of evidence) or high dissonance (conflict of strong evidence). However, dissonance is not effective for exploration in RS due to its focus on confusing items mostly derived based on the users' current interests. Figure 8 of the Appendix shows that ECQL achieves a much higher cumulative reward than SAC and $\epsilon$-greedy, which confirms its effectiveness in capturing users' long-term interest. Table 12 of Appendix E.4 presents a qualitative study to demonstrate that ECQL is able to recommend more diverse items than SAC and $\epsilon$-greedy relevant to user's preference. Finally, Table 11 of Appendix E.3 provides a comparison with a collaborative contextual bandit based method CoLin to further demonstrate the effective exploration of ECQL.

Reviewer comment (Theory):
To the best of my knowledge, the \epsilon-greedy strategy is a simple approach to balance the exploration vs. exploitation tradeoff of RL. As long as the RL model is tailored to RS in a right manner and the exploration vs. exploitation tradeoff is well balanced, the learned policy may not have the claimed weakness. Also, one can tune epsilon to attain different strengths of exploration.

Author response:
Dear Reviewer Yout,

Thank you for reviewing our paper and providing valuable comments. Specifically, we explain the major limitations of the existing RL-based methods and introduce the motivation and novelty of ECQL by elaborating that the systematic integration of evidence-based uncertainty and conservative learning is important to an effective and cautious recommendation policy. In contrast, $\epsilon$-greedy conducts random exploration with no control, so it is very likely that the explored items may contain the ones that the user is already familiar with or those that completely deviate from users' interest. Further, we compare with $\epsilon$-greedy and other RL exploration strategies such as soft-actor-critic (SAC) and a collaborative contextual bandit based method CoLin to demonstrate the superiority of vacuity-guided exploration.

Reviewer comment (Presentation):
I am also confused by Figure 1. Little details are provided. It is hard for me to judge whether Figure 1 is the outcome of improper applying of RL to RS or some other factors. For example, is it a consequence of the optimal policy of the RL having such weakness or the epsilon-greedy strategy leading to a sub-optimal strategy that has such weakness.

Author response:
**Q2.  For Figure 1. Little details are provided.  It is hard for me to judge whether Figure 1 is the outcome of improper applying of RL to RS or some other factors.**

In Figure 1, we illustrate that the existing RL methods based on $\epsilon$-greedy do not provide effective exploration. Their recommendations usually concentrate on a narrower items space, primarily focusing on highly rated items but only reflecting short-term interest. In contrast, the proposed ECQL method provides more informative items to help models learn the user's long-term interest.

Reviewer comment (Evaluation):
In the related work, I am not convinced by the claim that randomized exploration strategies are less effective at capturing users’ long-term preferences. Could you provide any evidence?

Author response:
**Q4. I am not convinced by the claim that randomize exploration strategies are less effective at capturing users' long-term preferences. Could provide any evidence?**

Please refer to our detailed response to Q1 that mentions multiple concrete evidences provided in the paper to explain why random exploration is less effective.

Reviewer comment (Theory):
It is not clear how the proposed exploration strategy connects to balancing the exploration vs. exploitation tradeoff. In terms of the balance between exploration and exploitation, it is unclear why it is better than the epsilon-greedy or fine-tuned epsilon-greedy.

Author response:
**Q5.  In terms of the balancing exploration vs. exploitation tradeoff, it is unclear why it is better than the epsilon-greedy or fine-tuned $\epsilon$-greedy.**

To achieve optimal policy in the guided exploration system, it is always important to balance exploration and exploitation. By leveraging second-order uncertainty (vacuity) guided exploration, we can effectively locate items that the model lacks the knowledge for now but may reflect the model's long-term interest to build a more precise user profile. Please refer to our discussion on the motivation and novelty of ECQL for more details.

Reviewer comment (Novelty):
In the Section 2, the authors do no include the recent studies about sequential recommendation. The most recent dynamic/sequential recommendation methods mentioned by them are proposed in 2019. The sequential recommendation methods developed in recent 4 year are not discussed in Section 2. Similarly, the authors are suggested to include some discussions about the RL methods developed in recent 3 years.

Author response:
**Q1. In the Section 2, the authors do not include the recent studies about sequential recommendation... Similarly, the authors are suggested to include some discussions about the RL methods developed in recent 3 years.**

Thanks for the suggestion. Please refer to the answer to Q1 in the general response.

Reviewer comment (Evaluation):
One advantage of the proposed method is to provide diverse recommendations. However, in Table 2, the authors do not analyze the diversity of the generated recommendation results.

Author response:
**Q2.  One advantage of the proposed method is to provide diverse recommendations. However, in Table 2, the authors do not analyze the diversity of the generated recommendation results.**

Thank you for the suggestion. We have provided the qualitative analysis of diversity in Table 12 of the Appendix. To further provide the quantitative evidence, we calculate the Gini index of the diversity for all the recommended results. First, we test 1,800 users from Netflix test set and then collect 16 (number of time steps) $\times$ 1,800 total recommendations. For each recommendation, we calculate its Gini index by separating 5 results into different categories and then calculate 
\begin{align}
Gini=1-\sum_{c=1}^{C}P(c)^2    
\end{align}
where $C$ is the number of categories and $P(c), c\in [1,C]$ is the probability for each category. Finally, we average the calculated Gini index for 1,800 recommendations and get the averaged Gini index as 0.71, which is close to 1. This indicates that the recommendation of our model is quite diverse and contains objects from different categories. Comparing to SAC and CoLin which have averaged Gini indexes 0.65 and 0.68 respectively, our ECQL achieves the most diverse recommendation thanks to the novel evidence-based exploration, which is also verified by Table 12 of the revised paper.

Reviewer comment (Experiments):
The authors are suggested to include more sequential recommendation methods proposed in 2021 and 2022 as baselines. TimeSVD++ and CKF are not needed to be used as baseline methods.

Author response:
**Q3. The authors are suggested to include more sequential recommendation methods proposed in 2021 and 2022 as baselines. TimeSVD++ and CKF are not needed to be used as baseline methods.**

Thank you for the suggestion. We have removed TimeSVD++ and CKF from the main comparison table and moved them to Appendix E.2 where we compare with some other baseline models. For the newer sequential recommendation models, in addition to CL4SRec (Xie et al., 2022), we have added two additional baselines: ResAct (Xue et al., 2023) and SAR (Antaris et al., 2021), and compared them with our approach on two larger datasets. Please refer to our response to Q4 below for details.

Reviewer comment (Experiments):
The Movielens-100K dataset is too small for experimental evaluation. The authors are suggested to include some larger datasets for experiments, for example Amazon review datasets.

Author response:
**Q4. The Movielens-100K dataset is too small for experimental evaluation. The authors are suggested to include some larger datasets for experiments, for example, Amazon review datasets.**

Thanks for the suggestion. Please refer to the answer to Q2 in the general response.

Reviewer comment (Experiments):
In the proposed ECQL, there are two main modules. However, there is no experiments studying the importance of these two parts. The authors are suggested to perform an ablation study.

Author response:
**Q5. In the proposed ECQL, there are two main modules. However, there is no experiments studying the importance of these two parts. The authors are suggested to perform an ablation study.**

Thanks for the suggestion. Please refer to the answer to Q3 in the general response for the detailed results.