Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models (2023)

Abstract: The recent success of large language models (LLMs) has shown great potential to develop more powerful conversational recommender systems (CRSs), which rely on natural language conversations to satisfy user needs. In this paper, we embark on an investigation into the utilization of ChatGPT for CRSs, revealing the inadequacy of the existing evaluation protocol. It might overemphasize the matching with ground-truth items annotated by humans while neglecting the interactive nature of CRSs. To overcome the limitation, we further propose an **i**nteractive **Eva**luation approach based on **L**L**M**s, named **iEvaLM**, which harnesses LLM-based user simulators. Our evaluation approach can simulate various system-user interaction scenarios. Through the experiments on two public CRS datasets, we demonstrate notable improvements compared to the prevailing evaluation protocol. Furthermore, we emphasize the evaluation of explainability, and ChatGPT showcases persuasive explanation generation for its recommendations. Our study contributes to a deeper comprehension of the untapped potential of LLMs for CRSs and provides a more flexible and realistic evaluation approach for future research about LLM-based CRSs.

## introduction conversational recommender systems (crss) aim to provide high-quality recommendation services through natural language conversations that span multiple rounds. typically, in crss, a recommender module provides recommendations based on user preferences from the conversation context, and a conversation module generates responses given the conversation context and item recommendation. since crss rely on the ability to understand and generate natural language conversations, capable approaches for crss have been built on pre-trained language models in existing literature @xcite . more recently, large language models (llms) @xcite , such as chatgpt, have shown that they are capable of solving various natural language tasks via conversations. since chatgpt has acquired a wealth of world knowledge during pre-training and is also specially optimized for conversation, it is expected to be an excellent crs. however, there still lacks a comprehensive study of how llms (e.g., chat-gpt) perform in conversational recommendation. to investigate the capacity of llms in crss, we conduct an empirical study on the performance of chatgpt on existing benchmark datasets. we follow the standard evaluation protocol and compare chatgpt against state-of-the-art crs methods. surprisingly, the finding is rather counter-intuitive: chatgpt shows unsatisfactory performance in this empirical evaluation. to comprehend the reason behind this discovery, we examine the failure cases and discover that the current evaluation protocol is the primary cause. it relies on the matching between manually annotated recommendations and conversations and might overemphasize the fitting of ground-truth items based on the conversation context. since most crs datasets are created in a chit-chat way, we find that these conversations are often vague about the user preference, making it difficult to exactly match the ground-truth items even for human annotation. in addition, the current evaluation protocol is based on fixed conversations, which does not take the interactive nature of conversational recommendation into account. similar findings have also been discussed on text generation tasks @xcite : traditional metrics (e.g., bleu and rouge) may not reflect the real capacities of llms. considering this issue, we aim to improve the evaluation approach, to make it more focused on the interactive capacities of crss. ideally, such an evaluation approach should be conducted by humans, since the performance of crss would finally be tested by real users in practice. however, user studies are both expensive and time-consuming, making them infeasible for large-scale evaluations. as a surrogate, user simulators can be used for evaluation. however, existing simulation methods are typically limited to pre-defined conversation flows or template-based utterances @xcite . to address these limitations, a more flexible user simulator that supports free-form interaction in crss is actually needed. to this end, this work further proposes an interactive evaluation approach based on llms, named ievalm, in which llm-based user simulation is conducted to examine the performance. our approach draws inspiration from the remarkable instruction-following capabilities exhibited by llms, which have already been leveraged for roleplay @xcite . with elaborately designed instructions, llms can interact with users in a highly cooperative manner. thus, we design our user simulators based on llms, which can flexibly adapt to different crss without further tuning. our evaluation approach frees crss from the constraints of rigid, human-written conversation texts, allowing them to interact with users in a more natural manner, which is close to the experience of real users. to give a comprehensive evaluation, we also consider two types of interaction: attribute-based question answering and free-form chit-chat. with this new evaluation approach, we observe significant improvements in the performance of chatgpt, as demonstrated through assessments conducted on two publicly available crs datasets. notably, the recall@10 metric has increased from 0.174 to 0.570 on the redial dataset with fiveround interaction, even surpassing the recall@50 result of the currently leading crs baseline. moreover, in our evaluation approach, we have taken the crucial aspect of explainability into consideration, wherein chatgpt exhibits proficiency in providing persuasive explanations for its recommendations. besides, existing crss can also benefit from the interaction, which is an important ability overlooked by the traditional evaluation. however, they perform much worse in the setting of attribute-based question answering on the opendialkg dataset, while chatgpt performs better in both settings on the two datasets. it demonstrates the superiority of chatgpt across different scenarios, which is expected for a general-purpose crs. we summarize our key contributions as follows: ## background and experimental setup in this section, we describe the task definition and experimental setup used in this work. ## chatgpt for conversational recommendation in this section, we first discuss how to adapt chat-gpt for crss, and then analyze its performance. ## a new evaluation approach for crss considering the issues with the existing evaluation protocol, in this section, we propose an alternative evaluation approach, ievalm, which features interactive evaluation with llm-based user simulation, as illustrated in figure 3 . we demonstrate its effectiveness and reliability through experiments. ## evaluation results in this section, we assess the quality of the user simulator and the performance of crss using our proposed evaluation approach. ## conclusion in this paper, we systematically examine the capability of chatgpt for conversational recommendation on existing benchmark datasets and propose an alternative evaluation approach, ievalm. first, we show that the performance of chatgpt was unsatisfactory. through analysis of failure cases, the root cause is the existing evaluation protocol, which overly emphasizes the fitting of ground-truth items based on conversation context. to address this issue, we propose an interactive evaluation approach using llm-based user simulators. through experiments with this new approach, we have the following findings: (1) chatgpt is powerful and becomes much better in our evaluation than the currently leading crss in both accuracy and explainability; (2) existing crss also get improved from the interaction, which is an important aspect overlooked by the traditional evaluation; and (3) chatgpt shows great potential as a generalpurpose crs under different settings and datasets. we also demonstrate the effectiveness and reliabil-ity of our evaluation approach. overall, our work contributes to the understanding and evaluation of llms such as chatgpt for conversational recommendation, paving the way for further research in this field in the era of llms.