R ecom M ind: Movie Recommendation Dialogue with Seeker’s Internal State (2024)

Abstract: Humans pay careful attention to the interlocutor’s internal state in dialogues. For example, in recommendation dialogues, we make recommendations while estimating the seeker’s internal state, such as his/her level of knowledge and interest. Since there are no existing annotated resources for the analysis and experiment, we constructed RecomMind, a movie recommendation dialogue dataset with annotations of the seeker’s internal state at the entity level. Each entity has a first-person label annotated by the seeker and a second-person label annotated by the recommender. Our analysis based on RecomMind reveals that the success of recommendations is enhanced when recommenders mention entities that seekers do not know but are interested in. We also propose a response generation framework that explicitly considers the seeker’s internal state, utilizing the chain-of-thought prompting. The human evaluation results show that our proposed method outperforms the baseline method in both consistency and the success of recommendations.

## introduction in human dialogues, individuals pay careful attention to their interlocutor's internal state (chiba et al., 2014), including their level of understanding and emotional states. particularly in recommendation dialogues, where a recommender suggests something to a seeker, it is crucial to estimate what the seeker knows and what they are interested in. this understanding allows for recommendations that better align with the seeker's preferences. in the past few years, many large language models (llms) have been actively developed and have achieved remarkable performance in various natural language processing tasks (brown et al., 2020; zhang et al., 2022; chowdhery et al., 2022; ope-nai, 2023). current llms are able to generate human-like responses without specialized modules to consider the interlocutors. however, it remains an open question whether llms need to explicitly consider the seeker's internal state and how to effectively implement it. to answer this question, we need dialogue data with careful and fine-grained annotations of the seeker's internal state. unfortunately, there are no existing recommendation dialogue datasets with internal state annotation. one possible solution is to annotate existing recommendation dialogue datasets (li et al., 2018; kang et al., 2019; moon et al., 2019; liu et al., 2020; hayati et al., 2020; zhou et al., 2020; jia et al., 2022) with the seeker's internal state. however, the internal state labels annotated by a third party may not accurately reflect the actual state (kajiwara et al., 2021). to obtain the actual internal state of seekers, it is necessary for the seekers themselves to perform the annotation. to account for the aforementioned requirement, we constructed recommind, a movie recommendation dialogue dataset in japanese. as illustrated in figure 1 foot_1 , the recommender suggests movies based on the seeker's preferences in a dialogue. during the dialogue, noun phrases, referred to as entities, are automatically extracted from utterances. both participants (i.e., the recommender and the seeker) annotate each extracted entity with the seeker's level of knowledge and interest at three levels: high, neutral, and low during or immediately after the dialogue. in this annotation, the seekers assign first-person labels, which reflect their own internal states. in contrast, recommenders, not knowing the seekers' actual internal states, assign second-person labels, which reflect their estimation of the seekers' internal states based on the interacseeker assigns first-person labels interest knowledge high high low low top gun: maverick neutral high neutral high avengers series high high high high top gun recommender assigns second-person labels tions. with these procedures, the seeker's internal states during a dialogue are recorded from the two perspectives of the recommender and seeker. using the constructed dataset, we analyze the relationship between the seeker's internal state and the recommendation success. our analysis reveals that entities without knowledge but with interest contribute to successful recommendations. this finding suggests that the recommender should focus on topics or subjects that the seeker lacks knowledge of yet is interested in. furthermore, we also propose a llm-based response generation framework that explicitly considers the seeker's internal state. specifically, we apply chain-of-thought prompting (wei et al., 2022) and estimate the seeker's internal state before generating a response. the human evaluation results demonstrate that our proposed method outperforms the baseline method, which does not explicitly consider the seeker's internal state, in both consistency and the successful recommendations. in summary, our contributions are as follows. ## related work our research centers on the interlocutor internal state in a dialogue, in particular, the level of knowledge and interest. here, we introduce the previous studies that deal with knowledge and interest in dialogues. miyazaki et al. (2013) proposed a method to estimate callers' levels of knowledge about particular themes (e.g., troubleshooting of products and services) in call center dialogues. their annotations are conducted at the dialogue level, whereas our dataset is annotated at the entity level. this allows for more fine-grained knowledge-state tracking and analysis. inspired by the theory of mind (premack and woodruff, 1978) and the common ground (clark, 1996), bara et al. (2021) created mindcraft dataset which considers the user's knowledge for situated dialogue in collaborative tasks. given the necessary knowledge and skills, two workers are asked to create a specific object together in the 3d virtual blocks world of minecraft. the players must periodically answer a question about the common ground (e.g., "do you think the other player knows how to make yel-low_wool?"). in this study, we consider the user's knowledge in a more realistic dialogue that contains both chit-chat and recommendations. modeling interlocutors' interests have been actively studied in the field of recommendation dialogue (kang et al., 2019; liu et al., 2020; zhou et al., 2020; jia et al., 2022). in gorecdialog (kang et al., 2019), each worker is given a set of five movies. the seeker's set represents their watching history, while the recommender's represents candidate movi es.the recommender should recommend the appropriate movie among the candidates to the seeker. durecdial (liu et al., 2020) is a recommendation dialogue dataset containing multiple dialogue types, such as question-answering and chit-chat. the recommender attempts to elicit the seeker's preferences, and the seeker responds based on a predefined user profile. these studies focus on the preferences for predefined objects (e.g., movies, user profiles). our dataset differs in that we annotate all entities appearing in dialogues with the seeker's interest. ## data collection we collect dialogues via crowdsourcing through a data supplier in japan. in this section, we describe how we collect the recommind dataset. ## experiment the analysis in section 3.3.2 suggests the importance of understanding the seeker's internal state at the entity level. thus, we propose a response generation framework that explicitly considers the seeker's internal state at the entity level. in this section, we describe our proposed method and verify its effectiveness. ## conclusion we constructed recommind, a recommendation dialogue dataset that features both first-and secondperson annotations of the seeker's internal state at the entity level. our dataset also has engaging dialogues with longer seeker's utterances, characterized by high scores in dialogue enjoyment and recommendation success. we also proposed a response generation framework that explicitly considers the seeker's internal state, applying chain-of-thought prompting to our task. the experimental results showed that our proposed method could generate responses that are more consistent and tailored to the seeker than the baseline method. our dataset has diverse and fine-grained annotations, which are useful for various tasks such as internal state estimation, external knowledge selection, and dialogue response generation. we hope our dataset will be useful for future research on recommendation dialogues.