Transparent and Scrutable Recommendations Using Natural Language User Profiles (2024)

Abstract: Recent state-of-the-art recommender systems predominantly rely on either implicit or explicit feedback from users to suggest new items. While effective in recommending novel options, many recommender systems often use uninterpretable embeddings to represent user preferences. This lack of transparency not only limits user understanding of why certain items are suggested but also reduces the user’s ability to scrutinize and modify their preferences, thereby affecting their ability to receive a list of preferred recommendations. Given the recent advances in Large Language Models (LLMs), we investigate how a properly crafted prompt can be used to summarize a user’s preferences from past reviews and recommend items based only on language-based preferences. In particular, we study how LLMs can be prompted to generate a natural language (NL) user profile that holistically describe a user’s preferences. These NL profiles can then be leveraged to fine-tune a LLM using only NL profiles to make transparent and scrutable recommendations. Furthermore, we validate the scrutability of our user profile-based recommender by investigating the impact on recommendation changes after editing NL user profiles. According to our evaluations of the model’s rating prediction performance on two benchmarking rating prediction datasets, we observe that this novel approach maintains a performance level on par with established recommender systems in a warm-start setting. With a systematic analysis into the effect of updating user profiles and system prompts, we show the advantage of our approach in easier adjustment of user preferences and a greater autonomy over users’ received recommendations.

## introduction personalized recommender systems often rely on building latent representations from past user interactions to provide recommendations. these models, while effective, suffer from a lack of interpretability on the learned users' preferences, making it challenging to provide intuitive explanations for recommendations. for example, many collaborative filtering (cf) techniques learn latent embeddings @xcite to represent information about every user and item. these embeddings are complex, continuous vectors that cannot be easily interpreted or modified @xcite . in fact, since collaborative filtering builds embeddings via past interactions, users would need to significantly change their interaction history to receive new recommendations that align with their current preferences. given the strong performance of large language models (llms) in a wide variety of nlp tasks @xcite as well as the inherent scrutable nature of text-based inputs, we explore how we can prompt llms with a natural language (nl) description of a user's preferences (i.e., user profiles) to provide greater transparency on how a model provides new recommendations. furthermore, by using nl preferences rather than embeddings, a user can easily scrutinize and update their preferences when needed. for recommender systems, we define trans-parency as a faithful representation behind the recommendation mechanism and scrutability as a direct and meaningful way to inspect and modify a user representation @xcite . addressing these issues is vital: if users receive poor recommendations, they should understand why and have the means to modify their representation for better results. research has shown that explanations that justify a recommendation help users make better and faster decisions and promote trust in the system @xcite . recent works in transparent recommendation systems have explored using pre-defined templates @xcite and automatically generated text @xcite @xcite to explain recommendations to users. however, these works focused on explaining individual item recommendations rather than holistic, personalized preferences. another approach to transparent and scrutable recommender systems is set-based preferences, which aim to explain a user's general preferences for a particular domain @xcite . specifically, rather than giving a rating for each individual item, a user can categorize groups of items into sets, which are then used to recommend new items. for example, a user can state that they generally like 'action' movies and generally dislike 'romance' movies. this transparency lets users understand the model's interpretation and adjust their profile accordingly, leading to more aligned recommendations. chang et al. @xcite have shown that set-based preferences allow users to elicit preferences faster, leading to higher satisfaction with the resulting recommended items. @xcite found that llms are competitive recommender systems in a near cold-start setting. in particular, they collected language-based user profiles from crowdsourced workers and found that by using few-shot prompting with the palm model @xcite , they were able to achieve similar performance to item-based collaborative filtering methods. in this paper, we propose user profile recommendation (upr) depicted in fig. 1 , a language-based approach to set-based, transparent recommendations in a warm-start setting. unlike a cold-start setting, a warm-start setting contains significantly more user-item interactions and could further improve the performance of collaborative filtering methods @xcite . in lieu of a real-world dataset of nl profiles, which is both expensive and challenging to collect, we instead simulate users writing and modifying nl profiles by prompting instruction-tuned llms to generate profiles based on past user reviews. we first exemplify that these machine-generated nl profiles qualitatively summarize a user's preferences concisely while maintaining a scrutable format. furthermore, according to the conducted experiments on two benchmark datasets, amazon review (movies and tv) and tripadvisor , we show how nl user profiles can be used to train a transparent, nl-based recommendation system based on scrutable nl profiles rather than uninterpretable user embeddings. we observe that our nl-based approach has competitive recommendation performance to popular baseline recommender systems while also being transparent and scrutable. our model enables a transparent and scrutinized recommendation process with competitive recommendation accuracy. in addition, past work suggests that a transparent and scrutable model is preferred because it allows users to understand the reasoning behind the model's recommendations and easily modify their preferences to receive new recommendations @xcite . to validate the advanced transparency and scrutability of our upr model, we investigate how editing user profiles affects recommendation performance. for reproducibility, we publicly release all data and code foot_0 . our key contributions include: ## conclusion and future work in this work, we propose a novel recommendation model that replaces uninterpretable user embeddings with transparent, natural language profiles that holistically describe a user's preferences. we show that instruction prompting can be used with large language models to create fluent, informative, and personalized user profiles based on user reviews. by encoding user preferences using natural language, we are able to develop a recommender system that is both transparent and scrutable. experimental results show that our method has comparable performance to popular recommender models. in addition, we show that by adding new preferences to the nl profile, we can quickly provide updated recommendations. for future work, we plan to investigate multi-turn updates to nl profiles to simulate user updates over time and explore how scrutability can be used to adjust recommendations to be less biased.