Aligning Large Language Models for Controllable Recommendations (2024)

Abstract: Inspired by the exceptional general intelligence of Large Language Models (LLMs), researchers have begun to explore their application in pioneering the next generation of recommender systems — systems that are conversational, explainable, and controllable. However, existing literature primarily concentrates on integrating domain-specific knowledge into LLMs to enhance accuracy using a fixed task template, often overlooking the diversity of recommendation tasks and the ability of LLMs to follow recommendation-specific instructions. To address this gap, we first introduce a collection of supervised learning tasks, augmented with labels derived from a conventional recommender model, aimed at explicitly improving LLMs’ proficiency in adhering to recommendation-specific instructions. Next, we propose a reinforcement learning-based alignment procedure to enhance LLMs’ generalization ability. Extensive experiments on two real-world datasets demonstrate that our approach significantly improves the capability of LLMs to respond to instructions within recommender systems, reducing formatting errors while maintaining a high level of accuracy.

## introduction recommender systems are designed to identify and suggest the most appropriate items to users from a vast array of candidates, based on the users' profiles, past interactions, and present intentions. witnessing the impressive capabilities of large language models (llms), such as knowledge retention, reasoning, and problem-solving, researchers are now exploring the integration of llms into the next wave of intelligent recommender systems, which aim to be conversational, explainable, and controllable. bridging the gap between the generalpurpose llms and the specific requirements of recommendation tasks poses a challenge. in this context, fine-tuning llms with domain-specific knowl-edge and recommendation-focused tasks emerges as a promising strategy to harness their potential for advanced recommendation purposes @xcite @xcite . typical approaches in the literature involve reformatting recommendation tasks -such as item reranking or click-through rate (ctr) prediction -into natural language constructs to facilitate the fine-tuning of llms. however, we have observed that llms fine-tuned through this straightforward method, albeit enhancing accuracy in offline evaluations, frequently generate outputs with domainspecific formatting errors. these errors may manifest as repeated items in the top-k recommendations or the inclusion of items previously interacted with by the user. additionally, these llms exhibit a limited ability to adhere to diverse recommendationspecific instructions. this compromises their effectiveness as interactive agents in real-world recommender systems. a vivid example can be found in section 3.4. in this paper, we investigate the alignment of an llm for recommender systems. our objective extends beyond merely improving the recommendation accuracy of an original llm; we aim to significantly enhance controllability and reduce formatting errors. drawing inspiration from the reinforcement learning from human feedback (rlhf) framework @xcite , our methodology is structured into two phases: the supervised learning (sl) stage and the reinforcement learning (rl) stage. to inject domain-specific knowledge and foster recommendation-relevant control capabilities within the llm, we devise a series of fine-tuning tasks, including item recommendation, item search, category control, and category proportion control. these tasks often necessitate generating a list of items that not only meet users' instructions but also maintain high quality, despite the typically sparse ground-truth signals found in user behavior history. to tackle this issue, we pro-pose augmenting supervised labels with predictions from a traditional recommender model, such as sasrec @xcite . these augmented labels can help distill knowledge from the traditionally trained recommender model and meet the dynamic requirements of recommendation instructions. after the sl stage, the llm exhibits a significantly enhanced ability to follow recommendationrelated instructions, surpassing existing approaches that solely fine-tune the llm on item recommendation and search tasks. nevertheless, the sl stage's data generation process inherently provides only positive examples for each instruction. to address scenarios where the llm generates suboptimal responses, we introduce an rl stage with carefully crafted reward signals to further refine the llm's capacity to follow instructions. to the best of our knowledge, this is the first study that employs both sl and rl stages to align llms for controllable recommendation purposes. we conduct comprehensive experiments on two real-world datasets, steam and amazon movie, with results demonstrating that our method markedly improves the llm's ability to follow instructions while simultaneously reducing formatting errors. our major contributions are summarized as follows: ## related work in recent years, the remarkable natural language processing capabilities of llms have inspired researchers to leverage them for recommendation tasks @xcite . early implementations utilized language models as feature extractors, creating knowledge-aware recommendation embeddings exemplified by models such as u-bert @xcite and userbert @xcite . with the advent of generative models like gpt, the focus has shifted towards generative recommendation models that frame recommendations as natural language generation tasks @xcite . initially, adaptation of llms to recommendation scenarios relied heavily on techniques like prompt engineering @xcite and contextual learning @xcite . however, these llms often underperform compared to traditional recommendation models trained on task-specific data, prompting the neces-instruction i cp≈50% 2 : you are an expert recommender engine. you need to generate a recommendation list simultaneously considering user's preference and user's intention. here are the user's historical interactions: 'the blind swordsman: zatoichi takeshi kitano; tadanobu asano japan', 'buffy the vampire slayer -the complete seventh season', 'number ones', '24: season 3', 'madonna -the immaculate collection', 'spider-man 2', 'heavy metal 2000 vhs', 'mi-5, vol. 1', 'mi-5, vol. 2', 'beyond re animator vhs', and user's intention: in the recommendation list, the proportion of 'art house & international, by original language, chinese' items should be approximately 50%. you need to generate a recommendation list with 5 different items. ## conclusion in conclusion, our paper presents a new approach to tailor llms for interactive recommender systems, combining a supervised learning phase with innovative control tasks and a reinforcement learning stage with specialized reward signals. our method successfully meets the detailed demands of recommendation contexts, enhancing llms' performance. experimental results show our approach exceeds current llm-based systems in precision, controllability, and presentation, offering a significant step towards refined and reliable recommendation services.