Generative Explore-Exploit: Training-free Optimization of Generative Recommender Systems using LLM Optimizers (2024)

Abstract: Recommender systems are widely used to suggest engaging content, and Large Language Models (LLMs) have given rise to generative recommenders. Such systems can directly generate items, including for open-set tasks like question suggestion. While the world knowledge of LLMs enables good recommendations, improving the generated content through user feedback is challenging as continuously fine-tuning LLMs is prohibitively expensive. We present a training-free approach for optimizing generative recommenders by connecting user feedback loops to LLM-based optimizers. We propose a generative explore-exploit method that can not only exploit generated items with known high engagement, but also actively explore and discover hidden population preferences to improve recommendation quality. We evaluate our approach on question generation in two domains (e-commerce and general knowledge), and model user feedback with Click Through Rate (CTR). Experiments show our LLM-based explore-exploit approach can iteratively improve recommendations and consistently increase CTR. Ablation analysis shows that generative exploration is key to learning user preferences, avoiding the pitfalls of greedy exploit-only approaches. A human evaluation strongly supports our quantitative findings.

## introduction recommender systems are widely used for various applications, including suggesting items in catalogs (music, books, videos) @xcite ), e-commerce @xcite , and search results @xcite . a core part of a recommender system is improving the relevance of recommended items based on the user's preferences. such preferences can be either explicit (e.g. user provided preferences about a brand in * work done during an internship at amazon. e-commerce queries), or implicit based on engagement or click through rate (ctr) of provided recommendations. furthermore, the more users interact with a recommender system, the more the recommended items are optimized to match such explicit or implicit user preferences. traditional recommender systems use optimization approaches such as (deep) collaborative filtering @xcite , matrix factorization @xcite , and reinforcement learning @xcite . a common thread is that they are all applied to a static set of items. however, the increasing capabilities of large language models (llms) has led to the development of generative recommender systems @xcite . generative recommenders can be used to directly generate items or text content for recommendation. the immense world knowledge of llms makes them excellent at certain generative recommendation tasks such as question or query suggestion @xcite . unlike traditional recommender systems, they do not choose from a fixed item set, and the range of valid items that could be generated is vast. we investigate how llm-based generative recommenders can be iteratively optimized based on implicit user feedback, e.g. by following ctr signals (cf. figure 1 ). such recommenders needs to support very large numbers of input contexts (e.g. items or queries). however, fine-tuning them to improve recommendations is prohibitively costly given their size. our approach is training free: improving item recommendations does not need finetuning; instead we propose novel methods to synthesize context-specific implicit engagement signals (e.g. ctr) as part of the llm input. we also propose a generative explore-exploit mechanism to generate and evaluate new candidate items. our work is applicable to numerous applications such as search @xcite , question answering @xcite , and query/question suggestion features in information seeking systems @xcite . to demonstrate our approach, we focus on the specific task of question generation (qg) @xcite . this involves suggesting engaging (e.g. high ctr) questions to help users explore a topic. experiments show that by leveraging both generative exploration and exploitation, our approach can adapt its recommendations to match the preferences of a population whose preferences are not directly observable. we show that a feedback loop based on ctr can successfully guide the llm to explore new questions types and topics, and exploit previously generated ones. finally, we show that our approach converges faster to questions that meet user needs, compared to baselines without ctr signals. our work makes the following contributions: prior work has employed llms within recommender systems for ctr prediction, aiming to estimate the probability of users engaging with items @xcite . similar to us, @xcite designed specific zero-shot and fewshot prompts to assess how llms predict recommendation ratings. to our knowledge, we are first to use llms to simultaneously generate recommendations and optimize ctr in the same framework. we do this by building upon the findings of @xcite , by leveraging llms ability to understand and make use of the connection between textual input (questions in this work) and its performance on a target task. our approach learns to generate questions that can achieve higher ctr. question generation. qg is important in various nlp applications @xcite such as reading comprehension @xcite , conversational recommendation @xcite . we specifically focus question suggestions, i.e., questions that users might want to ask a system. a notable use case is people also ask (paa) questions in web search @xcite , which suggests users potential next questions they can ask to further explore a topic. unlike paa where questions are typically static, we generate novel and more engaging suggestions based on engagement signals from users interactions (clicks). this helps create questions aligned with the interests of the user population. user click simulation. several recommender systems studies have developed ranking models to simulate user click behavior in the web search domain @xcite . recently, @xcite prompted llms with personas and rules to simulate user behavior within conversational recommender systems. inspired by their work, we design a user simulator that is representative of real-world user cohorts with varying personas and needs, and use llms to evaluate the question relevance to these personas. ## problem definition given a population of users @xmath0, . . . , u n }, a user u may search for multiple topics t 1 , . . . , t k (e.g. "biology", "smartphones"). the task is to generate a fixed set of questions to be suggested for a topic t, referred as item pool (ip), such that the questions are likely to be of interest to as many users as possible in u that have searched for t. actual user interests about t are hidden and can only be observed through interactions (e.g. clicks) with the generated items. using a sufficiently large llm with reasonable instruction following capabilities, we want to generate ip for t, which is a list of n questions that maximize click through rate (ctr) from all users @xmath1. ## generative recommender approach our training-free generative recommender approach is outlined in figure 2 . at a high level, our proposed approach iteratively improves llmgenerated outputs as follows. an llm is used to generate the initial candidate items for a given task, and the click-through rates (ctr) of these items are then measured to gauge user engagement and preference. based on these observed ctrs, a prompt is constructed to encapsulate this performance data. this prompt serves as a basis for applying a dual approach of generative exploration and exploitation using an llm optimizer via in-context learning. the optimizer refines the generated items by bal-ancing the exploration of new possibilities and the exploitation of known high-performing elements, to enhance the quality and engagement of the generated content. our method is suitable for tasks where many valid generations are possible, like summarization and question generation. our approach is iterative, with user interaction data collected in each round i in order to refine the ip i on a given topic t. more specifically, in each iteration: (i) the n worst items are dropped from ip, (ii) n new items are generated and added to ip, and (iii) the performance of items in the updated ip is observed via an interaction feedback loop. items in the i-th iteration are indicated by ip i . ## user click simulator developing and evaluating our approach requires user engagement data, which is expensive to collect, and involves complex privacy concerns. to enable fast offline experimentation, we simulate feedback by modeling user preferences and their click behavior on suggested questions using llms. we represent users with different interests and goals via specific prompts (see ## experimental setup here, we discuss the experimental setup, namely the domains on which we assess our proposed approach and competitors. furthermore, we explain in detail the personas used for experimentation, which aim at mimicking real user cohorts. finally, we will define the evaluation metrics used. domain e-commerce general knowledge source product category wikipedia article sample topics spray bottles stoicism home bar furniture tabata training cookware sets friedrich nietzsche lighting & ceiling fans chernobyl disaster tv antennas artificial superintelligence personas price discussion-focused quality history-focused brand reputation event-focused features & functionality person-focused ethical considerations location-focused ## results we present the experimental results for the two components of our approach: (i) item relevance scoring, and (ii) recommendation performance. ## considerations for online deployment in this work we used a simulator to facilitate offline development. this approach allowed us to develop and test various algorithms without needing to directly involve real users or incur the associated costs and risks. while having the simulator as a stand-in for real user feedback served to significantly expedite the development process, transitioning to real-world deployment presents a set of new challenges and requires modifications. the core requirement is to replace the simulatorgenerated feedback with actual user engagement. this means the recommendations generated by the system must be served to real users, and their interactions (such as ctr and dwell time) need to be collected and stored for each input context. as real engagement is noisy and sparse, implementing efficient data pipelines is key. caching might be needed to store and serve the item pool for each context in order to minimize latency. furthermore, the optimization iterations may need to run at fixed intervals, e.g. after every n user engagements or based on a predetermined time schedule. this periodic and continuous optimization ensures that the model evolves in response to the latest user data. finally, moving from the cohort-based personalization explored here (where recommendations rely on generalized user segments) to user-level personalization will require further changes. tailoring recommendations to individuals will require collecting more granular data. ## conclusion we proposed a novel method to improve llmbased generative recommender systems by iteratively refining recommendations based on implicit feedback loops from ctr signals. we additionally defined a user simulator to effectively simulate user interactions with such recommended items. our novel generative explore-exploit approach does not require any fine-tuning, and only relies on an llm optimizer using in-context learning by synthesizing observed ctr performance data and incorporating them into the prompt. experiments with our approach show that while leveraging historical ctr data is crucial to exploit known engagement patterns, the inclusion of a generative explore phase is equally important for discovering user preferences. evaluations on the task of question generation across two domains (e-commerce and world knowledge) show that our proposed generative recommender approach is able to generate questions that are highly relevant to its user population in just a few iterations, which in turn results in higher engagement as measured through ctr. the generative explore-exploit method is is particularly suitable for tasks where there are many potentially valid suggestions that can be generated. while we studied question generation in this paper, it can applied to a range of tasks such as summarization @xcite , personalized headline generations @xcite , and followup question suggestion @xcite . by avoiding the use of reward models or fine-tuning, our approach can effectively scale to scenarios with billions of items, while also being able to support user-level personalization.