Jump Starting Bandits with LLM -Generated Prior Knowledge (2024)

Abstract: We present substantial evidence demonstrating the benefits of integrating Large Language Models (LLMs) with a Contextual Multi-Armed Bandit framework. Contextual bandits have been widely used in recommendation systems to generate personalized suggestions based on user-specific contexts. We show that LLMs, pre-trained on extensive corpora rich in human knowledge and preferences, can simulate human behaviours well enough to jump-start contextual multi-armed bandits to reduce online learning regret. We propose an initialization algorithm for contextual bandits by prompting LLMs to produce a pre-training dataset of approximate human preferences for the bandit. This significantly reduces online learning regret and data-gathering costs for training such models. Our approach is validated empirically through two sets of experiments with different bandit setups: one which utilizes LLMs to serve as an oracle and a real-world experiment utilizing data from a conjoint survey experiment.

## introduction when users encounter content that caters to their needs and preferences, they are more inclined to engage, e.g., watch a movie @xcite or click on a news article @xcite . but how should a practitioner learn to personalize a campaign's content? would a user be more likely to respond if the content was presented in a formal or informal style @xcite ? or if it included a celebrity endorsement @xcite ? studies in behavioural economics @xcite proved that the answers can be heavily context-dependent @xcite . contextual multi-armed bandits (cbs) @xcite were invented to address this problem in the online learning setting-where an agent with no prior knowledge of user preferences is presented with users in a sequence, chooses a piece of content * work performed during an internship at borealis ai. to show the users based on the users' and the content's features, and updates itself based on feedback essentially instantaneously. the critical question for cbs is how to balance exploration (gathering information about users' preferences) and exploitation (utilizing the information it has gathered so far). while these agents are known to exhibit good asymptotic performance, their initial choices are essentially random. to improve performance for the first users in a campaign, researchers have focused on warm starting a cb @xcite using detailed records of users' past behaviours and preferences in similar campaigns. however, collecting such datasets poses significant challenges due to resource demands, data diversity requirements, and the need to comply with privacy regulations. in this paper, we show that large language models (llms) can solve this conundrum by jump starting contextual bandits using approximate human preference information already captured in them. this allows practitioners to automatically design and test context-dependent messaging with significantly lower upfront costs. prior work has shown conflicting evidence about whether llms faithfully simulate human behaviours. argyle et al. @xcite and simmons @xcite show llms can simulate human behaviour and preferences in certain circumstances, while santurkar et al. @xcite find that llm-generated response distributions may not match the true population distribution. our key insight is that even if llm-generated reward distributions do not perfectly match human preferences, they still provide a better than random baseline than that which cold-started cbs face. our contributions. first, we introduce a novel approach, contextual bandits with llm initialization (cbli), that leverages the power of llms to generate synthetic reward distributions for pretraining cbs for personalization in algorithm 1. second, we empirically demonstrate the effectiveness of our proposed method to pre-train a cb in two settings where natural language plays a critical role in the content shown to users. the first setting involves a standard contextual bandit whose arms represent the styles of marketing communication soliciting charity donations from simulated users. the second experiment uses a sleeping bandit setup with real-world human preferences. cbli achieves 14-17% and 19-20% reduction in early regret, respectively. furthermore, we show that even when certain privacy-sensitive attributes are withheld, cbli still achieves an 14.8% reduction in early regret in the second setting. finally, testing contextual bandit algorithms in real-world settings usually requires collecting a large amount of randomized log data, which can be costly and difficult while maintaining data diversity and privacy requirements. as an additional contribution, we introduce, to the ml and nlp community, the usage of conjoint survey experimental data @xcite as a way to benchmark bandit algorithms. conjoint experiment data collected in social science literature provides a substantial source of real-world data where users were presented with several potential choices and asked to record their preferences. coupled with the demographic information that is typically available from such experiments from which contexts can be constructed, they are an ideal candidate for testing contextual bandit algorithms. ## related work with the advent of web scale content, recommender systems that personalized content to specific users became an important part of navigating the internet. recently, deep neural networks have become more important to these systems @xcite , and the need to train these systems online and at scale has been a prominent concern @xcite . one such online training technique, multi-armed bandits, has proven especially popular, with a plethora of algorithms specialized to specific settings (see @xcite for an introduction). most relevant to this work are contextual bandits @xcite @xcite , which introduce side information to determine the optimal arm. in particular, zhang et al. @xcite consider warm starting cbs utilizing fully supervised, labelled data collected from human experts. while to the best of our knowledge, conjoint survey experiments have not be used in the evaluation of contextual bandits, conjoint analysis-the underlying statistical technique-has been used for feature selection in training cbs @xcite . outside of cbs, researchers have used ml techniques to improve the efficiency of conjoint analysis @xcite @xcite . recently, researchers have attempted to integrate llms into social science studies, specifically utilizing llms to simulate human behaviours in social psychology experiments @xcite and predict their opinions @xcite . additionally, a growing body of research has focused on evaluating llm performance and biases through survey questions. prior studies have examined how llms respond to multiple-choice questions from public opinion polls, revealing insights into their alignment with human responses and potential biases @xcite @xcite . santurkar et al. @xcite focus on diverse demographic groups across the u.s., while durmus et al. @xcite focus on non-american demographics, both finding that llms' responses to surveys show discrepancies with the collected answers from representative samples. furthermore, dominguez-olmedo et al. @xcite show that models' responses are sensitive to ordering and labelling biases, and they do not contain the statistical signals typically found in human populations. our work in this setting suggests that despite potential biases and disparities in llm-generated responses, they provide a good starting point for training contextual multi-armed bandits. ## preliminaries multi-armed bandits. in a multi-armed bandit problem, an agent, at each step @xmath0, is presented with a set of k possible arms and must choose an arm k t . after the agent chooses, they receive a reward r t,kt drawn from a reward distribution r(k t ), which is initially unknown. we focus on the stochastic setting, where each r t,k is sampled independently from r(k). the goal of the agent is to maximize the total reward t @xmath1,kt . to achieve this, the agent needs to balance the tradeoff between exploration (i.e., trying different arms to gather more information about their reward distribution) and exploitation (i.e., choosing among the arms that appear to offer the highest reward based on the collected information so far). ## method recent advances in large language models have shown remarkable progress in understanding and interpreting human expression @xcite . llms trained on extensive corpora rich in human knowledge preserve the capacity to perform well on a diverse array of tasks, even those dependent on human behaviour and preferences @xcite . in the context of personalization, llms offer two key advantages. first, they can be utilized to automatically design user-specific content in large volumes. second, llms can be used to simulate human interactions and predict their preferences, effectively serving as a proxy for collecting a dataset of users' interactions. we focus on the latter in our work, and provide a framework to utilize the power of llms to generate extensive artificial user interactions to later train our models. ## experiments we validate the effectiveness of our proposed approach through two sets of experiments. 1 in both sets of experiments, we adapt the linucb algorithm @xcite to train our bandit. the first set of experiments is based on synthetic data, and the second set uses a real-world dataset derived from a choicebased conjoint analysis @xcite . experimental details are in appendix c. ## conclusion we introduced a novel framework, contextual bandits with llm initialization (cbli), for jumpstarting contextual bandits with large language models (llms), aimed at enhancing early decisionmaking in personalized systems through pretraining on synthetic data. our experiments demonstrated the potential of llms to generate a dataset that, while not perfectly mirroring true human preferences, provides a significantly better starting point than the traditional cold-start scenario in bandit models. through rigorous testing in two different bandit settings, we have shown that cbli can reduce early regret by significant margins, indicating its practical effectiveness and robustness.