ICL -Bandit: Relevance Labeling in Advertisement Recommendation Systems via LLM (2025)

Abstract: Measuring the relevance between user queries and advertisements is a critical task for advertisement (ad) recommendation systems, such as Microsoft Bing Ads and Google Ads. Traditionally, this requires expert data labeling, which is both costly and time-consuming. Recent advances have explored using Large Language Models (LLMs) for labeling, but these models often lack domain-specific knowledge. In-context learning (ICL), which involves providing a few demonstrations, is a common practice to enhance LLM performance on domain-specific tasks. However, retrieving high-quality demonstrations in a vast exploration space remains challenging. In this paper, we introduce ICL-Bandit, a practical and effective approach that leverages ICL to enhance the query-ad relevance labeling capabilities of LLMs. We develop a novel bandit learning method to identify and provide superior demonstrations for ICL, thereby improving labeling performance. Experimental results demonstrate that ICL-Bandit achieves state-of-the-art performance compared to existing methods. Additionally, ICL-Bandit has been deployed in Company X, that serves billions of users worldwide, confirming its robustness and effectiveness.

## introduction in advertisement (ad) recommendation systems such as microsoft bing ads and google ads, highquality labeled data is of critical importance for training ad recommendation models, especially labeling the relevance between user query text and ad description text, as discussed in @xcite @xcite . the traditional approach is human labeling which is costly and inefficient. this is particularly challenging given the huge amount of data to be labelled, and labeling such relevance between user query and ad * this work was done when chiming was an intern at microsoft. advertisement: "remote patient monitoring systems -continuous care for heart failure patients" this query-ad pair is labeled as relevant since remote patient monitoring systems provide continuous care and real-time health data, enabling proactive management of heart failure, which is essential for reducing hospital readmission rates, and such manual labeling requires domain knowledge. recent advances in large language models (llms) have shown that llms are highly aligned with human judgments and even surpass human performance in certain tasks @xcite , such as topic identification and twitter relevance for political issues @xcite , general question-answering data generation @xcite , and instruction data generation @xcite . however, the lack of domain knowledge limits the performance of llm in the query-ad relevance labeling task. to address this challenge, many approaches employed in-context learning (icl) to incorporate domain-specific knowledge as extra context in the llm's prompt @xcite . besides, it is well known that the effectiveness of icl heavily depends on the quality of the provided demonstrations, which has motivated many works to explore effective demonstration retrieval methods for icl, such as @xcite @xcite , all of these methods aim to retrieve better examples from annotated training sets to enhance llms' domain knowledge. previous work on demonstration retrieval falls into two categories. one category involves off-theshelf retrievers like bm25 @xcite or knn @xcite , which can retrieve textually or semantically similar demonstrations. the other category focuses on training task-specific retrievers with positive and negative demonstrations. notable examples include rubin et al. @xcite , shi et al. @xcite , and xiaonan et al. @xcite , who leverage llm feedback (compare the labels generated by llm with the ground-truth labels, using them as the training signal) to train these retrievers via supervised or contrastive learning. however, the vast combination space of different demonstrations and queries poses a challenge. randomly sampling demonstrations to collect the llm's feedback may lead to large parts of "less useful" examples. some methods, like zhang et al. @xcite and mingkai et al. @xcite , employ reinforcement learning to actively sample demonstrations and obtain llm feedback. but these methods are limited in considering only a fixed number of candidate demonstrations, which reduces the action space for policy training. to overcome the challenges addressed above in sota methods, at first, we frame demonstration retrieval problem as a multi-armed bandit (mab) problem @xcite , and bandit algorithms solving mab problems @xcite have demonstrated excellent performance in addressing exploration and exploitation dilemma when dealing with largescale search spaces. this allows us to design effective exploration techniques for sampling demonstrations and obtaining llm feedback during retriever training. then, we propose a novel in-context learning (icl) algorithm, called icl-bandit, which leverages a stochastic bandit algorithm to empower icl at scale with diverse demonstration pools. the objective of icl-bandit is to retrieve demonstrations and maximize cumulative positive llm feedback over a series of retrievals. figure 1 shows a comparison on the example query-ad pair with demonstrations retrieved with knn and our icl-bandit, respectively. our contributions can be summarized as follows: ## icl-bandit with large and varied demonstration pool in this section, we introduce icl-bandit, a stochastic bandit algorithm designed to efficiently retrieve demonstrations and collect llm feedback during retriever training. the task is to precisely label query-ad relevance with icl, and our goal is to train a demonstration retriever which retrieves good demonstrations for icl. when training the retriever using llm feedback (we compare the output labels generated by the llm with the ground-truth labels, employing them as the reward signal), the key lies in how to effectively retrieve demonstrations from a large and diverse pool during the training process. addressing the exploration (searching for diverse and potentially informative demonstrations) versus exploitation (retrieving high-reward demonstrations) balance is pivotal. to tackle this challenge, we first formulate the demonstration retrieval task as a bandit problem. ## conclusion in this paper, we leverage llms to automate queryad relevance labeling for improved ad recommendation. to address the lack of domain-specific knowledge in llms, we adopt in-context learning (icl) and propose icl-bandit, a stochastic bandit algorithm for retrieving high-quality demonstrations and collecting llm feedback to train a retriever. our approach outperforms existing retrieval methods and has been successfully deployed in microsoft's ad recommendation system, delivering significant cost savings and strong real-world effectiveness.