B anner B ench: Benchmarking Vision Language Models for Multi-Ad Selection with Human Preferences (2025)

Abstract: Web banner advertisements, which are placed on websites to guide users to a targeted landing page (LP), are still often selected manually because human preferences are important in selecting which ads to deliver. To automate this process, we propose a new benchmark, BannerBench, to evaluate the human preference-driven banner selection process using vision-language models (VLMs). This benchmark assesses the degree of alignment with human preferences in two tasks: a ranking task and a best-choice task, both using sets of five images derived from a single LP. Our experiments show that VLMs are moderately correlated with human preferences on the ranking task. In the best-choice task, most VLMs perform close to chance level across various prompting strategies. These findings suggest that although VLMs have a basic understanding of human preferences, most of them struggle to pinpoint a single suitable option from many candidates.

## introduction online advertisements (ads) are designed to engage consumers by appealing to their interests and preferences. as advertising techniques have matured, users frequently encounter ads on the internet. in particular, banner ads are placed on websites to direct users to a targeted landing page (lp) when clicked. figure 1 illustrates a common workflow; multiple banners are created based on a single lp, and the task is to choose the most effective one for delivery. for effective ad delivery, it is crucial to select banners based on human preferences to encourage clicks. selecting banners that match human preferences is important for delivering effective banners and encouraging clicks. additionally, the banners should match the content of the lp to capture users' interest and encourage actual purchases or service sign-ups. therefore, selecting appropriate banners for delivery is a crucial step in maximizing ad effectiveness. at the same time, recent progress in automated ad creation @xcite @xcite @xcite , has made it easier to create many banners from a single lp. however, the selection process requires manual effort to choose the most appropriate banner. the large number of banners increases manual workload, highlighting the importance of developing fast and scalable methods for selecting the best banners to deliver. to facilitate the automation of this process using vision-language models (vlms), we propose a new benchmark, bannerbench, which evaluates the ability of vlms to identify the banner that best matches human preferences from a set of candidates. specifically, bannerbench includes two tasks. the first is a ranking task, where five banners created from a single lp are sorted based on human preference. the second is a best-choice task, where the model selects the single most preferred banner, reflecting situations when only a single ad slot is available. these tasks are designed to evaluate vlms' banner understanding capabilities from the perspective of human-aligned preference selection and their application in the ad delivery process. our experiments show that vlms perform reasonably well on the ranking task, and the results are moderately correlated with human preferences. on the best-choice task, their performance drops, and the agreement rates are close to chance level regardless of different prompting strategies. with fewer options, some models show improved performance on the best-choice task. these findings suggest that although vlms demonstrate a basic understanding of human preferences, most of them still struggle to make precise selections in special situations when required to select the single best option from many candidates. ## bannerbench we introduce bannerbench, a novel benchmark designed to evaluate the extent to which vlms can align with human preferences in the banner selection process. as shown in figure 2 , we first collect sets of banners that share the same promotional content. next, we annotate preference-based rankings for each set. finally, we formalize these annotated sets into benchmark tasks specifically designed to assess how well vlms align with human preferences in multiple banner selection scenarios. ## experimental results and discussions ranking. ## conclusion we proposed bannerbench, a novel benchmark to evaluate whether vlms can align with human preferences in the banner selection process. it covers two tasks: a ranking task and a best-choice task, using sets of five banners associated with a single lp. our experiments show that although vlms demonstrate a basic understanding of human preferences, most of them struggle to pinpoint a single suitable option. we hope this benchmark contributed to future research on developing and evaluating vlms in the real-world ad delivery process.