KAFA : Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models (2023)

Abstract: Image ad understanding is a crucial task with wide real-world applications. Although highly challenging with the involvement of diverse atypical scenes, real-world entities, and reasoning over scene-texts, how to interpret image ads is relatively under-explored, especially in the era of foundational vision-language models (VLMs) featuring impressive generalizability and adaptability. In this paper, we perform the first empirical study of image ad understanding through the lens of pre-trained VLMs. We benchmark and reveal practical challenges in adapting these VLMs to image ad understanding. We propose a simple feature adaptation strategy to effectively fuse multimodal information for image ads and further empower it with knowledge of real-world entities. We hope our study draws more attention to image ad understanding which is broadly relevant to the advertising industry.

## introduction as advertisements play an integral role in human society, image ad understanding has many real-world applications such as ad targeting @xcite , visual metaphor understanding @xcite and creative ad generation @xcite . it is also highly challenging due to several reasons, as exemplified in fig. 2 . first, image ads consist of diverse visual elements including non-photorealistic objects and atypical scenes synthesized creatively that are beyond common academic datasets. secondly, they involve knowledge of a large number of real-world entities such as brands and products where existing work @xcite struggles to cover. lastly, many adopt visual or multimodal rhetorics requiring reasoning over diverse visual elements including scene-texts, and sometimes even elude humans @xcite ˚work done in part during an internship at google. correspondence to zjia@eng.ucsd.edu. chilton, 2019). however, image ad understanding is relatively under-explored in the machine learning community, especially in the presence of recently developed foundational vision-language models (vlms) pre-trained using a tremendous number of image and text description data. the pre-trained vlms are shown to have great generalization capability, contain real-world knowledge (implicitly), and can be adapted to a wide range of downstream tasks in a data-efficient way @xcite . it is then natural to utilize vlms for image ad understanding. in this paper, we perform the first empirical study of adapting vlms to the task of decoding the overall messages delivered by image ads, which is usually formulated as visual question answering @xcite . specifically, we examine three popular pre-trained vlms that are alignmentbased and are publicly available, namely, clip @xcite , albef @xcite and lit @xcite . we examine zero-shot performance as well as adaptation strategies and reveal the practical challenges of applying vlms to image ads. we propose a simple feature adaptation strategy that effectively utilizes vlm features. we further propose to incorporate external brand knowledge (real-world entities) that brings a significant performance boost. our contributions are three-fold. first, we empirically find that the sheer scale of data & capacity of the model used in pretraining matters the most for the performance of image ad understanding, partly due to vlm's capability of storing real-world knowledge, which is not captured well by the commonly used metrics for comparing vlms. second, we reveal the practical challenges of adapting vlms for image ad understanding (i.e., overfitting to the limited training data & supervision signals and high computation burden of hard negative mining) and propose a simple solution (attention-based feature adaptation) that better leverages vlm features than previous adaptation strategies. lastly, we propose to leverage external knowledge for brand understanding that we have empirically shown to further enhance image ad understanding. together with the aforementioned adaptation strategy, we call our approach knowledge-augmented feature adaptation (kafa). ## related work image ad understanding learning to automatically interpret image ads was proposed by the pitt image ads dataset @xcite , where each ad is annotated by a caption that answers "what should i do according to the ad and why?" different from traditional image captioning, this task is highly non-trivial as discussed at the beginning of sec. 1. while prior methods utilize cultural connotations via external symbolic knowledge @xcite , capture relations between scene-texts and objects by @xcite , and leverage pre-trained language models to combine multimodel information @xcite , none have exploited vision-language models (vlms) and the knowledge of real-world entities (i.e., brands). besides the wide applications in the ad industry, later work hints that the study of image ads is relevant to much broader research topics @xcite . foundational alignment-based vlms a recent surge of collections of tremendous images paired with text descriptions @xcite enables alignment-based pretraining (i.e., contrastive learning) of foundational vlms that are efficient zero-shot or low-shot learners for downstream tasks. by learning to embed images and texts into a shared semantic space, they handle domain variations in an open-vocabulary manner (which involves real-world knowledge). among these are clip @xcite , align @xcite , lit @xcite and basic @xcite . another line of work further adopts masked language modeling, image captioning loss, and object-level alignment, e.g., albef @xcite , florence @xcite , coca @xcite and glip @xcite . transfer learning of vlms transfer learning of vlms has become popular with the zero-shot performance of clip in image classification tasks. a direct approach is to (partially) fine-tune the vlms with (optionally) additional neural networks tailored for downstream tasks, e.g., tap-c @xcite , cpt @xcite , kat @xcite and vl-adapter @xcite . another approach that bypasses the need of tuning the vlms is prompt learning. for instance, coop @xcite and cocoop @xcite only tune learnable inputs to the vlms. the third approach that further reduces memory and computation burden is feature adapters, where vlm features of the inputs are pre-computed before transfer learning. examples are clip-adapter @xcite , svl-adapter @xcite and attention-adapter @xcite . ## what really matters for pre-trained vlms in image ad understanding? the first insight of our empirical study is that the sheer size of data and the model used in pretraining is the key factor determining the performance of vlms for image ad understanding. to promote reproducibility, we evaluate three alignment-based vlms (i.e., clip, albef and lit) that are publicly accessible in a zero-shot manner on the pitt dataset @xcite , which formulates ad understanding as image-to-text retrieval. we adopt the official evaluation protocol, which asks the model to select one of the 3 correct messages conveyed by the image ad from a set of 15 candidates (including 12 wrong messages) for each of the 12805 test samples. specifically, given an alignment-based vlm, let us denote its encoders with normalized outputs as f i p¨q and f t p¨q for image and text branches, respectively. given an image x and the ground truth texts y, the vlm retrieves y from candidates cpxq according to the dot-product score f i pxq ¨ft pyq. we then measure the performance of the vlm with 3 metrics commonly used in the literature: accuracy (the percentage of images with any positive text retrieved with rank one), rank (how the top retrieved positive text is ranked averaged over all images), and the mean rank (the mean rank of the all positive texts averaged over all images). with the results reported in tab. 1, we have several findings. first, the more data used during the pretraining of a vlm, the better it generalizes to the image ad domain. for a comparison, clip has seen 400m image-text pairs, lit 100m, and albef 14m. second, the larger the capacity of a model, the better it understands image ads. we have evaluated different sizes of the clip model beyond the three sizes shown in tab. 1 and the trend keeps the same. third, commonly used metrics for comparing vlms, including zero-shot accuracy on the imagenet @xcite validation set (for which lit claims to outperform clip) and image-to-text retrieval precision on flickr30k @xcite (for which albef claims to outperform clip), do not reflect the performance of image ad understanding well. ## challenges in vlm adaptations to image ads and an intuitive solution with clip as the clear champion, we further study vlm adaptations for image ad understanding using the best clip model (vit-l/14@336px) as the backbone. we aim to enable better performance for image ad understanding by better adapting pretrained vlms to the image ad domain with the help of additional information such as scene-texts extracted from the image. ## improving image ad understanding with external knowledge to further improve image ad understanding, we propose to leverage external knowledge of real-world entities, namely product and brand information. the major focus of advertisements is to promote brand awareness @xcite . sometimes brand information is even a necessity to interpret ads correctly since it eliminates ambiguities and gives visual cues to the audiences (e.g., the ad for a cleaning product in fig. 2 ). it is then natural to empower feature adapters introduced previously with a brand understanding module that extracts brand information from images. here we present our training-free brand understanding module that considerably exploits vlms. ## conclusion in this paper, we study the adaptation of pretrained alignment-based vlms for the challenging image ad understanding task. we benchmark and reveal practical challenges in adapting vlms, propose a simple and intuitive (yet effective) strategy for feature adaptations, and further improve image ad understanding with external brand knowledge. while we mainly focus on the image-to-text retrieval task for its simplicity, we believe further studies can extend it to directly generating text descriptions given image ads or even generating image ads given the descriptions. we hope our study draws more attention to image ad understanding that are relevant to the advertising industry and provide insights for a broader machine learning community.