Don’t Buy it! Reassessing the Ad Understanding Abilities of Contrastive Multimodal Models (2024)

Abstract: Image-based advertisements are complex multimodal stimuli that often contain unusual visual elements and figurative language. Previous research on automatic ad understanding has reported impressive zero-shot accuracy of contrastive vision-and-language models (VLMs) on an ad-explanation retrieval task. Here, we examine the original task setup and show that contrastive VLMs can solve it by exploiting grounding heuristics. To control for this confound, we introduce TRADE, a new evaluation test set with adversarial grounded explanations. While these explanations look implausible to humans, we show that they “fool” four different contrastive VLMs. Our findings highlight the need for an improved operationalisation of automatic ad understanding that truly evaluates VLMs’ multimodal reasoning abilities. We make our code and TRADE available at https://github.com/dmg-illc/trade.

## introduction image-based advertisement is not only a crucial component of marketing campaigns, but also an interesting example of sophisticated multimodal communication. ads often feature unusual visual elements (e.g., objects that are non-photorealistic, outside of their usual context, atypical, etc.) or examples of figurative language (e.g., metaphors, allegories, play on words, etc.) designed to make a long-lasting impression on the viewer. figure 1 provides an example of an ad with a non-photorealistic object (a whale made of wires) that, as the text suggests, is used to convey a complex metaphorical message about the product (i.e., a wireless device). these elaborate uses of images and text make automatic ad understanding a challenging task requiring multiple non-trivial abilities, e.g., object detection, scene-text extraction, figurative language understanding, and complex image-text integration. ad understanding was first proposed as a deep-learning task by @xcite , who introduced the pitt ads dataset, consisting of imagebased ads along with explanations capturing their underlying message (e.g., i should purchase this stereo system because wireless is less messy). this dataset was then used in a retrieval-based challenge requiring to identify a plausible explanation for an ad within a set of possible candidates. 1 early work on this task has employed ensemble predictors @xcite and graph neural networks @xcite that were designed and trained ad hoc. more recently, the development of large vision-andlanguage models (vlms) pretrained with imagetext matching (itm) objectives has opened the possibility of performing the task in zero-shot, i.e. by using an off-the-shelf model instead of training one from scratch. following this approach, @xcite tested multiple vlms (albef, @xcite @xcite on the task by computing imagetext alignment scores between ads and their possible explanations. they observed an excellent zeroshot performance for all models, documenting an accuracy of 95.2% for clip. while the results reported by @xcite seem to suggest that the tested models developed the reasoning abilities necessary to succeed at ad understanding, we note that this conclusion is in contrast with a great deal of existing work. extensive research investigating whether vlms develop reasoning skills as a result of their contrastive itm pretraining has exposed several weaknesses of these models. they have been shown to be limited in their abilities to identify noun mismatches in image captions @xcite , reason compositionally @xcite , capture spatial relations @xcite , understand verbs (in- brand name brand name wires are under extinction. dvd theater [brand name]. now wireless. i should get a [wbn] bike because they have been around for a while 1. i should buy a [brand name] because i will not need the wires 2. i should use wireless instead of wires because it will help reduce waste in the world 3. i should not eat meat because it supports the killing of animals 4. i should not wear fur because it kills animals 5. i should buy ice cream because it's on sale for the company being in business for 16 years 6. i should fund [wbn] because we should take back control 7. i should purchase this stereo system because wireless is less messy 8. i should use wires because they are like whales risking extinction 1. i should use wireless instead of wires because it will help reduce waste in the world 2. i should use caution when throwing away cables because whales risk extinction 3. ## a closer look at the evaluation setup the pitt ads dataset 2 by @xcite consists of 64832 ads, each annotated with 3 explanations in english written by 3 different expert annotators. these explanations (in the form i should ⟨action⟩ because ⟨reason⟩) aim at capturing the persuasive message behind the ads. while explanations may be subjective, the intuition behind the image-to-text retrieval task proposed along with 2 https://people.cs.pitt.edu/~kovashka/ads/ the dataset is that a model which can understand ads should be able to match them with a plausible explanation. specifically, each ad is paired with 15 messages, 3 positives corresponding to the annotations for that ad and 12 negatives randomly sampled from annotations for different ads. figure 1 provides an overview of the task setup. previous work has hinted at possible limitations of the evaluation setup. @xcite observed a significant overlap between the text present in the ad and the matching explanations and noticed this was "a major discriminating factor" that their finetuned bert model could exploit. similarly, @xcite pointed out that the candidate set lacks "hard negatives" and proposed to increase the set size, but could not provide a solution ensuring the negatives were actually hard. we conduct a quantitative analysis on the original evaluation setup to uncover potential shortcuts that vlms may be exploiting to solve the task. we hypothesise that the models may take advantage of two factors that do not necessarily reflect ad understanding: simple relationships between (1) the candidate explanations and the text present in the ad (i.e., the degree of textual grounding of the explanations) and (2) the entities mentioned in the explanations and those depicted in the image (their degree of visual grounding). to test our hypotheses, we define several visual-and textualgrounding scores and check whether they correlate with the clip-based alignment score used by @xcite to retrieve the ad explanations. 3 textual-grounding scores are computed between candidate explanations and the text extracted from the ad with optical character recognition (ocr). we calculate (1) text overlap as the proportion of content-word lemmas from the explanation that are also present in the ocr-extracted text, and (2) text similarity as the cosine similarity between a sentence-level embedding of the explanation and that of the ocr-extracted text, derived with mpnet @xcite . visual-grounding scores include (1) object mention as the proportion of nouns in the candidate explanation that are present in a set of objects we automatically extracted from the image by a resnet50 model @xcite , and (2) caption similarity as the cosine similarity between the sentencelevel embedding of the candidate explanation and the embedding of the ad caption we obtained with blip-2 @xcite . our motivation for examining both detected objects and generated captions is driven by the observation that they capture complementary information. more specifically, while detected objects are not mediated by language models, they may often incorporate non-salient objects that people would unlikely mention when describing a picture or contain lexical choices that differ from the human ones. on the other hand, generated captions refer to objects in a more human-like way but, at the same time, may contain hallucinations due to linguistic priors. we compute the grounding scores and clip's alignment score for the test split of the pitt ads dataset, consisting of 12805 samples. as hypothesised, we observe a positive correlation between all our grounding scores and clip's alignment score. all the spearman's correlation coefficients are significant (p ≪ 0.001) and range from 0.14 and 0.61 (see appendix a for details). in addition, as shown in overall, these results indicate that the original evaluation setup is flawed and that the outstanding zero-shot performance obtained by vlms on the retrieval task may be due to simple image-text alignment. ## trade: a new adversarial test set to test the extent to which vlms capture elaborate visuo-linguistic relationships present in imagebased ads beyond image-text alignment, we develop trade (truly adversarial ad understanding evaluation), a new diagnostic test set with adversarial negative explanations. trade consists of 300 randomly selected ads from the pitt ads dataset, each associated with 3 options (1 positive and 2 negatives). concretely, for each of these ads, we randomly select one valid explanation from the available annotations and create two adversarial negative explanations-see figures 1 and 2 for examples (more examples in appendix c). the adversarial explanations were created by 4 expert annotators who were instructed to do their best to come up with non-plausible explanations that nevertheless mention objects and fragments of text present in the image. annotators were also asked to approximately match the length of the positive explanation when writing these adversarial sentences. appendix b contains more details about the creation of the adversarial negatives, including the guidelines provided to the annotators. we validate trade in two ways. first, we compute the textual-and visual-grounding scores introduced in section 2. this shows that in trade the gap between positive and negative explanations is radically reduced compared to the original setup, as can be seen in trade and trade-control are publicly available at https://github.com/dmg-illc/trade under a creative commons attribution 4.0 international (cc-by) license. ## experiments we use trade to test four contrastive pretrained vlms zero-shot. three of these models (clip, @xcite @xcite have been shown to achieve high zero-shot performance on the original task setup @xcite . here we challenge them with trade and consider an additional model (align, jia et al. 2021). ## conclusions our work exposes key limitations of the evaluation setup that was previously used to benchmark vlm's ad understanding abilities. we introduce a new adversarial test set (trade) that controls for the identified issues and show that, while humans excel, contrastive vlms perform at chance level on trade. this result has the following implications. first, it shows that, when processing imagebased ads, contrastive vlms are strongly biased towards textually and visually grounded explanations, regardless of their plausibility. this is in agreement with previous work @xcite @xcite @xcite and points to the need to use caution when interpreting models' zero-shot accuracy on "naturalistic" (i.e., non-adversarial) setups as proof that they develop sophisticated reasoning abilities via pretraining.