Understanding Advertisements with BERT (2020)

Abstract: We consider a task based on CVPR 2018 challenge dataset on advertisement (Ad) understanding. The task involves detecting the viewer’s interpretation of an Ad image captured as text. Recent results have shown that the embedded scene-text in the image holds a vital cue for this task. Motivated by this, we fine-tune the base BERT model for a sentence-pair classification task. Despite utilizing the scene-text as the only source of visual information, we could achieve a hit-or-miss accuracy of 84.95% on the challenge test data. To enable BERT to process other visual information, we append image captions to the scene-text. This achieves an accuracy of 89.69%, which is an improvement of 4.7%. This is the best reported result for this task.

## introduction the advertisement understanding challenge dataset of cvpr 2018 collected textual inputs from a set of viewers to capture their interpretations of ad images @xcite . the task is to rank the given valid and negatively sampled invalid interpretations of an image. initial approaches to the problem tried capturing the visual semantics with a combination of object proposal features and relationships of objects with common symbolism @xcite @xcite . @xcite have obtained a significant improvement in performance by utilizing the text embedded in the image (termed as the scene-text) as another channel of information. these approaches do not evaluate the validity of an interpretation by using attention to associate the words and phrases of the interpretation to fragments of textual and visual cues in the image. for example, in the ad of the car company in figure 1 , the words and phrases from a viewer's input 'i should buy this car because it would add some excitement to my life' can be associated with the object 'car' in the image and the phrase 'add spark to life' in the scene-text. to capture these mappings, we need a model that can simultaneously pay attention to the image and the interpretations at various levels of granularity. the recently proposed bert pre-trained language model @xcite has provided excellent performance on several nlp tasks. the underlying attention-based transformer architecture @xcite allows bert to capture contextual representations. we leverage the pretrained base bert model to capture a contextual representation of the viewer's interpretation with respect to the visual and textual cues in the image. one of the challenges we face is to provide information on visual cues to the bert model. we overcome this challenge by extracting densecap captions(densecaps) @xcite to provide textual information about the image objects, their properties, and interactions. this is motivated by the approaches of visual question answering (vqa) @xcite , question generation @xcite , which talk about leveraging more abstract text or concept-level information instead of pixellevel information of an image. we fine-tune bert for the sentence-pair classifi-cation task, where the scene-text and the densecaps form the first sentence, and the viewer's interpretation forms the second sentence. with this approach, we achieve an accuracy of 89.69% and recall@3 of 2.411, which is by far the best reported result for this task. ## proposed approach to abstract concepts from the pixel stream, we extract densecaps foot_0 @xcite of the image. we use google vision api 2 to extract scenetext from the image. we append the densecaps to the extracted scene-text to form a composite textual signal. this text is paired with an arp to form sentence pairs, that are served as inputs to bert, as shown in figure 2 . the average token length of sentence-pairs is 147. for the samples for which the sentence-pair token length goes beyond the maximum allowed length (512 tokens) of the base bert model, we truncate the length of the composite textual signal of the image. to avoid a significant information loss due to the truncation, we arrange the densecaps in decreasing order of their confidence score. bert @xcite has been pre-trained to use the [cls] pin output for sentence-pair classification. hence, we use the [cls] pin output and fine-tune (bert ft st+c) (ref. ## results and analysis in this section, we compare the performance of the models as presented in ## conclusion and future work the scene-text holds vital information and can be used to achieve good accuracy on this task. syntax matches play a vital role in achieving the accuracy, but are not entirely the reason behind it. although the conversion of visual cues to captions cause a loss of information, the addition of scene-text mitigates most of the loss. using attention to associate the arps with the textual and visual cues is helping the task. better emotion, scene, scene-text, object detection and captions might lead to further improvement of performance.