Ad Lingua: Text Classification Improves Symbolism Prediction in Image Advertisements (2020)

Abstract: Understanding image advertisements is a challenging task, often requiring non-literal interpretation. We argue that standard image-based predictions are insufficient for symbolism prediction. Following the intuition that texts and images are complementary in advertising, we introduce a multimodal ensemble of a state of the art image-based classifier, a classifier based on an object detection architecture, and a fine-tuned language model applied to texts extracted from ads by OCR. The resulting system establishes a new state of the art in symbolism prediction.

## introduction visual advertisement, both image and video, can efficiently convey persuasive messages to potential customers. much of the power of visual advertising comes from multiple ways to interact with the user, embedding messages in both literal and symbolic forms. one of the most complex tasks in ad analysis is symbol interpretation, a much harder problem than, for instance, object detection. for example, in the lee denim jeans brand ad shown in fig. 1 a human figure is depicted with a lion's head, which might arguably symbolize "courage", "strong character", etc. in real life, lions are rarely seen wearing denim jeans, so analyzing this scene with a machine learning model trained on regular photographs would be quite challenging at least in terms of object detection. moreover, establishing a direct association between the (possibly detected) lion and human "courage" is a hard problem by itself. both factors make symbol interpretation difficult. but understanding ads computationally has important applications: decoded messages can help improve ad targeting, tools for message understanding can lead to better descriptions of visual content, and they can also inform the users how exactly they are being persuaded. @xcite present a crowdsourced dataset of advertisements, including images and videos. they introduced several annotation tasks: topic detection, sentiment detection, symbol recognition, strategy analysis, slogan annotation, and question answering for texts related to the ads' messages and motivation. in this work we focus on the image-based multi-label classification task for symbols. in this problem, each annotated image in the dataset has several bounding boxes and textual description that can be mapped to a limited number of categories of symbols. related work has mostly concentrated on this dataset, and it often combines text and images, but problem settings vary widely. in particular, the ad-vise framework embeds text and images in a joint embedding space in the context of choosing human statements describing the ad's message @xcite . @xcite introduce various feature extraction methods to capture relations between ad images and text. @xcite use a co-attention mechanism to align objects and symbols. @xcite suggest a data fusion approach to combine text and images for topic classification. multi-modal image+text prediction is an important research field usually concerned with other tasks; for surveys of this field see, e.g., @xcite . input image imagebased prediction object detectionbased prediction ocr text-based prediction linear ensemble figure 2: our approach. tesseract (smith, 2007) bite the boredom, tct am 5 bite-sized treats with a crunchy outside pur rco e eb pree ee rar ed oe 2 ; rn good east + tesseract (kopeykina and savchenko, 2019) bite dale boredom, unleash the fun! bite sized treats rh pe tah outside and delicious aay filled sita bo ii te sti) me kfc macaroni and cheese. sl ppps m)/7-) uc so good ## data we have used advertisement images from the dataset by @xcite , annotated with the help of mturkers. in particular, we have used symbolism annotations. in media studies, certain content that stands for a conceptual symbol is called a "signifier" (in the case of this dataset, signifiers are marked with bounding boxes on the image), and the symbols themselves are the "signified" @xcite . @xcite report that the annotators found a total of 13,938 images requiring non-literal interpretation, which was treated as an indicator that the ad contains symbolism. following the approach of @xcite , we treat symbol prediction as a multi-label classification problem, using the two provided label sets with 221 and 53 labels. in the original paper, the label set has been reduced from 221 to 53 via clustering foot_1 , and the data was split into training/test sets as 80%:20%. ## methods we propose a composite approach shown in general in fig. 2 , combining features mined from an ad image and from the text extracted on that image. in this section, we elaborate on the methods used on each of these feature extraction steps and in the final ensembling. image-based prediction. the image-based classifier is a central part of the proposed system. we have trained several state of the art convolutional architectures, namely mobilenet v1 @xcite , inception v3 @xcite , and efficientnet-b3 @xcite , for the multi-label classification task with c labels as follows: (1) add a new head with c outputs with sigmoid activations and train this head over 5 epochs with the adam optimizer (with frozen weights of the baseline model) for the binary cross-entropy loss; (2) train the entire model over 5 epochs with the adam optimizer; object detection. we have also tried an approach that uses the location of signifiers (symbols) on input images to improve symbol recognition. we used the faster r-cnn model @xcite with incep-tionresnet backbone trained on openimages v4 with 601 categories @xcite . objects were detected in both training and validation sets, and we retained only those objects from the training set that intersect with given symbols with iou (intersection over union) over 0.6 (fixed threshold); the objects were cropped and put in correspondence with symbol labels. we solved multilabel classification for symbol recognition by feeding the images of these objects into pre-trained cnn feature extractors, namely mobilenets v1/v2 and efficientnets b0/b3 @xcite , with a single dense layer on top of them for classification. the final decision for a test image was made as follows: (1) detect objects using the same faster r-cnn; (2) extract their visual features with pre-trained cnns; (3) classify the features with the shallow network; (4) unite its outputs (scores or predictions of symbol posterior probabilities) with non-maximal suppression, return only symbols with scores exceeding a given threshold. the f1 measure scores of this pipeline are lower than those of the previous approach. however, the methods are arguably very different, so joining both in a single ensemble might improve the results (see below). image-to-text: optical character recognition and captioning. we test our main hypothesis (that text is complementary to visual content) by extracting the text via the following ocr techniques: (1) open source ocr solution tesseract @xcite , (2) an approach for text processing from @xcite , in which the bounding boxes of text regions are obtained using the east text detector @xcite and the words in each region are recognized with tesseract ocr engine without text detection ("-oem 1 -psm 13"); ( 3 ) google text recognition library (google vision api for android) available at com.google.android.gms.vision.text, following the approach by @xcite ; (4) ocr data from cloudvision used by @xcite 3 . our experiments indicate that ocr quality has a very significant impact on the output of the predictive system. unfortunately, the tesseract text detector proved to be far from accurate on the advertisement dataset. it successfully detected text on only half of the images (1182 out of 2084 validation images), much less than commercial solutions: 1739 detected by google cloudvision api and 1894 provided by @xcite . east+tesseract approach extracted texts from 1943 validation images, but low recognition quality and arguably random text block order led to inferior prediction results; see fig. 3 for an illustration. the best-performing texts were those obtained from @xcite . in a different take on extracting text, instead of ocr we have tried to predict symbols based on text obtained via image captioning. the task of generating image captions requires to produce image textual descriptions that not only express content information of the input source but also should be naturally coherent. in our case we have chosen this approach for experiments as a completely different way to obtain texts related to the images of interest @xcite . we have used one of the best-performing image captioning models, namely ar-net @xcite pre-trained on google's conceptual captions @xcite . ar-net achieves a notable improvement in captioning due to a specific regularization strategy: previous rnn state reconstruction allows the gradient and state information to propagate through time more robustly. we note that the idea of combining captions and recognized texts is potentially fruitful because these texts are usually very different. the densecap @xcite approach, extracting captioned bounded boxes given an image, might be especially promising. we leave this idea for further study. text-based models. we have applied several text classification techniques to ocr results to establish new baselines, training a bert-based multi-label classification model using the simple transformers library (rajapakse, 2020) based on the transformers library @xcite . we have compared three architectures: (1) bert @xcite ; we used bert-base-uncased from wolf et al. (2019); (2) roberta @xcite ; we used roberta-base from wolf et al. (2019) (3) bag-of-ngrams baseline, tokenizing extracted texts and preserving only 10,000 most frequent unigrams and bigrams. as a multi-label classification model we have used logistic regression @xcite , one per each label. we have also experimented with multi-output logistic regression trained on sgns @xcite and fasttext @xcite representations, but the prediction quality was clearly worse. in preprocessing, for training we filtered out items without recognized text from the train set and lowercased the text. in the experiments, models were trained in the following settings: 15 training epochs, batch size 16, learning rate 4e-5, and other parameters at default values from @xcite . all text-based models perform clearly inferior to image-based ones; this is natural because text is not always present, often short, and even the best ocr methods make quite a lot of mistakes. however, combining image-and text-based approaches can yield significant improvements. ensemble. due to the high risk of overfitting on a small dataset, we have chosen a simple weighted sum of image-based, object detection, and text-based predictions as an ensembling strategy. for an input ad a, each model in the ensemble yields a vector f (a) ∈ [0, 1] |l| , where l is a set of labels (classes), for each data point, and the resulting ensemble outputs 1 if λ img f img (a) + λ obj f obj (e) + λ txt f txt (a) > θ, where f * (e) ∈ [0, 1] |l| are the predictions of individual models described above, and coefficients λ * , * λ * = 1, and the threshold θ are tunable parameters. for tuning, we have used the predictions of all three "elementary models" (image-, object-, and text-based) on both training and test sets, sampled (5 times) a fraction of the training set (0.1 and 0.05 for 221 and 53 labels respectively; we use the training set due to the small dataset size), then sampled λ img , λ obj and λ txt from a dirichlet distribution and evaluated the f 1 macro (not micro as an extra measure against overfitting) on the chosen training set subsample for every θ from the set {0.0, 0.05, ...1.0}. then we averaged the 5 sets of parameters. it order to compare against image-only baselines on the whole test set, we also trained a similar blend of image-and object-based classifiers (2 models) and used it as a backoff model. ## results and discussion we see that bag-of-ngrams is a strong baseline in this task, in some cases even outperforming finetuned bert and roberta. the results confirm our main conclusion: while text-based models alone are hopelessly outmatched even on images with ocr-detected text, they do add a significant boost to image-based models when ensembled together. another important point is the importance of ocr quality: significant gains are achieved only with the best ocr techniques, while adding text recognized by the basic tesseract fails to achieve meaningful improvements. in general, ## conclusion we have presented a novel approach to symbol classification on multimodal advertisement data, improving upon state of the art results already with pure image-based approaches and showing further improvements with text-based methods. we have introduced linear ensembles of the developed imagebased models and text-based models that operate with ocr-extracted text, demonstrating superior performance for the ensembles and significant improvements over state of the art for both datasets (label types). possible directions for future work include: (1) ocr postprocessing improvement: spelling correction, noise removal, etc.; (2) enhancing text-based prediction and/or object detection using thesauri, knowledge graphs, e.g., replacing specific entities with hypernyms similar to @xcite or word associations datasets, e.g., wordgame @xcite ; (3) developing a joint architecture for images and texts: obviously, simple blending might not be the best choice for the task.