Optimizing Multimodal Large Language Models for Detection of Alcohol Advertisements via Adaptive Prompting (2024)

Abstract: Adolescents exposed to advertisements promoting addictive substances exhibit a higher likelihood of subsequent substance use. The predominant source for youth exposure to such advertisements is through online content accessed via smartphones. Detecting these advertisements is crucial for establishing and maintaining a safer online environment for young people. In our study, we utilized Multimodal Large Language Models (MLLMs) to identify addictive substance advertisements in digital media. The performance of MLLMs depends on the quality of the prompt used to instruct the model. To optimize our prompts, an adaptive prompt engineering approach was implemented, leveraging a genetic algorithm to refine and enhance the prompts. To evaluate the model’s performance, we augmented the RICO dataset, consisting of Android user interface screenshots, by superimposing alcohol ads onto them. Our results indicate that the MLLM can detect advertisements promoting alcohol with a 0.94 accuracy and a 0.94 F1 score.

## introduction the exposure of adolescents to advertisements promoting addictive substances is a risk factor for the subsequent development of maladaptive substance use patterns @xcite . in the case of alcohol, exposure to alcohol advertising and the level of endorsement for alcohol-related advertisements among twelve-year-olds significantly affect the severity of alcohol-related issues experienced by individuals at age fifteen @xcite . this impact is mediated by the escalation in alcohol consumption during this age period. historically, studies examining the connection between exposure to addictive substance marketing and early use initiation among teenagers has predominantly centred on well-established mediums like television and newspapers @xcite . however, the marketing landscape has evolved, with social media and web platforms emerging as dominant sources for advertising addictive substances. this shift is attributed to the under-regulation of these platforms and their widespread popularity among teenagers @xcite @xcite . in addition to advertisements sponsored by alcohol companies, there is a proliferation of user-generated content actively promoting the consumption of these substances. this phenomenon results in socially amplified advertising on social networking sites, presenting challenges in terms of regulation and monitoring @xcite . multimodal large language models (mllms) can process data from multiple modalities, such as text, images, and audio. in this study, we employed an mllm to automate the detection of alcohol advertisements within digital media. similar to large language models (llms), the efficacy of an mllm is contingent upon the instructive prompt's quality @xcite . while substantial efforts have been directed toward prompt engineering for models that can only process text @xcite @xcite , the exploration of prompt engineering for models capable of handling both text and images remains relatively underexplored. to optimize the instruction prompt for our mllm, we employed a genetic algorithm for prompt generation and selection. each of the instruction prompts represented an individual in our genetic algorithm. through an iterative process of mutating and reproducing the fittest prompts, we identified the one yielding the best results. each of the instruction prompts were crafted based on the following prompt engineering techniques: chain-of-thought (cot) @xcite , generated knowledge (gk) prompting @xcite , self-critique @xcite , and expert prompting @xcite . thus, our research also provides insights into the effectiveness of different prompt engineering techniques for mllms. to evaluate the performance of our model, we augmented the rico dataset @xcite . the rico dataset comprises screenshots of user interfaces from various android apps, such as social, dating, and communication apps. to augment the dataset, we incorporated advertisements from alcohol companies by superimposing them onto the rico images. the mllm was employed to classify the images based on the presence or absence of alcohol ads. the evaluation involved measuring the accuracy and f1 score of the classifier. our main contributions are as follows: ## method 3.1 data collection to construct our training and testing dataset, we utilized the rico dataset @xcite by extracting 2,100 distinct user interface (ui) screenshots from it. additionally, we employed a web scraper to gather images from google featuring alcohol advertisements. please see appendix a for the terms used to search for alcohol ad images. an author of the paper reviewed the downloaded images to remove non-alcohol-related ones, resulting in a curated dataset of 2,100 different alcohol ad images. these advertisement images were resized to one-eighth of the ui images and superimposed onto them. please refer to figure 5 in appendix b for an example of a superimposed image. consequently, the resultant dataset comprised 4,200 images, which were partitioned into a stratified training and testing datasets, allocating 3,200 for training and 1,000 for testing. the training dataset was then divided into batches of 200 image each, each one containing an equal number of images with and without alcohol ads. ## results in this section, we present our findings derived from the evaluation tasks. the subsequent section is dedicated to a comprehensive discussion and analysis of the implications arising from these outcomes. the optimized prompts ultimately resulting from the genetic algorithms were as follows: ## discussion the most effective prompts and prevalent mutation types observed throughout multiple generations stemmed from the cot prompt engineering technique, with the top-performing prompts from the final generation being a product of a cot mutation prompt. however, upon examining the prompts, we noted their integration of elements from different prompt engineering methods. prompts created from the generated knowledge mutation prompts consistently include enumerations of components for image inspection, as shown in this optimized prompts. therefore, our findings suggest that the optimal prompt engineering approach involves a blend of different techniques. the performance of the open-source model in detecting alcohol ads in images is comparable to that of the proprietary model. this is a promising result, as it enables researchers to analyze sensitive images without the necessity of sending them to third-party organizations. moreover, the fact that the model is open-source potentially reduces costs, hence increasing accessibility to the tools in less well-resourced settings. our adaptive prompt engineering technique presents a more accessible approach for public health researchers seeking to apply automated methods to the identification of other types of harmful online content. notably, our method reduces the need for users to possess a background in machine learning for training to optimize an mllm. additionally, it operates without reliance on the model's proprietary weights or architecture, which can be inaccessible. furthermore, users are not required to possess prompt engineering experience, as state-of-the-art prompt engineering techniques are already integrated into the algorithm. moreover, our algorithm allows for easy upgrades upon the discovery of new prompt engineering techniques, requiring only their addition to the mutation pool. ## conclusion we developed a genetic algorithm to optimize the prompt for mllms to detect harmful content in images. we also extended the rico dataset which contains ui screenshots by superimposing alcohol advertisements. the optimal prompt achieved an accuracy score of 0.94 and a f1 score of 0.94. the mutation prompts utilized in our algorithm were derived from prompt engineering techniques traditionally employed for llms. however, these approaches had not been previously tested within the framework of mllms. by tracking the performance of each mutation type, we identified that, within the realm of mllms, the cot and the generated knowledge mutations outperformed the expert and self-critique approaches. although our algorithm was initially designed and tested for detecting alcohol advertisements, it can be extended to identify other harmful substances such as tobacco and drugs when provided with the appropriate data sets. we envision that by adapting our algorithm online platforms can detect and remove harmful content, thereby fostering a safer online environment.