BDA at S em E val-2024 Task 4: Detection of Persuasion in Memes Across Languages with Ensemble Learning and External Knowledge (2024)

Abstract: This paper outlines our multimodal ensemble learning system for identifying persuasion techniques in memes. We contribute an approach which utilises the novel inclusion of consistent named visual entities extracted using Google Vision’s API as an external knowledge source, joined to our multimodal ensemble via late fusion. As well as detailing our experiments in ensemble combinations, fusion methods and data augmentation, we explore the impact of including external data and summarise post-evaluation improvements to our architecture based on analysis of the task results.

## introduction in this paper, we describe our approach to identifying persuasion techniques for semeval 2024 task 4. the task involves the identification of up to 22 persuasion techniques in memes, which are inherently multimodal. we participated in subtask2a and subtask2b. subtask2a is a multilabel classification task, requiring the identification of 22 persuasion techniques using both textual and visual content. the subtask is evaluated by a hierarchical f1, as each label is part of a subset of techniques and contains a parent node. subtask2b is a binary classification task, determining the presence or absence of any persuasion technique within a meme (propagandistic or non-propagandistic). for both subtasks, training data is provided in the english language and a development set also in english. as well as english, 3 surprise languages in arabic, north macedonian and bulgarian were provided to officially evaluate our approach @xcite . our system architecture is an amalgamation of traditional nlp and vision models, exploring late and early fusion techniques as well as carefully crafted confidence thresholds. we extend beyond the training data by incorporating resources such as google vision foot_0 , which provides consistent named visual entities extracted from the image regardless of language; in a multilingual context this reduces reliance on sentence spans or tokens, which can be problematic due to linguistic variations in unseen language data. we also make our code publicly available. 2 ## background identifying persuasion techniques in memes is necessary endeavour for combating misinformation and fostering critical media consumption among the public, and the focus of a number of ongoing research areas for the prevention of harmful content, propaganda or disinformation spread through memes @xcite @xcite . propaganda is generally referred to as information which is purposefully shaped or presented to support a particular agenda, often utilising the persuasion techniques in this shared task. previous shared tasks have also considered the identification of persuasion techniques in text only @xcite , multimodal contexts using memes @xcite , and persuasion techniques in multilingual text @xcite . se-meval 2024 task 4 is a shared task of a similar nature, however the task considers both image and text as well as multilingual test data. as meaning is often generating through the interaction of both modalities in memes, meme related tasks are typically approached using pretrained convolutional neural networks @xcite @xcite or vision transformers @xcite in combination with language models. our ensemble approach therefore explores cnns for the binary classifica-tion task; for the more complex multilabel classification, we explore clip @xcite to leverage its significant pretraining on large-scale natural language descriptions and images, as well as its notable performance in zero-shot classification and related downstream multimodal tasks such as social media sentiment analysis @xcite . our motivation for including external knowledge sources is inspired by previous successful applications of external information @xcite and ongoing research to improve meme-related tasks with the addition of structured knowledge to provide context to memes @xcite . ## exploratory analysis we briefly explore the task data and use this analysis to inform our approach, particularly for the more challenging subtask2a. exploring subtask2a, we calculated tf-idf vectors for texts within each label and calculated the cosine similarity between these vectors. we noted that, for the majority of labels, there is significant crossover in textual content. we also examine the number of labels in a single meme, as subtask2a was a multilabel classification problem where each meme could have more than one persuasion technique, in figure 1 . given this crossover, we initially explored leveraging the annotation guidelines for the task, which provides concrete examples of how to label each persuasion technique. we noted the annotation guidelines primarily provided examples annotation based on the location of nouns or adjectives per technique, but provided few examples of non-european languages aside from russian. however, the guidelines did note the presence of 'personal characteristics, organisations, political orientation or opinions' in some techniques @xcite . we therefore explore a more concise representation of these attributes using the google vision api to extract 'web entities' and visual concepts from an image. for multilingual data, this allows us to rely less on sentence spans or tokens -elements that vary across language -and instead leverage visual entities that could consistently represent information for each label regardless of textual content. in our main system approach includes ensembling nlp models with vision models for both subtasks. we experimented with bert @xcite and roberta @xcite family models as well as vgg19 @xcite , resnet50 @xcite and clip @xcite . for subtask2a, we initially design an architecture that combines multilingual text processing with visual analysis. our vision stream also includes web entities from google vision, processed by a single bert model. our subtask2b system similarly integrates visual and textual modalities with experiments in late and early fusion. we also include additional novel implementations beyond an ensemble of pretrained models: external knowledge: we use google vision to extract information from meme images. the google vision api annotates an image using web detection, returning a list of predicted labels for objects, people or concepts in an image, as well as matching urls and the google knowledge graph id @xcite . we utilise only the named visual entities, with an example in ## experimental setup we combine the training and validation sets for subtask2a and subtask2b to train each architecture, a total of 7,500 for subtask2a and 1,350 for sub-task2b originally in english. we test our approach on the development set in english (1,000 samples for subtask2a and 300 for subtask2b). detailed in section 5.1, the total samples are increased by direct translating data for both subtasks. for all experiments, we set the validation split in the model to 30% of the total training data. when multiple languages are included in the data, we stratify the training and test splits based on language. the number of epochs is determined by no improvement to validation loss after 5 epochs. we find that the majority of the language models ## development set results we detail the results of our experiments for sub-task2a in for subtask2a, we found the triad combination performed best with bert (trained on the extracted google vision entities, model bert(ex) in whilst we explored the use of ptc to finetune our models, we found that, due to the different naming conventions of some techniques, performance did not improve with incorporation of the ptc data. we also noted the ptc data was drawn from a different domain (e.g., news articles) were the context of techniques would be longer than short sentences in memes, and potentially this corpus was less effective as a finetuning dataset for the task. we originally aimed to leverage clip's text and image embeddings to inform a novel early fusion neural network model for multilabel multiclass persuasion techniques classification. however, this architecture including clip was slightly less effective than others. the reasons behind this suboptimal performance could be multifaceted, including the complexity and subtlety of propagandistic content within memes, the inherent challenges of cross-modal understanding in this particular domain. one reason is suggested that, whilst the visual modality is important for identifying whether a technique is present, distinguishing between the specific types of techniques may primarily be a linguistic task. for subtask2b, our architecture achieved overall better scores than subtask2a. we tested architectures retrained for a binary classification task from subtask2a on subtask2b as a comparison, noting these models did not perform as well. in subtask2b, therefore, the vision modality was significant in the binary classification task. we note from the results monolingual language models outperform multilingual models, and suggest this may be due to the limited sample size for the augmented data in subtask2b. in line with our system strategy, we include bert(ex) only in conjunction with multilingual models, as the aim of this additional data is to improve zero-shot classification irrespective of language. we observed significant performance increase using the bert(ex) model in late fusion for subtask2b. ## test set performance and analysis for the test set, we submitted the best performing model from each subtask experiment. for sub-task2a, this was the triad + bert(ex) with late fusion. for subtask2b, we submitted the vgg19 + bert model for english test sets and the vgg19 + xlm-roberta-base + bert(ex) for all other languages. evaluating our results on the test set in ## conclusion and future work we presented our ensemble learning approach to semeval-2024 task 4, including a number of experiments with early and late fusion, the inclusion of external knowledge and modifying the label threshold. we found that the inclusion of external sources of knowledge, even basic descriptive entities as in our experiments, improved performance on both subtasks especially using late fusion. by their nature, memes are multimodal; our approach to subtask2a still utilised visual elements via entities extracted from the image, and thus provided essential context to interpret ambiguous textual content, however we found the balance between visual and textual importance varied across meme types and tasks. whilst subtask2a benefited from the integration of visual entities as a more concise representation of the visual modality, we found that much of the context required for identifying specific techniques required either better cross-modal understanding or finer text analysis. in contrast, subtask2b benefited from a strong visual model. the identification of named entities in visual modality of memes is a potential future area of research, as this would enable drawing on complex stores of knowledge (e.g., knowledge graphs) for deeper cross-modal understanding when disentangling persuasion techniques. we further suggest that there is promise in generating more high quality, multilingual data for persuasion techniques across languages based on our experiments with augmented data, particularly for low-resource languages. although we augmented the task data to cover more languages using direct translation, a limitation in this method is the inability to change the visual modality. we also note there is a cultural element to memes not considered in current research. we identified that north macedonian memes were visually different from other memes; the different cultural perspectives and practices in developing memes is under-researched, with only limited studies investigating global meme practices @xcite . as well varied training data, a better understanding of cultural meme production could contribute to defining the most appropriate approach for zero-shot multilingual meme tasks.