LMEME at S em E val-2024 Task 4: Teacher Student Fusion - Integrating CLIP with LLM s for Enhanced Persuasion Detection (2024)

Abstract: This paper describes our system used in the SemEval-2024 Task 4 Multilingual Detection of Persuasion Techniques in Memes. Our team proposes a detection system that employs a Teacher Student Fusion framework. Initially, a Large Language Model serves as the teacher, engaging in abductive reasoning on multimodal inputs to generate background knowledge on persuasion techniques, assisting in the training of a smaller downstream model. The student model adopts CLIP as an encoder for text and image features, and we incorporate an attention mechanism for modality alignment. Ultimately, our proposed system achieves a Macro-F1 score of 0.8103, ranking 1st out of 20 on the leaderboard of Subtask 2b in English. In Bulgarian, Macedonian and Arabic, our detection capabilities are ranked 1/15, 3/15 and 14/15.

## introduction memes are one of the most popular content types in online disinformation activities. they thrive on social media platforms, effortlessly reaching vast audiences. memes within disinformation activities employ various rhetorical and psychological techniques, such as oversimplification of causation, insults, and defamation, to achieve their impact on users. in this context, meme detection is crucial for identifying and reducing the spread of false information. the semeval-2024 task 4 @xcite ) is a multilingual detection task involving persuasion techniques in memes, and we participate in subtask 2b, the binary classification task, to identify whether it contains a persuasion technique or no technique.in addition to english, the task also includes three test datasets in different languages, which are only released together with the test data in the final phase of the task.the purpose of this design is to evaluate the model's performance in zero-shot scenarios, specifically its capability on languages it has not encountered before. the key to meme detection lies in uncovering rich correlations within memes between seemingly unrelated text and image components, particularly when there is no apparent connection between the text and image. in cases where the implicit meaning needs deeper exploration and understanding, traditional detection methods often fall short, as they approach meme detection in a straightforward end-to-end manner, overlooking a profound comprehension of meme text and images. recently, large language models (llms) have found success in complex reasoning. they could reveal the underlying implicit meanings beneath the surface of memes, enabling the assessment of whether persuasion techniques are present. inspired by heuristic teaching @xcite , where a teacher with rich experience can impart to students correct thinking and reasoning based on questions and corresponding answers, the students then learn how to deduce their own ways to the correct answers from questions. to better harness the powerful reasoning capabilities and knowledge reservoir of llms, we proposed a teacher student fusion detection system based on the clip @xcite model and llms. this system operates in two stages: in the first stage, as the teacher model, the llm is used to extract prior background knowledge related to persuasion techniques from memes; in the second stage, leveraging this prior knowledge, we fine-tune a smaller student model to detect whether memes contain persuasion techniques. ## system overview we define a meme detection dataset that potentially contains persuasion techniques as a set of memes where each meme @xmath0, t, y} is a triplet representing a visual content i that is associated with the textual t , and a ground-truth label @xmath1, non_propagandistic}. the core idea of our teacher student fusion model is to reason and develop a cognition-level rationale beyond the recognition-level perception @xcite by constraining the relationships between visual and textual elements in memes. to better utilize multimodal reasoning distilled from llms, this task is formulated as a natural language generation paradigm, where our model takes the text t and image i as the input and generates a textual output of the label y to clearly express whether at least one persuasion technique is present in the meme or not. in this paper, we propose to utilize abductive reasoning from llms with multimodal inputs to train smaller downstream models. our overall framework is illustrated in figure 1 , which consists of abductive reasoning from llms and model fine-tuning. ## conclusion this paper provides a detailed exposition of our approach in addressing subtask 2b of semeval2024 task 4. our teacher student fusion system initially leverages the large language model as the teacher model to generate background knowledge regarding whether memes contain persuasion techniques. subsequently, we incorporate this knowledge to fine-tune the student model by minimizing crossentropy loss, sharing learned parameters with the predictive parameters of the model. finally, we proceed with predictions using the trained model. in the future, we will explore alternative encoding methods, better aligned image and textual semantic fusion methods, llms of different sizes and types, and different ways of prompting llms to generate enhanced background knowledge for meme persuasion detection.