M c R ock at S em E val-2024 Task 4: Mistral 7 B for Multilingual Detection of Persuasion Techniques In Memes (2024)

Abstract: One of the most widely used content types in internet misinformation campaigns is memes. Since they can readily reach a big number of users on social media sites, they are most successful there. Memes used in a disinformation campaign include a variety of rhetorical and psychological strategies, including smearing, name-calling, and causal oversimplification, to achieve their goal of influencing the users. The shared task’s objective is to develop models for recognizing these strategies solely in a meme’s textual content (Subtask 1) and in a multimodal context where both the textual and visual material must be analysed simultaneously (Subtasks two and three). In this paper, we discuss the application of a Mistral 7B model to address the Subtask one in English. Find the persuasive strategy that a meme employs from a hierarchy of twenty based just on its “textual content.” Only a portion of the reward is awarded if the technique’s ancestor node is chosen. This classification issue is multilabel hierarchical. Our approach based on the use of a Mistral 7B model obtains a Hierarchical F1 of 0.42 a Hierarchical Precision of 0.30 and a Hierarchical Recall of 0.71. Our selected approach is able to outperform the baseline provided for the competition.

## introduction when information is intentionally crafted to serve a predetermined agenda, we often classify it as propaganda @xcite . propaganda employs various psychological and rhetorical strategies to achieve its goals @xcite . these methods encompass the utilization of logical fallacies and the manipulation of audience emotions @xcite . logical fallacies can be particularly deceptive as they may initially appear sound and impartial, yet upon closer examination, it becomes evident that the conclusion cannot be logically derived from the premises. another tactic involves employing emotionally charged language to sway the audience's opinion, bypassing rational analysis in favour of an emotional connection. memes, typically comprising images overlaid with text, serve as a platform for propagandistic dissemination. within deceptive memes, images either reinforce or complement textual techniques, or they themselves convey persuasive strategies. to address these objectives, there is an ongoing demand for automated tools capable of extracting and categorizing data from online sources, facilitating the response to both established and emerging societal concerns. recent advancements in machine and deep learning architectures have spurred heightened interest in natural language processing (nlp). substantial endeavours have been directed towards devising techniques for the automated identification and categorization of textual content accessible on the internet today. in the literature, to perform text classification tasks, several strategies have already been proposed. in the last fifteen years, some of the most successful strategies have been based on svm @xcite , on convolutional neural network (cnn) @xcite , on graph neural network (gnn) @xcite , on ensemble models @xcite and, recently, on transformers @xcite . the surge in the adoption of transformer-based architectures within academic research has been further propelled by diverse methodologies showcased at semeval 2024. these methodologies address a range of tasks and yield notable outcomes. for instance, in task 2 @xcite , t5 is utilized to confront the challenge of identifying the inference relation between plain language statements and clinical trial reports @xcite . in task 10, a mistral 7b model is employed to perform emotion recognition in conversation (erc) within hindi-english code-mixed conversations @xcite . additionally, in task 8 @xcite , a distilbert model is leveraged to identify machine-generated text @xcite . finally, for the task 4 at semeval 2024 @xcite -multilingual detection of persuasion techniques in memes -three subtasks were proposed. as already stated, in a disinformation campaign, memes effectively manipulate users through various rhetorical and psychological strategies, including causal oversimplification, name-calling, and smear tactics. the objective of this shared task is to develop models capable of detecting these techniques within the textual content of memes alone (subtask 1), as well as within a multimodal framework where both textual and visual elements are analysed jointly (subtasks 2 and 3). to face with the first subtask in english, we proposed a transformer-based approach which made use of mistral 7b @xcite . we used the model in a particular few-shot way described in the rest of this paper. specifically, we provided the definitions of the 20 techniques to the model to identify, given each sample, all the techniques detected. we opted for mistral 7b because the comparative analysis between mistral 7b and other leading models, namely llama 2 and llama 1, reveals noteworthy advancements in common nlp tasks. across multiple benchmark evaluations, mistral 7b consistently exhibits superior performance in comparison to llama 2, a prominent open 13b model. moreover, its efficacy extends beyond mere parity with, but rather exceeds, the achievements of llama 1, a state-of-the-art 34b model, particularly in tasks pertaining to reasoning, mathematics, and code generation. these findings underscore mistral 7b's substantive contributions to the advancement of nlp, suggesting its potential as a benchmark model in the field. the rest of the paper is made as follows. in section 2 we provide some background on the task 4 hosted at semeval 2024. in section 3 we provide a description of the models presented. in section 4 we provide details about the experimental setup to replicate our work. in section 5, the results of the official task and some discussions are provided. in section 6 we present our conclusion and proposals for future works. we make all the code publicly available and reusable on github 1 . this section furnishes background information regarding task 4 (subtask 1), held at semeval 2024. the task entails identifying, based solely on the textual content of a meme, which of the 20 persuasion techniques, organized hierarchically, are employed. the selection of an ancestor node of a technique warrants only partial reward. the task thus presents a hierarchical, multilabel classification challenge. the hierarchical structure is illustrated in the official task's page, with 22 techniques depicted, although "transfer" and "appeal to strong emotion" are excluded from subtask 1. for comprehensive details, please refer to provided resources. for all subtasks, the annotations from the ptc corpus, comprising over 20,000 sentences, were utilized where feasible. although the corpus pertains to news articles, annotations adhere to identical guidelines, albeit with fewer techniques considered. as highlighted by the task coordinators, certain meme content may be deemed offensive or excessively potent by certain audiences. a similar multilingual corpus was also accessible during se-meval 2023 @xcite . here again, the corpus revolves around news articles across nine languages, yet the number of techniques and annotation guidelines differ marginally. a training set for local system development was additionally provided. furthermore, the organizers furnished a development set and a public leaderboard for real-time result sharing among task participants. ultimately, the organizers supplied a test set devoid of annotations and an online submission platform to evaluate the system performance. subtask 1 relies on the textual content extracted from memes as input data. training, development, and test sets for all subtasks are disseminated as json files, with each subtask having its own individual file. for subtasks 2a and 2b, in addition to the meme's textual content, input data includes the meme's image. in the figure 1 , is reported a sample from the official competition website foot_0 . given the figure 1 : ## system overview even if it has already been proved that the transformers are not necessarily the best option for any text classification task @xcite , depending on the goal some strategies like domain-specific fine-tuning @xcite , or data augmentation @xcite can be beneficial in several applications. our approach is a few-shot one (littenberg-tobias et al., 2022) and make use of the abovementioned mistral 7b. mistral 7b, a language model equipped with 7 billion parameters, is designed to excel in both performance and efficiency. compared to the leading open 13b model (llama 2), mistral 7b demonstrates superior performance across all evaluated benchmarks. moreover, it outperforms the top released 34b model (llama 1) in tasks related to reasoning, mathematics, and code generation. the model leverages groupedquery attention (gqa) to expedite inference, along with sliding window attention (swa) to efficiently process sequences of varying lengths while minimizing inference costs. additionally, a fine-tuned variant, mistral 7b -instruct, tailored for adhering to instructions, surpasses the llama 2 13b -chat model across both human and automated benchmarks. the introduction of mistral 7b instruct underscores the ease with which the base model can be fine-tuned to achieve notable performance enhancements. notably, this variant lacks any moderation mechanisms. the mistral 7b instruct variant requires a specific input format, as stated below: ## experimental setup we implemented our model on google colab. the library we used comes from hugging face and as already mentioned is mistral 7b foot_2 . we employed the v0.2 iteration of mistral 7b, which represents an enhanced version of the mistral-7b-instruct-v0.1 model. to harness the capabilities of instruction fine-tuning, prompts must be enclosed within @xcite and [/inst] tokens. additionally, the initial instruction should commence with a sentence identifier. the next instructions should not. the assistant generation will be ended by the end-ofsentence token id. we also imported the llama library @xcite from llama_cpp. the library is fully described on github foot_3 . the dataset provided for all the phases are available on the official competition page. we did not perform any additional fine-tuning on the model. to run the experiment, a t4 gpu from google has been used. after the generation of predictions, we exported the results on the format required by the organizers. as already mentioned, all of our code is available on github. ## results the evaluation was done by submitting to the leaderboard the predictions provided by the model. subtask 1 and 2a are reliant on a hierarchical structure. the gold label consistently corresponds to a leaf node within the directed acyclic graph (dag). however, any node within the dag can serve as a predicted label: however, it's worth noting that the hierarchical structure can be disregarded by confining predictions solely to technique names. this approach renders the task analogous to semeval 2023 task 3 @xcite . an illustrative example of the evaluation function can be accessed online foot_4 . in this case, the subtask consists of a hierarchical multilabel classification task. drawing from the aforementioned figure depicting the hierarchy, any node within the dag can be designated as a predicted label. the gold label consistently corresponds to a leaf node within the dag. hierarchical-f1, detailed in @xcite , is employed as the official evaluation metric. in the ## conclusion this paper presents the application of mistral 7bmodel for addressing the task 4 at semeval 2024. in the table are shown the results obtained by the first three users and by the last one. furthermore, is included the result of the baseline considered and of our approach making use of mistral 7b. in parentheses is reported the position in the official final ranking. for our submission, we decided to follow a fewshot learning approach, employing as-is, an indomain pre-trained transformer. after several experiments, we found beneficial to build a prompt containing the definitions of the techniques in memes. then we provide, as a prompt, the definitions together with a sample. the model was asked to select all the techniques detected in the sentence. the task is challenging, and there is still opportunity for improvement, as can be noted looking at the final ranking. possible alternative approaches include utilizing the zero-shot capabilities of other models like gpt and t5, increasing the size of the training set by using further data, or directly integrating ontology-based domain knowledge differently than what has been proposed in our work. further improvements could be obtained with a fine-tuning and modelling the problem as a different text classification task. furthermore, given the interesting results recently provided on a plethora of tasks, also other few-shot learning @xcite @xcite or data augmentation strategies @xcite @xcite could be employed to improve the results. looking at the final ranking, our simple approach exhibits some room for improvements. however, it is worth notice that required no further pre-training and the computational cost to address the task is manageable with the free online resources offered by google colab. also, thanks to the proposed approach, we have been able to outperform the baseline provided by the task organizers.