Pauk at S em E val-2024 Task 4: A Neuro-Symbolic Method for Consistent Classification of Propaganda Techniques in Memes (2024)

Abstract: Memes play a key role in most modern informa-tion campaigns, particularly propaganda cam-paigns. Identifying the persuasive techniquespresent in memes is an important step in de-veloping systems to recognize and curtail pro-paganda. This work presents a framework toidentify the persuasive techniques present inmemes for the SemEval 2024 Task 4, accordingto a hierarchical taxonomy of propaganda tech-niques. The framework involves a knowledgedistillation method, where the base model is acombination of DeBERTa and ResNET usedto classify the text and image, and the teachermodel consists of a group of weakly enforcedlogic rules that promote the hierarchy of per-suasion techniques. The addition of the logicrule layer for knowledge distillation shows im-provement in respecting the hierarchy of thetaxonomy with a slight boost in performance.

## introduction propaganda has long been used in media as a communication technique to influence people to subscribe to a particular idea or ideology. identifying the presence of propaganda in media is an important subtask in building systems that can curtail the effect of propaganda campaigns. in modern times, a common way for propaganda to be spread is via memes. a meme is a either a short video or image, often overlayed with text, that is widely circulated over the internet. identifying the presence and type of persuasion techniques used in memes is an important problem to be solved. the organizers of semeval 2024 task 4 propose a shared task for this very problem @xcite . the shared task involves three sub-tasks: sub-task 1, identifying the persuasion technique(s) involved in the textual content of the meme; sub-task 2a, identifying which persuasion techniques are involved in both the visual and textual content of the meme; and sub-task 2b, identifying whether or not any persuasion technique is present in the visual and textual content of the meme. while the training data for all sub-tasks is in english, the evaluation phase includes different test sets for english, arabic, bulgarian, and north macedonian. the proposed framework will focus only on the english version of the first two sub-tasks. both 1 and 2a are hierarchical multi-label classification problems, where the goal is to classify the correct persuasion techniques used in the meme. the hierarchical nature of the persuasion techniques adds an extra element to this classification. all of the possible persuasion techniques are organized in a directed acyclic graph (dag), and full credit for a prediction is only given when the correct leaf node is predicted. partial credit is given when any of its ancestors are given as a prediction. there is a lot of previous work in the area of propaganda and persuasion identification. much of this work has been based on previous semeval shared tasks. the semeval 2021 task 6 is almost identical to the task explored here, without the hierarchical label structure @xcite . the best-performing approaches on this task consisted of the use of a fine-tuned, text-based transformer for the textual content of the meme, some cnn or transformer based vision model to extract features from the image, and then a consolidation of the resulting embeddings via simple aggregation such as concatenation or average @xcite . another semeval task in 2023 focused on the identification of propaganda techniques in the text of news articles @xcite . in this case the best models used fine-tuned bert based transformers @xcite . the approach presented in this paper will also leverage a combination of a text-based transformer and a visual neural model. the key addition will be the incorporation of logic rules that encode the relationship between the possible hierarchical output classes. these relationships can be modeled by simple rules where the presence of a particular persuasion technique implies that its parent technique in the hierarchy is also present. for example, the rule straw @xmath0. these rules will be used to promote predictions of persuasive techniques that respect the hierarchy. to test hierarchical consistency, we measure the number of hierarchy violations in predictions, where the model predicts a persuasive technique but not one of its ancestors in the hierarchy. to incorporate logic rules, we take inspiration from the teacher-student logic rule framework proposed by @xcite that distills the information encoded in logic rules into the neural network parameters. the focus of this work is to explore if the incorporation of logic rules can improve on the results of neural based modes, while also producing more consistent results. the intuition is that distilling these hierarchical relationships into the weights of the network will allow the network to better recognize patterns that correspond to types of persuasive techniques, and ultimately make better predictions. we show that the addition of these logic rules does result in much more consistent predictions with a slight improvement in f1 scores. ## background and related work propaganda and persuasion there is ample existing work in the identification of propaganda techniques. a similar semeval task was proposed in 2021 to classify memes without including the hierarchy requirement @xcite . @xcite proposed a framework for this task involving a text-based transformer built on roberta, a visual feature extractor, and then a final transformer which takes as input the output of roberta and the visual feature extractor. the authors consider two methods for this final encoder, a text pretrained transformer and a multi-modal transformer. the multi-modal transformer approach works the best, and they slightly improve on this score by combining several models together in an ensemble, which gives them state of the art results for the task. @xcite take a slightly different approach with this task. they similarly decompose the problem into a bert based transformer and a visual feature extractor, but use a simpler method for combining the output embeddings by simply concatenating the output features before using a final classifier. they also experiment with a few different types of image feature extractors: a model specifically tuned to recognize faces, an optical character recognition (ocr) model tuned to recognize text in an image, and the best performing extractor, which is a region based image feature extraction model that feeds into a multi-modal model. other previous semeval tasks have focused just on the identification of propaganda techniques in text by leveraging multilingual datasets of news articles @xcite . @xcite and @xcite presented the two top performing systems for this task. both use very similar approaches, leveraging a fine tuned version of roberta for classification. @xcite had an interesting additional class weighting mechanism to try to improve performance on the under represented classes in the dataset. the approach proposed will leverage a lot of the same ideas regarding the usage of bert based transformers for textual content, as well as visual feature extractors. where this approach differs is in the incorporation of logic rules representing the relationship between propaganda techniques. hierarchical classification hierarchical multi label classification problems are split into two types: local methods, where an independent classifier is used for each node or for each level of the hierarchy; and global methods, which consider the entire hierarchy all at once @xcite . in this paper, the interest is in exploring a global method that leverages logic rules to represent the relationship between classes in the hierarchy. similar work has already been done in this area. @xcite propose a coherent hierarchical multi-label classification network (c-hmcnn) which uses a constraint layer on top of the regular network, as well as a specialized loss function to require the hierarchical constraints be satisfied. the constraint used is a simple one: the output probability of a subclass of a particular class in the hierarchy must be less than or equal to the output probability of its super-class. the approach for this project will be similar, but will follow more closely to the general logic rule integration method proposed by @xcite . this framework consists of a teacher-student network, where the teacher network encodes logic rules as soft logic and distills that knowledge into the student network. this framework allows for more flexibility in the kinds of logic rules inte-grated into a network. additionally, it is less strict when enforcing the rules on the outputs of the neural network, allowing for a better balance between the signal coming from the direct supervision and the hierarchical knowledge. ## system overview the proposed model architecture is based on a student-teacher knowledge distillation approach consisting of several components, which vary depending on the sub-task, but share a common structure. regardless of task, base classifiers are used to encode raw textual and/or visual content. these resulting embeddings are then concatenated and used for predictions by the student network. the teacher network consists of a logic rule layer on top of the base student model that encodes the hierarchical information of the propaganda techniques. this logic layer is based on the teacher-student framework proposed by @xcite . the framework distills the knowledge from the teacher network into the student network by training the student network to simultaneously emulate the gold labels and the teacher predictions. ## experimental setup and results the method outlined above is evaluated on two separate but related tasks provided by the organizers of the semeval 2024 task 4 @xcite . the details of these tasks and results are described below. model implementations for both sub-tasks leverage tensorflow for modeling and hugging face for the deberta and resnet models @xcite ). foot_0 ## conclusion and future work the framework presented attempts to solve the task of identification of persuasive techniques in memes. the key innovation involved in this framework compared to previous work done in this space is the integration of a logic rule knowledge distillation layer that weakly applies rules encoding the hierarchy of persuasion techniques. this layer is applied on top of a base model using a transformer based deberta model for the textual component and a resnet for the image component. we find that the logic rule network has some positive effect, consistently resulting in fewer hierarchical violations and a slight improvement in micro f1 scores. however, these logic rules do not lead to better predictions on the leaf node techniques themselves. there were some difficulties in integrating these logic rules. the way the framework is set up, violations of the rules result in low probabilities for predictions by the teacher model. the part of the loss function that involves the kl divergence between these student and teacher predictions can cause the network to learn one of two aspects to minimize this loss. the first option is to raise the prediction probability of the rule violating ancestor label in the student network which will result in no rule violation and therefore no addition to the loss. alternatively, the student network predictions can be lowered even further which also minimizes the kl divergence. the goal is for the former result to be learned, but it seems that often the latter is learned especially when the rule confidences are very high. further work can be done to explore alternative logic rule interactions or loss function formulations to ensure the latter is always learned by the network. in addition to improvements in the hierarchical logic rule integrations themselves, more work can be done to improve the base model by exploring other image processing methods outside of using a basic resnet. additionally, more intelligent ways of combining the textual hidden states and image hidden states can be explored, such as the use of a basic attention mechanism. finally, the exploration of additional logic rules that promote parts of the textual content of an example that may indicate a particular persuasion technique could be experimented with. this may be especially useful for persuasion techniques with low coverage in the dataset.