BAMBAS at S em E val-2024 Task 4: How far can we get without looking at hierarchies? (2024)

Abstract: This paper describes the BAMBAS team’s participation in SemEval-2024 Task 4 Subtask 1, which focused on the multilabel classification of persuasion techniques in the textual content of Internet memes. We explored a lightweight approach that does not consider the hierarchy of labels. First, we get the text embeddings leveraging the multilingual tweets-based language model, Bernice. Next, we use those embeddings to train a separate binary classifier for each label, adopting independent oversampling strategies in each model in a binary-relevance style. We tested our approach over the English dataset, exceeding the baseline by 21 percentage points, while ranking in 23th in terms of hierarchical F1 and 11st in terms of hierarchical recall.

## introduction in the multilabel classification problem (mlc), each instance may belong to zero, one, or multiple class labels. the goal is to learn a system to infer the correct labels of previously unseen instances @xcite . mlc has several real-world applications, ranging from text categorization @xcite to protein and gene function prediction @xcite . this work addresses a critical novel application of mlc: detecting persuasion techniques in memes, considering only their textual content, a subtask of semeval-2024 task4 1 . the merriam-webster dictionary foot_1 defines meme as "an amusing or interesting item (such as a captioned picture or video) or genre of items that is spread widely online, especially through social media". nonetheless, and unfortunately, in recent years, memes have been used not only to amuse people but also as a tool for disseminating disinformation in political campaigns @xcite . malicious actors embed sophisticated propaganda and persuasion techniques within these memes, employing psychological and rhetorical strategies. this manipulation extends to the memes' textual and visual components @xcite . like other computational propaganda @xcite , memes significantly influence public opinion. their effectiveness stems from their widespread reach, potentially impacting millions of internet users globally. additionally, memes are often not perceived as propaganda by these users, primarily because they do not mirror the appearance of conventional political advertisements @xcite . as an effort to address this problem, semeval-2024 task 4 @xcite promoted a challenge in which competitors should develop algorithms to identify the use of persuasion techniques in memes, considering only their textual content (subtask 1) or text and image together (subtasks 2a and 2b). in this paper, we describe our approach to addressing subtask 1. for this subtask, the shared-task organizers made available a collection of 8,500 texts in english extracted from real internet memes (7,000 for training and the remaining divided into validation and dev sets). each text may be assigned to a set of labels that indicate the persuasion techniques present in it foot_2 . there are a set of 20 possible labels organized in a hierarchy -thus, we have a hierarchical multilabel classification problem @xcite . some texts can have no label assigned, indicating they do not correspond to propaganda. the shared task aimed to produce the best model according to the hierarchical-f1 metric. test collec-tions in four different languages were made available: english, bulgarian, north macedonian, and arabic. our team (bambas) participated in the english challenge along with 31 other teams. we explored a lightweight approach based on three components. the first component is a language model from which we extract embedding features leveraging the [cls] token. the second component is a binary relevance-based strategy to train 20 separate binary classifiers (one for each existing label) @xcite . our central inquiry focused on assessing the extent to which such a lightweight model that does not engage with the intricacies of hierarchical structures could be effective. the third core component handles the inherent imbalance of multilabel hierarchical problems by employing an independent oversampling strategy @xcite to reduce the imbalance between negative and positive examples present in each binary problem derived. the hierarchical-f1 score of our submitted solution exceeded the baseline by 21 percentage points. in the hierarchical f1-based rank, we were the 23 th out of 31 teams. however, when considering the hierarchical recall, we were ranked as 11 st foot_3 . the rest of the paper is organized as follows. section 2 briefly overviews mlc concepts relevant to this paper. section 3 details our proposed system. in sections 4 and 5, we present the experimental methodology and report the results, respectively. finally, section 6 brings the conclusion and future research directions. ## background over the last 20 years, mlc has been one of the most active research topics in machine learning @xcite . among the several methods for multilabel learning in the literature @xcite , binary relevance (br) @xcite ) stands out as one of the most prominent methods. this approach decomposes the multilabel problem into q binary problems, where q is the number of labels. then, one binary classifier is independently trained for each label. the labels of new instances are predicted by combining the outputs of each classifier. the br method offers several key advantages. firstly, its simplicity and intuitiveness make it highly accessible. additionally, br models can predict label sets not present in the training set, owing to their composition as a series of independent binary classifiers. most crucially, br has consistently exhibited high prediction accuracy values across various domains. in a recent extensive experimental comparison @xcite involving 26 methods across 42 datasets, models utilizing br outperformed all models trained with other different transformation strategies. nonetheless, the br method suffers from three major drawbacks. first, it ignores the possible correlations among labels @xcite . second, br has high training and prediction times for problems in which the number of labels is huge (tens of thousands to millions) @xcite . third, its predictive performance is affected by class imbalance, which occurs when the number of examples relevant to each label is much inferior to the number of irrelevant ones @xcite . we consider that the first two drawbacks are not crucial for addressing semeval-2024 task 4 subtask 1, as the number of labels in the problem is not large (@xmath0. more specifically, we found that the highest pearson correlation value is 0.13 -between labels "glittering generalities (virtue)" and "flag-waving". on the other hand, we consider that the issue of class imbalance needs to be taken into account as the imbalance ratio (ratio of negative to positive examples) is 47.38 on average in the training set, and the maximum value reaches 332.33 for the label "obfuscation, intentional vagueness, confusion". our approach is detailed in the next section. ## system overview the shared task proposed in semeval-2024 task 4 comprises an output of one or more labels -in case the meme is a propaganda -disposed in a hierarchical taxonomy of persuasion techniques. the root of such hierarchy is naturally labeled persuasion, while the second level has three possible branches: ethos, pathos, logos. while ethos and logos conduct to labels in a third level, pathos branch connects directly to the persuasion techniques -the leaves of the tree. this way, the final output can be one or more paths from the root to some leaf. handling such a hierarchical structure directly is quite challenging in machine learning. the algorithms should accurately predict multiple outputs while respecting the labels' hierarchical relationships. however, errors can propagate down the hierarchy. moreover, some paths have very few instances, adding another layer of complexity to the problem: data sparsity and imbalance. therefore, our primary solution to the problem was to investigate how far an algorithm that disregards the hierarchy could go. additionally, we also decided not to handle the multiple labels directly. however, employ the binary-relevance approach and consider a component to handle imbalance by adding synthetic instances with smote @xcite and randomoversampler @xcite , for each binary problem. algorithm 1 depicts the training procedure and algorithm 2 the inference. our method hinges on three core components. the first one creates the features from the meme textual content, leveraging a pre-trained language model (line 3 in algorithm 1). the second component addresses class imbalance by creating synthetic instances (line 10 in algorithm 1). the third component trains independent binary classifiers (line 12 in algorithm 1), employing the binary-relevance strategy. during the inference phase, each label classifier undergoes evaluation, and the instance is assigned all the positive classifications predicted by each classifier. in line with our straightforward premise, we have implemented a feature-based strategy that utilizes embeddings from pre-trained language models (ptlms). the textual content of each meme is processed through the ptlm, allowing our system to capture the numeric feature vector from the [cls] token. this method effectively harnesses the power of ptlms to distill complex language information into a manageable form for further training our classifiers. our selection choice for ptlms includes a writing free-style multilingual model, namely, xlm-roberta @xcite and two informal writing style models, one monolingual (bertweet (nguyen et al., 2020)) and one multilingual @xcite ). xlm-roberta is a multilingual adaptation of the roberta model, pre-trained on a 2.5tb of data across 100 languages. roberta itself is a transformers model trained on large raw text corpora. the key training method used is masked language modeling (mlm), where 15% of the words in a sentence are masked, and the model predicts these masked words, learning a bidirectional representation of the sentence. bertweet is a monolingual model trained from 850m tweets. it has the same architecture as bert-base but was trained using the roberta pre-training procedure. bernice is a multilingual roberta language model trained from 2.5 billion tweets. ## experimental setup our solution was implemented using hugging-face (wolf et al., 2020) 5 and scikit-learn (pedregosa et al., 2011) 6 libraries. the experiments were conducted on an nvidia dgx-1, using a single tesla p100 gpu with 16gb of vram. we conducted a step-by-step analysis to reach the final modeling decisions. the intermediate results, necessary to decide the components of our final solution, are reported with the validation set. the final solution was trained with the training set and we report the results of the english dev and test set. we proceed this way because the dev set could only be measured with the submission page before the release of its gold labels. all results are reported with the competition's evaluation metric, a hierarchical variant of the f1score @xcite . the metric considers the classification taxonomy, rewarding a full score for exact leaves prediction, and rewarding a partial score for ancestor predictions. the closer the predicted ancestor is to the correct labels, the higher the partial score. additionally, we report the hierarchical variants of precision and recall. the first analysis consists of defining the ptlm to extract the embeddings. we did not employ oversampling during this phase and applied a binaryrelevance model using logistic regression. the ptlm and logistic regression hyperparameters were left as default. the meme textual content is presented to the ptlm without any pre-processing. next, we explore 6 other classifiers besides logistic regression: decision tree, extra tree, extra trees, knn, random forest, and ridge classifier. the last analysis focused on selecting the best oversampling strategy. we experimented with smote and a random oversampling strategy, both implemented in the imbalanced-learn library 7 . all the results so far included 20 binary classifiers, each associated with a persuasion technique in the leaves of the tree. then, we investigate a final possibility of including some internal nodes related to the classes that were worst classified. the best model from those analyses was submitted to the competition. ## results finally, we included additional classifiers to some internal nodes of the hierarchy. such an extension includes only the internal nodes corresponding to the least accurately classified leaf nodes. these nodes are "ad hominem", "distraction" and "logos". given the preceding results, we selected that approach for the final submission. ## conclusion this paper addresses the semeval-2024 competition with a lightweight solution to investigate how a model that neglects the hierarchy would behave in a hierarchical task. our solution uses a tweets-based ptlm as a feature extractor, generates synthetic data to account for imbalance, and employs a binary relevance strategy to handle multiple labels. our next step is to investigate training a structured output classifier that predicts the paths in the hierarchy. moreover, given that oversampling strategies enhanced the performance of most of the classes, we plan to design other strategies explicitly tailored to the style of memes.