O tterly O bsessed W ith S emantics at S em E val-2024 Task 4: Developing a Hierarchical Multi-Label Classification Head for Large Language Models (2024)

Abstract: For our submission for Subtask 1, we developed a custom classification head that is designed to be applied atop of a Large Language Model. We reconstructed the hierarchy across multiple fully connected layers, allowing us to incorporate previous foundational decisions in subsequent, more fine-grained layers. To find the best hyperparameters, we conducted a grid-search and to compete in the multilingual setting, we translated all documents to English.

## introduction memes are widely used for communicating in the digital age, often laced with sarcasm and humor. however, beyond their role in everyday conversation, memes are increasingly recognized for their persuasive and manipulative potential. they hold power to subtly influence opinions, incite reactions, or shape public discourse and perception. given this dual nature of memes as both funny communication tool and vehicle for manipulation, there arises a need to dissect and understand the persuasion techniques embedded within them. a proper understanding of this domain enhances the ability to reflect on and emotionally defend against manipulation. in this context, large language models (llms) emerge as valuable assets in analyzing and deciphering the persuasive elements within these authors contributed equally to this work. memes. their automated, rapid processing capabilities make them well-suited for parsing through vast amounts of meme data, extracting patterns, and discerning underlying features. recognizing the importance of this topic, the semeval 2024 task 4 subtask 1 focuses on identifying persuasion techniques used in memes @xcite . the aim of the first subtask is to classify the textual content from memes into various hierarchically structured persuasion techniques. in this paper, we provide a detailed description of our system including the custom classification head we designed in order to incorporate the hierarchy of the labels. our system was able to achieve the third place on the english test dataset. furthermore, we outperformed all other systems on the bulgarian, north macedonian and arabic test sets. in summary, (i) we created a custom classification head well-suited for hierarchical settings, (ii) developed a strategy for languages where less training data is available, (iii) analyzed the influence of different hyperparameters and strategies in the context of multi-label classification problems. our code is publicly available foot_0 . ## related work in the context of multi-label classification, the primary aim is to identify all relevant classes associated with a given sample. additionally, in a hierarchical classification setting the labels are partially ordered, ranging from broader generic categories to narrowed specific instances @xcite . there is a large variety of approaches for this task. while earlier methods were based on treestructures and graphs, more recent approaches rely on deep learning models @xcite . this section introduces various models adaptable to the task of hierarchical multi-label classification. ## dataset the organizers provided english datasets for training, validation and testing (7000/500/1500, train/val/test). additionally a dev set (1000) was published to enable comparison of participating systems on a separate leaderboard ahead of the final submission on the test data. each sample within these datasets consists of an unique id, the url linking to the source of the meme, the transcribed text content and a list of associated labels. for example, the text: stay on high moral ground and we will win -raphael warnoc, has the associated labels: appeal to authority and glittering generalities (virtue). the labels of the memes are structured hierarchically, with ethos, pathos and logos in the first layer. in total, there are 28 labels with 20 persuasion techniques in the last layer, which we will refer to as leaves. it is important to note that the leaves are not distributed equally within the datasets. while smears (1990), loaded language (1750) and name calling/labeling (1518) appear most frequently in the training data, presenting irrelevant data (red herring) (59) and obfuscation, intentional vagueness, confusion (21) occur most rarely. the final submissions were made on the english test dataset. in addition, the hosts released testing data for north macedonian (259), bulgarian (436) and arabic (100) @xcite . ## system overview this section provides an in-depth description of our system. to integrate the hierarchical structure of the labels, we introduce a custom classification head that is designed to be applied atop various pre-trained large language models. ## experimental setup in order to approximate optimal parameters for the llama 2 models, we conducted a grid-search for various bert and roberta models as these require less computational resources. during training, we utilized gradient accumulation to reach a gradient update every 128 samples. all models were trained for ten epochs, with a learning rate of either 5 × 10 -4 or 5 × 10 -5 and the adam optimizer (kingma and ba, 2014). we further included the two different pre-processing styles as well as the binary-cross entropy loss and its weighted variation as hyperparameters. lastly, we performed all experiments with and without our custom classification head. training was conducted on either nvidia geforce rtx 4090 or nvidia a100 gpus. due to the large size of the llama 2 models, we used low-rank-adaptation to greatly reduce the number of trainable parameters for this model-family @xcite . we used the provided training dataset for training and the validation dataset to test generalization capabilities after each epoch. in the final stage, we assessed our system's performance on the dev dataset, utilizing a hierarchical version of the f1-score metric (hf1) following @xcite . the full set of hyperparameters we used is shown in ## results this paragraph discusses the influence of various hyperparameters, our ranking on the leaderboard and provides a detailed error analysis. ## conclusion in this paper, we introduced a robust system to classify hierarchically structured persuasion techniques in a meme-corpus for the semeval challenge 2024 task 4 subtask 1. our system achieved a top-three ranking for each language individually and outperforms every other system averaged over all languages. a key aspect of our approach is the incorporation of the label hierarchy using a custom classification head that models the individual layers of the hierarchy. this classification head can be used atop of different llms and improves the performance by up to 11 percent points. we employed a grid-search across various models and hyperparameters to approximate optimal parameters for a llama 2 13b model that then produces the embedding for the classification head. interestingly, weighting the loss to increase the influence of classes with fewer samples did not improve the overall performance. in addition, picking the same classification threshold for each class worked better than searching one for each label individually. there are multiple possibilities to build upon the success of our system: first, the organizers suggested similar data sources that could be used for pre-training. additionally, upgrading to a bigger llm, such as llama 2 70b, known for its superior performance over smaller llama 2 variants, could further elevate our system's capabilities. moreover, extending our hyperparameter tuning could uncover better model configurations. our methodology for parent-node selection could be refined by discarding parent nodes selected by children if the ancestor itself has low confidence. lastly, feature stacking could be used to create a powerful model that incorporates features generated by other models in its classification head.