S hef CDT eam at S em E val-2024 Task 4: A Text-to-Text Model for Multi-Label Classification (2024)

Abstract: This paper presents our findings for SemEval2024 Task 4. We submit only to subtask 1, applying the text-to-text framework using a FLAN-T5 model with a combination of parameter efficient fine-tuning methods - low-rankadaptation and prompt tuning. Overall, we find that the system performs well in English, but performance is limited in Bulgarian, North Macedonian and Arabic. Our analysis raises interesting questions about the effects of labelorder and label names when applying the text-to-text framework.

## introduction social media platforms have become increasingly popular over time @xcite . whilst this enables greater public discourse, information and disinformation can also be presented purposefully to influence opinions online . therefore, it is important to explore the detection of persuasion techniques. by fulfilling this goal, strategies that counteract false or misleading narratives can developed, and internet users can be empowered to think more critically about what they see online. this paper describes our submission for semeval-2024 task 4: multilingual detection of persuasion techniques in memes. we took a text only approach, and as such we only tackled subtask 1 -given only the "textual content" of a meme, our system must identify which persuasion techniques (of a possible 20) are used @xcite . the labels are organized in a hierarchy (see figure 1 ) and multiple labels may apply to the same data point. for example: ## background research on identifying persuasion techniques in memes builds on the efforts of propaganda detection @xcite . rashkin et al. (2017) @xcite took a more finegrained approach. they developed a dataset of news articles with an annotation schema consisting of 18 propaganda techniques. they proposed a multi-granularity network using contextual embeddings derived with bert (see also @xcite . @xcite presents a multilingual and multifaceted dataset of news articles, annotated with genre, framing and persuasion tech-niques. they also evaluated the performance of a transformer model at various granularity levels -token-level, sentence-level, paragraph-level, and document-level. to the best of our knowledge, there has been no work completed on exploring text-to-text (also known as sequence-to-sequence, or seq2seq) models for this multilingual, multi-label classification task in the domain of meme language. text-to-text models take in text as input and output new text. models such as t5 can be applied to many different tasks under the text-to-text framework @xcite . they have also been shown to be effective in zero-shot settings @xcite . ## system overview we use flan-t5 (chung et al., 2022) as our base model. flan-t5 was created by fine-tuning t5 @xcite on a mixture of tasks including text classification, question answering, and translation. the model regards every task as a text-to-text task. we train in two steps: ## experimental setup for hardware reasons, we use a sharded version of flan-t5-xxl 1 loaded in 8-bit precision. the training set (@xmath0. preprocessing was required to transform the data into an appropriate format for text-to-text training. we transform the input text to lower case, and for lora we prepended a simple task prompt. for example: we use the peft implementation of lora and prompt tuning @xcite . for lora, we train for 5 epochs with a learning rate of 0.001. we mostly use the same hyperparameters for prompt tuning as mozes et al. (2023) on t5-xxl. we initialise the prompt as: 'which persuasion techniques are in this text? text: ' more details on hyperparameters for both training steps can be found in appendix b. the evaluation measure used in this task is hierarchical f1 @xcite , which takes into account the tree structure of the labels when calculating model performance. ## results our final results are summarised in the model failed to generalize to the fourth language, arabic, despite its presence in the flan-t5 training data -we did not make a submission for this language as the model predicted no labels for all inputs. to investigate the errors of our model, we analysed the data on a per-label basis using our best performing language, english. instead of using hierarchical f1, we split the multilabel task into 20 binary tasks (one for the prediction of each label) and calculated the average f1 score for each. in general, our system performed better on labels that were common in the training data (see figure 4 ). several labels with very low training set prevalence had f1 scores of zero. ## conclusion in this paper we present a case study for the application of the text-to-text framework to multilabel classification. while our model exhibits some strengths, it did not achieve performance on par with top-ranking results. however, our analysis shows the potential for label names to affect performance, and suggests that shuffling labels during training may lead to increased performance.