G rey B ox at S em E val-2024 Task 4: Progressive Fine-tuning (for Multilingual Detection of Propaganda Techniques) (2024)

Abstract: We introduce a novel fine-tuning approach that effectively primes transformer-based language models to detect rhetorical and psychological techniques within internet memes. Our end-to-end system retains multilingual and task-general capacities from pretraining stages while adapting to domain intricacies using an increasingly targeted set of examples– achieving competitive rankings across English, Bulgarian, and North Macedonian. We find that our monolingual post-training regimen is sufficient to improve task performance in 17 language varieties beyond equivalent zero-shot capabilities despite English-only data. To promote further research, we release our code publicly on GitHub.

## introduction & background the digital age has radically transformed the nature of propaganda and disinformation, requiring innovative detection mechanisms attuned to these shifts @xcite @xcite . previous work on propaganda detection @xcite leveraged a logistic regression model to determine whether or not a given passage was propagandistic using vectors based on linguistic inquiry and word count (liwc), tf-idf, bert, and sentence features. these researchers have reported an f1 score of 66.16%, which significantly outperformed their baseline model. @xcite used a similar architecture on the task, achieving improved performance by replacing manual feature selection with induced sentence-level and articlelevel vectors. @xcite used a variety of machine learning models, including logistic regression, to create an ensemble classifier for covid-19 misinformation. more recently, there has been an emergence of work focusing on detection of propaganda in memes, @xcite releasing a corpus of memes, handlabeled with one of 22 propaganda techniques, and utilizing a fusion of large language models (llms) to successfully identify labels for a shared task: "multilingual detection of persuasion techniques in memes" (semeval 2024 task 4). the purpose of the shared task is to foster the development of systems which detect rhetorical and psychological devices, often propagandistic in nature, from memes (a more comprehensive explanation is available in @xcite . it contains the following subtasks: our system primarily tackles subtask 1, using the text of a given meme to identify which, if any, of the devices are present. our approach builds on @xcite , in tackling the challenge by leveraging the comprehensive pretraining of large language models (llms) and fine-tuning it with human-annotated examples. the multilingual and multi-task capabilities of llms have been well established, however lowresource languages and tasks often require additional data to meet or exceed human level performance. given that fine-tuning generally degrades baseline model capabilities @xcite , this reality presents obstacles when available language data does not extend to desired task contexts or vice-versa. through interative refinement, we discover that successive fine-tuning rounds -encompassing increasing task-specific data -result in models which better adapt to our specific task while 1 888 also retaining sufficient multilingual capabilities. our approach to split the post-training regimen into multiple steps finds support in prior research. @xcite found that multi-stage fine-tuning has downstream benefits, particularly in low-resource settings. @xcite examined the challenge of class imbalance by introducing a two-stage fine-tuning strategy in which they initially adjusted the model with a class-balanced 'reweighting' loss to ensure that underrepresented classes are not overlooked. our system makes use of the provided english meme data, manually labeled according to the requirements of the corresponding task. a total of 18,650 training examples generated from 11,111 unique memes were provided across the training, development, and validation splits. this paper describes our system and explores how progressive fine-tuning learns the syntactic and semantic properties of memes, with potential future applications in a variety of tasks. for more details, please see the task paper @xcite . ## system overview our system leverages a novel, multi-stage finetuning process which progressively adapts a pretrained llm (gpt 3.5-turbo foot_0 ) to the task of identifying persuasion techniques in memes. this process consists of two distinct fine-tuning steps (see figure 1 ): ## analysis we benchmark the performance of our progressively fine-tuned model, and its intermediates, on the subtask 1 development set. to further explore multilingual capabilities across post-training steps, we create 16 translated versions 5 of the held-out data encompassing a wide variety of languages. ## conclusion our work highlights the challenges inherent in adapting language models to tasks where relevant information deviates in format and/or linguistic scope from that of the desired output. our results indicate that progressive fine-tuning offers a promising method for bridging this gap. by tailoring a standard llm to effectively identify persuasion techniques within multilingual memes, we demonstrate the potential for decoupling syntactic requirements from task-specific 'understanding'. although monolingual in post-training, this method yielded performance gains across all evaluated lan-4 891 guages compared to zero-shot settings, implying similar capabilities across a wide variety of use cases. nevertheless, this work prompts further questions regarding the interplay between pre-training corpora, post-training regimes, and the nature of evaluation data. our results also call for further work in understanding how the data integration process impacts downstream performance-specifically in comparing our progressive fine-tuning approach to more common single-stage methods. crucially, our findings reinforce the urgent need to investigate and mitigate biases in llms @xcite ) that impact their performance across varied language communities and use cases.