S ussex AI at A r AIE val Shared Task: Mitigating Class Imbalance in A rabic Propaganda Detection (2024)

Abstract: In this paper, we are exploring mitigating class imbalancein Arabic propaganda detection. Given amultigenre text which could be a news paragraphor a tweet, the objective is to identify the propagandatechnique employed in the text along withthe exact span(s) where each technique occurs. Weapproach this task as a sequence tagging task. Weutilise AraBERT for sequence classification andimplement data augmentation and random truncationmethods to mitigate the class imbalance withinthe dataset. We demonstrate the importance ofconsidering macro-F1 as well as micro-F1 whenevaluating classifier performance in this scenario.

## introduction as propaganda aims at promoting certain narratives to achieve specific purposes or advance certain agendas @xcite , it can be used to manipulate people's opinions towards certain policies or ideologies. therefore, developing systems to identify propaganda techniques in texts can help individuals and organisations combat bias, promote more accurate information and make informed decisions on what to believe. this is particularly important in today's polarized political and ideological climates, where false or misleading information can be used to manipulate public opinion. early research on propaganda detection has focused mainly on article and document level classification in english such as @xcite where the authors built a corpus of news articles from different sources and labelled them broadly into four categories: propaganda, hoax, trusted, and satire. moving from the document level to a finegrained analysis of texts, @xcite conducted a fine-grained analysis of texts by detecting fragments that have propaganda and their type. in their research, the authors derived 18 propaganda techniques from literature and annotated articles from propagandist and non-propagandist news outlets. subsequently, @xcite formulated the semeval-2020 task 11 on detection of propaganda techniques in news articles, comprising two subtasks: span identification and propaganda technique classification. the aim of the current araieval shared task is to extend this line of work to the detection of propaganda in arabic text. for the unimodal (text) propagandistic technique detection task @xcite , given a multigenre text which could be a news paragraph or a tweet, the objective is to identify the propaganda technique employed in the text along with the exact span(s) where each technique occurs foot_0 . inspired by the success of bert-based approaches to propaganda detection in english, we approach this task as a sequence tagging task and utilise an @xcite token classification model. however, it is well known that class imbalance can be a problem for supervised machine learning methods. with few examples of minority classes in the training data, classifiers will tend to learn better representations of the majority classes. class imbalance can be mitigated using undersampling, oversampling and other data augmentation techniques. however, a testing regime which uses micro f1 score over imbalanced classes will naturally favour methods which are biased towards predicting majority classes. in extreme cases, a classifier could fail to predict any class except the most frequently occurring class and appear to be better than a classifier which can detect less frequently occurring classes. we argue that the macro f1 score should also be considered as it provides a more balanced assessment of the model's classi-fication performance across all classes, regardless of their individual frequencies. equipped with a classifier that can detect potential cases of minority classes, it would then be possible to bring a human into the loop to curate these examples and ultimately bootstrap a better classifier. in this paper, we explore the effect that class imbalance within the araieval dataset has on an off-the-shelf sequence tagger using arabert. we also investigate the effect of a data augmentation method and a simple random truncation method on models' performance using both the micro f1 score and the macro f1 score. the rest of this paper is organised as follows: we discuss the dataset and the class distribution (section 2); we outline the arabert token classification model used as well as the data augmentation and random truncation methods we have employed to mitigate class imbalance (section 3); we provide a detailed discussion of the evaluation metrics (section 4); we discuss our experimental results (section 5);6 we provide an overview of propaganda detection related work; and finally present our conclusions and directions for future work (section 7). ## data in this paper, we are using the araieval24 dataset foot_1 . training, development, and test data files were made available in json format. datasets with annotated instances of propaganda techniques and spans were provided for training and development. each instance of a propaganda span is represented as a dictionary containing the start and end characters of the span in addition to the propaganda technique, and the corresponding text snippet. ## methodology we approach the unimodal (text) propagandistic technique detection task as a sequence tagging task and we utilise arabert token classification model for this purpose. we fine-tuned the arabert model for token classification using the automod-elfortokenclassification class from the transformers library. the model was trained using the trainer class with the following key parameters: an output directory arabert_token_classification for saving checkpoints and logs, evaluation and save strategies set to epoch, an initial learning rate of 2e-5, 10 training epochs, a weight decay of 0.01, and automatic pushing of the model to the hugging face model hub post-training. ## evaluation metrics for our experiment results, we are reporting using the macro f1 score as well as the micro f1 score. our use of the macro f1 score is motivated by the inherent class imbalance within the dataset. the micro f1 score computes a weighted average of f1 scores across all classes, taking into account the number of instances of each class in the dataset, and is essentially equivalent to computing accuracy @xcite in the case of an imbalanced dataset where one or two classes have much higher frequency than other classes, the micro f1 score tends to be dominated by the performance of the majority class(s). it can be a useful measure of overall performance but can also be misleading particularly with regard to a model's effectiveness in classifying minority classes. in contrast, the macro f1 score computes the f1 score for each class individually and then averages these scores across all classes, assigning equal weight to each class regardless of its size. this approach ensures that the evaluation metric is not biased towards the dominant class(s) and accurately captures the model's performance across all classes, including minority classes. in the context of imbalanced data representation, the macro f1 score provides a more balanced assessment of the model's classification performance. ## results and discussion here we present our evaluation results on the test set provided. figure 3 shows the f1 scores for all of the classes for the standard non-augmented arabert model, the augmented arabert model and the random truncation arabert model. from figure 3 , we note that the standard nonaugmented arabert model fails to predict more than 12 of the minority classes. applying data augmentation has worsened the model's performance with the overall performance decreasing from a macro f1 score of 0.05 to 0.03. the decline in the model's performance following augmentation could be attributed to the existing class imbalance between propaganda and non-propaganda tokens. this imbalance possibly led to a disproportionate representation in the augmented data, thereby exacerbating the original class imbalance and consequently compromising the model's overall effectiveness. in contrast to non-augmented and augmented arabert models, applying truncation has remarkably improved the overall performance of the arabert model from a macro f1 score of 0.05 to 0.10. figure3 shows that the model performance has improved across the board and most of the minority classes are now being predicted. however, just considering the micro-f1 scores in moving from the document level, @xcite are the first to conduct a fine-grained analysis of texts detecting fragments with propaganda and identifying their type. in contrast to @xcite line of research which focuses on identifying propaganda in news articles, @xcite shift their attention to propaganda detection on social media, specifically twitter. they introduced an end-to-end transformerbased model enhanced with a multi-view approach that incorporates; context, relational data, and external knowledge into the representations. additionally, @xcite introduced an annotated dataset containing 950 arabic further enriching the resources available for social media analysis. recently, @xcite attempted leveraging gpt-4(openai, 2023) to detect propaganda spans and identify propaganda techniques using a zero-shot setting. furthermore, there have been initiatives such as @xcite and @xcite to address multimodal content like memes, expanding the scope of propaganda analysis to include various media formats. ## conclusion and future work in this paper, we introduced random truncation as a methodology to address the class imbalance in the araieval24 train dataset. our method has proved to be effective in enhancing the model performance across the majority of the classes. additionally, we emphasize the importance of employing the macro f1 score, instead of or as well as the micro f1 score, as a more suitable evaluation metric for the task of propaganda detection. considering the inherent class imbalance within the dataset, the macro f1 score ensures a balanced evaluation of the models' performance compared to the micro f1 score which is susceptible to bias influenced by the dominant class(s) in an imbalanced dataset. in the future, we will explore applying active learning strategies as another method for discovering and sampling more instances representing minority classes aiming to enhance the class balance of the train data. equipped with a more effective minority class classifier, we investigate applying it to large unannotated corpora, curating the examples found and ultimately bootstrapping a better classifier.