Pythoneers at WANLP 2022 Shared Task: Monolingual A ra BERT for A rabic Propaganda Detection and Span Extraction (2022)

Abstract: In this paper, we present two deep learning approaches that are based on AraBERT, submitted to the Propaganda Detection shared task of the Seventh Workshop for Arabic Natural Language Processing (WANLP 2022). Propaganda detection consists of two main sub-tasks, mainly propaganda identification and span extraction. We present one system per sub-task. The first system is a Multi-Task Learning model that consists of a shared AraBERT encoder with task-specific binary classification layers. This model is trained to jointly learn one binary classification task per propaganda method. The second system is an AraBERT model with a Conditional Random Field (CRF) layer. We achieved rank 3 on the first sub-task and rank 1 on the second sub-task.

## introduction social media platforms have been one of the main mediums of communication and source of information for most internet users. these platforms, as useful as they might be, can also be used to deceive and manipulate individuals. this is mostly done through propaganda techniques. propaganda can be defined as the expression of opinion that is crafted to deliberately manipulate people's beliefs, attitudes, or actions, achieving a set of specified goals @xcite . this is done by presenting certain arguments to divert the attention of the victims from everything but their own propaganda. since fallacies and propaganda devices overlap, researchers have defined propaganda techniques in terms of argumentative fallacies @xcite . several initiatives were made to detect propaganda on social media. for instance @xcite provided a fine-grained propaganda analysis and a corpus of news articles annotated with 18 propaganda techniques. this corpus was employed at semeval-2020 for propaganda identification @xcite , then at nlp4if-2020 for span detection respectively @xcite . in this paper, we present our solution to the propaganda 2022 shared task @xcite . the propaganda 2022 shared task is one of the first shared tasks of its kind and is held with the 7th arabic natural language processing workshop (wanlp 2022) co-located with the emnlp 2022 conference in abu dhabi @xcite . the goal of the task is to build models for identifying propaganda techniques in arabic tweets. it provides two sub-tasks; the goal of the first sub-task is to detect the propaganda technique used in the tweet (if any), while the goal of the second sub-task is to identify the span of the text covered by each technique. as mentioned by @xcite , the best-performing systems in the propaganda shared tasks used bidirectional encoder representations from transformers (bert) @xcite to generate contextual representations of the text. therefore, we propose to fine-tune an arabic variant of bert called arabert for each sub-task. the system submitted to the first sub-task is a multitask model that performs binary classification per propaganda technique. the system submitted for the second sub-task is an arabert model finetuned with a conditional random fields (crf) layer. both systems achieved top rankings on the leaderboard; the first system ranked third with a micro-averaged f1-score of 0.602, while the second system ranked first with a micro-averaged f1-score of 0.396. this paper is structured as follows: section 2 describes the data used for each sub-task, as well as the data preprocessing techniques employed. section 3 gives an overview of the fine-tuning process of bert models. section 4 presents the systems submitted to sub-tasks 1 and 2 respectively. in section 5, we show the results and discuss them briefly. finally, we present the related work section in section 6 and conclude the paper with section 7. ## results and discussion we conduct the analysis on the original training set. as mentioned previously, the training set is quite small (around 500 samples for training, covering 880 total labels). we notice that 51% of the tweets contain one propaganda technique, while 29% contain two propaganda techniques, and 20% of the tweets have more than three propaganda techniques. this makes the task quite challenging, as there might be instances with more than one propaganda technique present at the same time, while others with no propaganda technique at all. therefore, treating the task as multiple binary classification techniques is suitable as we are able to independently predict the presence of different techniques, while at the same time learning their co-occurrence information through sharing the same base model. for sub-task 1, the model's performance on the test set was on par with its performance on the dev_test set (similar f1-scores achieved per label, and overall). for sub-task 2, the model generalized very well and scored a much higher f1-score on the test set compared to the dev_test set. we analyze these results with respect to the distribution of the samples among the different labels. we notice that 85% of the labels in the training set are covered by 9 propaganda techniques. furthermore, the rest of the techniques have less than 10 samples in the training set. these samples might not be good representatives of their propaganda techniques that the multi-task model can generalize from. perhaps training the multi-task model to achieve a higher performance on the 9 most common techniques would have resulted in a more accurate performance of the system. there is also a need to increase the number of instances of the propaganda techniques that rarely occur in the training set. this can be done using a data augmentation method guided using domain knowledge. on a last note, both systems were tested on the straw man propaganda technique that did not occur in any set. ## related work in this section, we present some of the previous work conducted for propaganda detection, also covering the conference and labs of the evaluation forum (clef) checkthat! lab that employs factchecking (where the propaganda sentences can be viewed as fake claims). researchers provided multiple datasets to tackle the propaganda detection task. for instance, @xcite collected news articles from reliable and unreliable sources, and labeled them using distant supervision to four classes: propaganda, trusted, hoax, or satire. habernal and gurevych (2017) presented a corpus of 1.3k arguments annotated with five fallacies. furthermore, @xcite presented a corpus of news articles annotated with 18 propaganda techniques. the annotations identify the minimal fragments related to the propaganda technique (i.e., the span), instead of flagging the whole sentence. on another hand, clef provided the check-that! lab that supported the automatic identification and verification of claims in its multiple editions that are held every year @xcite @xcite @xcite @xcite @xcite . the lab provided multiple tasks around fact-checking, with the following tasks: claim detection, claim matching, evidence retrieval, and claim verification. we briefly describe each task. the claim detection task estimates the check-worthiness of the claim by predicting which claims should be prioritized for factchecking. the claim matching task determines whether a new claim is similar to a claim that has already been fact-checked; if a similar claim is found, there is no need to fact check the new claim again. the evidence retrieval task finds information that can verify a claim, by asking the participants to rank the set of evidence based on their usefulness for fact-checking a certain claim. finally, the claim verification task is a verdict prediction task in which the claim is either deemed factually true, half-true or false based on the retrieved evidence. ## conclusion in this paper, we introduced two arabert-based systems to tackle propaganda identification and span detection. we conclude that identifying propaganda techniques in arabic tweets is a challenging task. the most challenging aspect of this task lies in the small dataset used (504 samples covering 880 labels) as well as the multi-propaganda aspect of the tweets. even though the proposed systems did not employ any data augmentation technique, they achieved ranks 3 and 1 on sub-tasks 3 and 1. in future work, we propose to focus the training on the binary classification heads that handle propaganda issues that are more commonly faced by users on social media (such as loaded language and name calling/labeling). focusing our attention on these classification heads would help build models that will protect the users from the most present propaganda attacks on the web.