CNLP - NITS - PP at WANLP 2022 Shared Task: Propaganda Detection in A rabic using Data Augmentation and A ra BERT Pre-trained Model (2022)

Abstract: In today’s time, online users are regularly exposed to media posts that are propagandistic. Several strategies have been developed to promote safer media consumption in Arabic to combat this. However, there is a limited available multilabel annotated social media dataset. In this work, we have used a pre-trained AraBERT twitter-base model on an expanded train data via data augmentation. Our team CNLP-NITS-PP, has achieved the third rank in subtask 1 at WANLP-2022, for propaganda detection in Arabic (shared task) in terms of micro-F1 score of 0.602.

## introduction communities are significantly impacted by the propagation of rumors, false information, or incomplete information, particularly if the process is spearheaded by the media. in the minds of the target populations, who are consistently the targets of propaganda, the illusion becomes true. propaganda is regarded as one of the most effective political tools in the modern period and consistently succeeds in drawing sizable populations. social media is now utilized to distribute propaganda and bogus or misleading news to divert attention away from more pressing problems. depending on the technology employed, a variety of materials and media are used to spread propaganda. the most modern methods @xcite for detecting propaganda are based on language models, which mostly use transformer-based architectures. there are publicly available language models for arabic, such as arabert (antoun et al., 2020), aragpt2 @xcite and araelec-tra @xcite . the challenging issue is the requirement of a sufficient amount of annotated multilabel dataset that must cover different varieties or types of propaganda in order to utilize advanced deep learning-based techniques. to encounter this issue, we have increased the train data by augmenting data to the original train set. we have noticed that people often use dictionary/root or stemmed words on social media platforms without adhering to proper grammar. therefore, we have used arabic light stemmer @xcite in the train data and prepared 1, 008 additional synthetic dataset that is directly augmented with the original train data. moreover, arabert (antoun et al., 2020) twitter-base model is utilized in this work and attains competitive results in terms of standard evaluation metrics (as reported in section 5). ## related work in this section, we briefly present the related works of propaganda detection which have been studied recently. by identifying all text fragments that contain propaganda techniques and their type, the authors @xcite undertake finegrained analysis of texts. they have contributed a corpus of news articles that are annotated using 18 propaganda techniques at the fragment level and designed a suitable evaluation measure. also, a multigranularity neural network is designed and attained better performance than the bert-based baseline system. in @xcite , the authors introduce a multi-label multimodal task to detect the different types of propaganda techniques used in memes and release a corpus that includes 950 memes annotated with 22 propaganda techniques. @xcite organizes semeval-2021 task 6 which include subtasks of detecting the persuasion techniques in the text, the text spans where the persuasion techniques are used, and detection of particular technique present in the entire meme (text and image). they explored the benefits of text and image modalities for the detection techniques in the respective shared tasks. moreover, @xcite proposed to use of interpretable features with pre-trained language models for detecting deception techniques. ## dataset description the dataset foot_0 used has been provided by the organizers of wanlp 2022 for the shared task on propaganda detection in arabic @xcite . the dataset consists of the text of arabic tweets and the list of propaganda techniques used in them. there are a total of 21 propaganda techniques, namely, "appeal to authority", "appeal to fear/prejudice", "black-andwhite fallacy/dictatorship", "causal oversimplification", "doubt", "exaggeration/minimisation", "flag-waving", "glittering generalities (virtue)", "loaded language", "misrepresentation of someone's position (straw man)", "name calling/labeling", "obfuscation", "intentional vagueness", "confusion", "presenting irrelevant data (red herring)", "reductio ad hitlerum", "repetition", "slogans", "smears", "thought-terminating cliché", "whataboutism", "bandwagon", and a "no technique" label to indicate no propaganda techniques have been used. the train, validation, and final test set consist of 504, 104, and 440 number of tweets. for data augmentation, for each tweet in the training set, we have used an arabic light stemmer foot_1 @xcite to get the root and stem and obtained synthetic data are added to the training set with the same labels. this brought up the number of training samples to 1512. ## system description the arabert (antoun et al., 2020) twitter-base model is utilized for the task of multilabel propaganda classification and used example source code (antoun et al., 2020) foot_2 for text classification. however, the example code (antoun et al., 2020) is restricted for single-label classification. to prepare it for multi-label classification, we have changed the input labels to the model to be one hot encoded to indicate multiple labels and modify the macro-f1 scorer to give a score for multiple labels. we used data augmentation; in particular, generated synthetic training data using root and stem substitution from the original train samples and prepared additional synthetic examples. for preprocessing, the default arabertpreprocessor foot_3 has been used. during training to get the predicted labels for one tweet, we selected the number of predicted labels corresponding to the number of true labels for that tweet. for training, we have used 0.1 drop-out, adam optimizer with a default learning rate, and a batch size of 16. the model is trained on a single nvidia quadro p2000 gpu for 5 epochs based on early stopping criteria, i.e, the model training is halted if it does not converge on the validation set for more than 5 epochs. the training process took less than 5 minutes. to make predictions with the model, the sentiment analysis pipeline is used from huggingface transformers foot_4 , which returns scores corresponding to each of the labels for a given input. then we selected all the labels that provide a score greater than or equal to 0.32 as the predicted labels. we observed multiple scores for predictions on the validation test set and found that most correct labels have a score greater than 0.30 and there was a large gap in the score for the labels that have scored less than 0.30. ## discussion in this work, we have presented preliminary experimental work in subtask 1 only at wanlp 2022 shared task. in future work, we need to explore and examine different deep-learning-based models such as aragpt2, araelectra, araxlnet on the same benchmark data set released in this shared task and utilize another dataset, namely, ptc cor-pus foot_6 for both tasks, i.e., multilabel classification and sequence tagging tasks. moreover, we will manually observe the benchmark dataset to identify the clue for the expansion of train data, the novelty in the multilabel annotation, preprocessing, and model training that could be increased accuracy in multilabel classification and sequence tagging tasks. ## conclusion this paper demonstrates our work in subtask 1, propaganda detection in arabic shared task at wanlp-2022. to handle the data scarcity problem in this shared task, we have proposed to use a data augmentation strategy and utilization of a domainspecific pre-trained language model (arabert twitter-base model) that shows remarkable results. this work motivates us to explore propaganda detection in indian languages which will be beneficial for a multilingual country like india.