A ra BEM at WANLP 2022 Shared Task: Propaganda Detection in A rabic Tweets (2022)

Abstract: Propaganda is information or ideas that an organized group or government spreads to influence peopleś opinions, especially by not giving all the facts or secretly emphasizing only one way of looking at the points. The ability to automatically detect propaganda-related linguistic signs is a challenging task that researchers in the NLP community have recently started to address. This paper presents the participation of our team AraBEM in the propaganda detection shared task on Arabic tweets. Our system utilized a pre-trained BERT model to perform multi-class binary classification. It attained the best score at 0.602 micro-f1, ranking third on subtask-1, which identifies the propaganda techniques as a multilabel classification problem with a baseline of 0.079.

## introduction with the increasing popularity of social media (sm) in our modern society, social media platforms like twitter have become essential means for influencing others. people on social media tend to convey their views and perspectives more freely. the ability of sm users to freely express their views has allowed interested parties to profile users based on how they express their opinions on social media. as such, the past few years have witnessed a great interest in targeting a broader spectrum of audiences via twitter and other sm platforms. parties like political and advertisement campaigns are competing to reach out to the broadest audience base possible and hence influence the general publicś view. it can be seen that the more ability a particular part has to influence peopleś opinions, the more powerful it becomes @xcite . the term propaganda is defined in the cambridge dictionary as information or ideas that an organised group or government spreads to influence people's opinions by not giving all the facts or secretly emphasising only one way of looking at the facts. 1 the spread of propaganda exploits the anonymity of the internet, the micro-profiling ability of sm platforms, and the power of automatically creating and managing coordinated networks of accounts to reach a large number of sm users with persuasive messages @xcite . spreading propaganda to promote a specific agenda has become a business @xcite . in this context, the concept of automatic propaganda detection has risen recently @xcite . researchers have focused on utilising stateof-the-art nlp techniques to develop systems for automatic propaganda detection. the main challenges in this regard include difficulty identifying and extracting the linguistic signs of propaganda use. this is particularly difficult due to the cunning and indirect ways propaganda can be expressed. as such, detecting propaganda-related techniques can be challenging even for a human expert. regarding arabic, propaganda detection can be a more challenging task due to several additional factors, including the limited availability of linguistic resources (e.g., corpus) and the morphologically-rich nature of the arabic language @xcite . to bridge this research gap, a shared task about auto-detection of propaganda in arabic social media has been launched. foot_1 subtasks in this shared task and a comprehensive description of the shared task are discussed in @xcite . in this work, our team participated in the first subtask-1, ranking our system in the third position. ## related work the literature on propaganda detection as an nlp task reveals an increasing interest in exploring this research area @xcite . previous work on propaganda detection indicates several common challenges associated with this task. specifically, the limited availability of the annotated dataset, the ability to convey propaganda with means other than text (e.g., images) @xcite and the difficulty of spotting direct and indirect propaganda techniques are among the most prominent challenges of the task of propaganda detection. a total of eighteen propaganda techniques have been identified in previous work. however, experts stated that propaganda techniques are not fixed and keep evolving @xcite . previous work on propaganda detection has mainly focused on english (chaudhari and pawar, 2022), as a well-resourced language. in addition, researchers highlighted that most propagandarelated languages tend to appear in biased news and sm platforms, unleashing different directions like promoting political agendas and radicalisation @xcite . @xcite summarised the features utilised in existing systems for detecting propaganda techniques. this includes user-based, time-based, metadata-based and context-based features, n-grams, and pre-trained models (e.g., bert). in @xcite , the authors performed text mining on some of the propaganda content published by isis to recruit women from around the world. the authors applied a lexicalbased emotion analysis method to detect emotions most likely to be evoked in readers of these materials. regarding propaganda detection in arabic, literature shows that few previous attempts have been made to address this issue @xcite @xcite . this need has provoked the lunching of a shared task of propaganda detection in arabic and releasing a newly built dataset annotated specifically for propaganda techniques @xcite . ## data in this work, we utilise the dataset released for the shared task described in this overview paper @xcite . an initial observation reveals highly unbalanced classes in the obtained dataset, as shown in ## system we explored approaches used in previous work on detecting propaganda or misleading language in social media. we noted that previous systems utilised different methods ranging from traditional machine learning @xcite to modern neural-based systems @xcite . our team decided to use a pre-trained model, specifically bert, which has performed well in previous work on propaganda detection in the news @xcite . the model has also been successfully used to detect auto-generated arabic tweets, aiming to spot propaganda accounts @xcite . we use bert for multi-class binary label classification. the token size we use is 70 based on our calculation of the average tweet length. the output of running bert on the shared-task dataset is several tensors, each associated with the possibility of the presence of propaganda techniques. to decide on the threshold, we ran several experiments and used the threshold 0.2, showing that any value above this threshold would be considered a presence of a propaganda technique. a possible future expansion of this work can include experimenting with a different threshold for each propaganda technique to test its impact on the overall system performance. we fine-tuned the pre-trained model using the training and development data. foot_3 ## results and discussion we used the script the shared task organisers provided to evaluate our system. the best results sub-mitted by our system were reported at a micro f-1 score of 0.602, ranking third place in the leaderboard of the shared task. the details of all results can be found in @xcite . overall, the scores attained by the participating systems reflect the difficulty of the task of auto-detection of propaganda language in arabic tweets. we believe a possible explanation is a small size and highly unbalanced annotated dataset provided with the shared task. identifying and annotating propaganda techniques can be challenging even for a human expert, @xcite and as such, expanding the scale of the dataset by using methods like data augmentation might help improve the performance. another issue is that some propaganda techniques, are more frequently used than others, like loaded language. as mentioned in section 3 and ## conclusion the occurrence of propaganda and misleading information to promote a specific agenda has coincided with the growing popularity of sm platforms like twitter. before that, news outlets would generally be the primary source of information for people; hence, the broadcast news would usually come trustworthy with no need to question or cross-check their legitimacy. contrarily, the broad spectrum of audiences on sm platforms and the ability to readily access and spread propaganda to promote specific agendas has attracted interesting parties (e.g., political, advertisement, radicalization). as such, the need for nlp researchers to come together to address propaganda and fake news detection, especially in social media, has emerged. in this context, a shared task has been launched to detect propaganda techniques in arabic tweets automatically, and an annotated dataset has been released as part of the shared task. our team, arabem, has participated in subtask-1, which is about classifying propaganda techniques, and ranked in the third position attaining a micro f-1 score of 0.602 compared to a baseline of 0.079. we used a pre-trained bert model and decided on 0.2 as the threshold for tensors to determine if a propaganda technique was spotted. overall, the results indicate a good performance among the participating team. however, more investigations are still required to enhance the system's ability to identify propaganda techniques accurately. future directions for expanding this research include experimenting with different pre-trained models and threshold settings. in addition, more investigations on data balancing methods like data augmentation would shed light on distinct possibilities for performance improvement. more experiments should be done to assess the systems' ability to detect the more cunning and indirect ways of creating and spreading propaganda.