Nexus at A r AIE val Shared Task: Fine-Tuning A rabic Language Models for Propaganda and Disinformation Detection (2023)

Abstract: The spread of disinformation and propagandistic content poses a threat to societal harmony, undermining informed decision-making and trust in reliable sources. Online platforms often serve as breeding grounds for such content, and malicious actors exploit the vulnerabilities of audiences to shape public opinion. Although there have been research efforts aimed at the automatic identification of disinformation and propaganda in social media content, there remain challenges in terms of performance. The ArAIEval shared task aims to further research on these particular issues within the context of the Arabic language. In this paper, we discuss our participation in these shared tasks. We competed in subtasks 1A and 2A, where our submitted system secured positions 9th and 10th, respectively. Our experiments consist of fine-tuning transformer models and using zero- and few-shot learning with GPT-4.

## introduction in various communication channels, propaganda, also known as persuasive techniques, is disseminated through a wide set of methods. these techniques can range from appealing to the audience's emotions-known as the "emotional technique" -to employing logical fallacies. examples of such fallacies include "straw man" arguments, which misrepresent someone's opinion; covert "ad hominem" attacks; and "red herrings", which introduce irrelevant data to divert attention from the issue at hand @xcite . previous research in this area has taken various approaches to identify propagandistic content. these include assessing content based on writing style and readability levels in articles @xcite , examining sentences and specific fragments within news articles using fine-grained techniques @xcite , as well as evaluating memes for propagandistic elements @xcite . moreover, malicious actors manipulate media platforms to shape public opinion, disseminate hate speech, target individuals' subconscious minds, spread offensive content, and fabricate falsehoods, among other. these efforts are part of broader strategies to influence people's thoughts and actions @xcite @xcite . in a broader context, the proliferation of such disinformation can pose significant threats to societal harmony and undermine the trust individuals have in reliable sources @xcite . currently, these manipulative strategies are widespread across various online platforms, where they are employed to influence public opinion and distort perceptions, taking advantage of the vulnerabilities of unsuspecting audiences @xcite . the far-reaching consequences of misinformation and propaganda include the incitement of prejudices and discriminatory behaviors, as well as the exacerbation of social divisions and polarization @xcite @xcite . in extreme cases, such false narratives can even fuel radicalization, threatening societal stability. ultimately, the spread of misinformation undermines democracy by depriving citizens of the accurate information needed for informed decision-making @xcite . the digital age has expanded the reach of propaganda, subtly influencing individuals' perspectives even in their most private spheres. since propaganda can manifest in a variety of forms, detecting it and other types of misinformation has always been a challenging task. this task necessitates a deeper analysis of the context in which the content is presented. therefore, the goal of the shared task is to advance research by developing methods and algorithms for identifying disinformation and propagandistic content. in figure 1 , we provide examples that depict such content. in the araieval shared task at arabicnlp 2023 @xcite , there are two tasks with two subtasks each: (i) task 1 persuasion technique detection and (ii) task 2: disinformation detection. each has two subtasks. we used pre-trained transformer-based models to fine-tune them on the task specific datasets. we participated in subtasks 1a and 2a, where we fine-tuned pretrained models to predict whether the texts contain persuasion techniques (1a) or are disinformative (2a). we also explored zero-shot and few-shot learning using gpt-4 to understand its performance for these tasks. both subtasks in which we participated fall under binary classification settings. ## related work in this section, we discuss the research related to the automatic detection of persuasion techniques and disinformation. over the past few decades, the use of persuasion techniques, often in the form of propaganda, has proliferated on social media platforms, aiming to influence or mislead audiences. this has become a major concern for a wide range of stakeholders, including social media companies and government agencies. in response to this growing issue, the emerging field of "computational propaganda" aims to automatically identify such manipulative techniques across various forms of content-textual, visual, and multimodal (e.g., memes). recently, the study by @xcite curated a variety of persuasive techniques. these range from emotional manipulations, such as using loaded language and appeal to fear, to logical fallacies like straw man (misrepresenting someone's opinion) and red herring (introducing irrelevant data). the study primarily focused on textual content, such as newspaper articles. in a similar vein, @xcite organized a shared task on the "detection of propaganda techniques in news articles." building on these previous efforts, @xcite foot_0 orchestrated the semeval-2021 shared task 6 on detection of propaganda techniques in memes in 2021. this task had a multimodal setup, integrating both text and images, and challenged participants to construct systems capable of identifying the propaganda techniques employed in specific memes. efforts have also been made towards multilingual propaganda detection. @xcite demonstrates that multilingual models significantly outperform monolingual ones, even in languages that are unseen. while most of these efforts have focused primarily on english, @xcite organized a shared task on fine-grained propaganda techniques in arabic to enrich the field of arabic ai research. this event attracted numerous participants. in addition to the use of propaganda, malicious social media users frequently disseminate disinformative content-including hate speech, offensive material, rumors, and spam-to advance social and political agendas or to harm individuals, entities, and organizations. to address this issue, the current literature has explored automated techniques for detecting disinformation on social media platforms. for example, the study by @xcite investigated the detection of fake news and hate speech in ethiopian social media. the researchers found that a hybrid approach, combining both deep learning and traditional machine learning techniques, proved to be the most effective in identifying disinformation in that context. in the field of arabic social media, numerous researchers have used various approaches for disinformation detection. for example, the study by @xcite focused on identifying hate speech and offensive content in arabic social media platforms. by employing transfer learning techniques, they found that bert @xcite and @xcite yielded the highest accuracy rates, at 98% and 96%, respectively. other significant contributions to the area of arabic hate speech and offensive content detection include works by @xcite and @xcite . ## task and dataset as discussed earlier we used the datasets released as a part of the araieval shared task @xcite . we participated in subtask 1a and 2a. they are defined as follows. subtask 1a: given a multigenre (tweet and news paragraphs of the news articles) snippet, identify whether it contains content with persuasion technique. this is a binary classification task. the data for subtask 1a is composed of ids, text, and labels. these labels are either 'true' or 'false', indicating whether the content contains a propagandistic technique. as observed in our analysis, there is a significant skew in the label distribution. as shown in subtask 2a: given a tweet, categorize whether it is disinformative. this is a binary classification task. the data format for subtask 2a is identical to that of subtask 1a. similar to subtask 1a, this subtask also shows a skewed label distribution. specifically, only 18.8% of the data is tagged as disinfo, while the remaining 79% carries the no-disinfo tag, as can be seen in for our experiments, we used the same training, development, and test datasets as provided by the organizers. details on the data distribution can be found in evaluation measures: the official evaluation metric for subtask a is micro-f1, while for subtask b, it is macro-f1. ## conclusion and future work in this paper, we report on our participation in the araieval 2023 shared task, which focuses on propaganda and disinformation detection. we experimented with various transformer-based models and fine-tuned them for our specific tasks. despite challenges such as imbalanced data, we optimized our models and achieved commendable results. our submitted system ranked 9th and 10th in subtasks 1a and 2a, respectively, on the leaderboard. in the future, our research will take advantage of the latest large language models (llms) such as llama, alpaca, bloom and more. we plan to do more experiment with data augmentation.