Raphael at A r AIE val Shared Task: Understanding Persuasive Language and Tone, an LLM Approach (2023)

Abstract: The widespread dissemination of propaganda and disinformation on both social media and mainstream media platforms has become an urgent concern, attracting the interest of various stakeholders such as government bodies and social media companies. The challenge intensifies when dealing with understudied languages like Arabic. In this paper, we outline our approach for detecting persuasion techniques in Arabic tweets and news article paragraphs. We submitted our system to ArAIEval 2023 Shared Task 1, covering both subtasks. Our main contributions include utilizing GPT-3 to discern tone and potential persuasion techniques in text, exploring various base language models, and employing a multi-task learning approach for the specified subtasks.

## introduction in today's world, an average person encounters a plethora of information via social media platforms and online news resources. while this accessibility to up-to-date news and opinions is convenient and keeps individuals informed, it also raises concerns about disinformation, propaganda, hate speech, and political bias. over the past decade, various efforts have been made to detect disinformation and fake news through the analysis of textual content in articles @xcite , social media content @xcite , website metrics @xcite , and images @xcite . while significant progress has been made in the detection of disinformation, bias, and hate speech, the automated detection of propaganda and persuasion is a relatively newer domain. propaganda and persuasion serve as instruments to influence public opinion or evoke emotional responses and has proved to be very harmful for society in last decade. although there is substantial work in propaganda detection in english @xcite , arabic remain understudied for this problem. collecting datasets and training models for arabic becomes more challenging because of its numerous dialects spoken all around the world. the araieval 2023 task 1 @xcite is a collaborative effort that aims to address this gap by detecting persuasion techniques in arabic. task 1 is divided into two subtasks: subtask 1a is a binary classification task that detects the presence of a persuasion technique in a given arabic text, while subtask 1b is a multi-label classification problem that identifies which of the 24 possible persuasion techniques are present. in our experiments, we designed and implemented multiple systems for both subtasks. we found that leveraging outputs from large language models (llms) for supervised training led to significant performance gains. our systems ranked 4th and 3rd in subtasks 1a and 1b respectively on development sets and 5th and 6th in subtasks 1a and 1b, respectively on test sets. in the following sections, we discuss related work and elaborate on our experiments and submissions. we have made our code, prompts, and gpt-3.5 outputs publicly available for future reproducibility foot_0 . ## related work disinformation and fake news detection is a vibrant area of research within the nlp community, with methodologies ranging from text-based approaches to multimodal analyses that incorporate images and graphs. propaganda detection has also garnered attention; @xcite introduced a shared task that identifies both the span and type of propaganda technique present. this was extended by @xcite , who added a sub-task focused on recognizing persuasion techniques in memes, thus incorporating image modality. to the best of our knowledge, apart from araieval 2023 @xcite , @xcite is the only other work that specifically focuses on the detection of propaganda/persuasion techniques in the arabic language. our submitted system relies solely on the dataset provided by the organizers, without any additional data or augmentations. subtask 1a is a binary classification task featuring two labels: 'true' and 'false,' which signify the presence or absence of persuasion techniques in the text. subtask 1b, a multi-label classification problem, involves 24 labels representing various potential persuasion techniques. the data for these tasks come from two sources: tweets and paragraphs from news articles. during data preprocessing, we removed emojis and the text string "link" from all entries. ## system for both subtasks, we conducted multiple experiments that included using various base models, employing large language models (llms) for reasoning, and adjusting both the architecture and loss functions. we discuss these major components and their applications in the subsequent sub-sections. ## experiment setting all our experiments are done on single 12 gb gpu and our models take 5-15 minutes to be trained. we use "bert-base-cased", "roberta-base" and "mar-bertv2" variants from huggingface @xcite as our base models. we train our models upto 7 epochs and use adamw optimizer @xcite with learning rate being set to 2e-5, and epsilon set to 1e-8. ## results ## discussions and limitations our experiments indicate that using prompts with large language models (llms) and leveraging their outputs as features in supervised training environments show promise, especially for understudied languages like arabic. in the future, we plan to explore additional llms, with a preference for open-source options. another exciting avenue we aim to investigate is fine-tuning these llms on arabic-specific data to enhance performance. we also aim to benchmark the only llm performance without using their outputs for supervised models. one limitation we've identified is the high computational/financial cost associated with closed llm inference. however, this challenge may be mitigated as more open-source llms become available and as optimization techniques such as peft @xcite , qlora @xcite , and quantization continue to evolve. ## conclusion the widespread dissemination of propaganda and misinformation through various media channels, including social media and mainstream outlets, has garnered considerable attention from key players like government agencies and social media companies. in this study, we outline our methodology for identifying persuasive tactics employed in arabiclanguage tweets and text segments. for the 2023 araieval shared task 1, we have used gpt-3.5 as the cornerstone of our system to analyze the tone and potential persuasion strategies in the text. we have also discussed the limitations of the system proposed and suggested to incorporate open source llms and multiple optimization techniques in our future work.