P rop X plain: Can LLM s Enable Explainable Propaganda Detection? (2025)

Abstract: There has been significant research on propagandistic content detection across different modalities and languages. However, most studies have primarily focused on detection, with little attention given to explanations justifying the predicted label. This is largely due to the lack of resources that provide explanations alongside annotated labels. To address this issue, we propose a multilingual (i.e., Arabic and English) explanation-enhanced dataset, the first of its kind. Additionally, we introduce an explanation-enhanced LLM for both label detection and rationale-based explanation generation. Our findings indicate that the model performs comparably while also generating explanations. We will make the dataset and experimental resources publicly available for the research community (https://github.com/firojalam/PropXplain).

## introduction the proliferation of propagandistic content in online and social media poses a significant challenge to information credibility, shaping public opinion through manipulative rhetorical strategies @xcite . automatic propaganda detection has been an active area of research, with studies focusing on textual @xcite , multimodal @xcite , and multilingual approaches @xcite . however, most existing systems lack rational explanations that could improve media literacy and calibrate trust in predictions. @xcite developed interpretable models for propaganda detection in news articles, combining qualitative features with pre-trained language models to enhance transparency. more recently, @xcite conducted a user study in which gpt-4 was used for propaganda detection and explanation generation. they demonstrate that explanations foster critical thinking and highlight their importance. however, the current literature has paid little to no attention to developing datasets that include explanations alongside annotated propaganda labels. to address this gap, we propose a large multilingual (i.e., arabic and english) explanation-enhanced dataset for propaganda detection. we build upon existing datasets, including arpro @xcite and the semeval-2023 english dataset @xcite , enhancing them with explanations. given the complexity of manually generating explanations and the higher reliability reported for gpt-4based explanation generation @xcite , we opted to use a stronger llm for explanation generation and manually checked for quality assurance. figure 1 shows an example of a news sentence, its explanation, and human evaluation process. the developed dataset can be used to train specialized llms for propaganda detection and to provide explanations for their predictions. to this end, our contributions to this study are as follows: ## dataset we investigate llms' ability for explainable propaganda detection in both a high-resource language (english) and a lower-resource language (arabic). in this work, we extend existing datasets with natural language annotation explanations generated by openai o1, and evaluated by humans. ## llm for detection and explanation model. for developing an explanation-enhanced llm, we adapted llama 3.1 8b instruct, a robust open-source model with strong multilingual capabilities @xcite . we selected the 8b variant over larger versions (70b, 45b) due to the high computational cost of fine-tuning and inference. llama-3.1 8b has also shown strong performance in relevant multilingual tasks @xcite . instruction-following dataset. we constructed the instruction-following datasets with the aim of enhancing the model's generalizability and to guide the llm to follow user instructions, which is a standard approach to fine-tune an llm @xcite . to create versatile instructions, we prompt state-of-the-art llms including gpt-4o, and claude-3.5-sonnet to generate instructions (see appendix d). using each llm, we created ten diverse english instructions per language. each instruction is uniformly distributed across dataset samples. each sample is structured with system, user and assistant prompts, where user prompt is defined as instruction + {input_text}, and assistant prompt is defined as label: {class_label} explanation: {explanation}. training. due to limited computational resources, we adopted low-rank adaptation (lora) @xcite for training as a parameterefficient fine-tuning technique. lora captures task-specific updates through low-rank matrices that approximate full weight updates. ## results and discussion we compare our proposed fine-tuned llama 3.1 8b instruct model to baseline models: fine-tuned transformer models using arabert (as reported in @xcite ) and bert-base for arabic and english, respectively. these models are commonly-used for the task @xcite . note that bert based models are used for label prediction only. we have also fine-tuned llama without explanations to ensure a fair comparison. for arabic, the macro-f1 score achieved is 0.780, representing a 3.0% improvement over the fine-tuned arabert. for english, the results differ slightly: fine-tuned llama attains a macro-f1 of 0.686, which is 0.5% lower than that of arabert. additionally, we compare the model's performance to two llms: gpt-4o and un-finetuned llama 3.1 8b instruct. as as for its performance in explanation, in reference to the gold explanations, we observe a 25% and 40% improvements over the base model for english and arabic, respectively. the fine-tuned model shows better alignment with gold explanations as demonstrated by the example in ## related work automatic detection of misinformation and propagandistic content has gained significant attention over the past years. research has explored various problems, including cross-lingual propaganda analysis @xcite , news article propaganda detection @xcite , and misinformation and propaganda related to politics and war. building on the seminal work @xcite , resources have been developed for multilingual @xcite and multimodal setups @xcite . reasoning-based explanations in nlp have advanced fact-checking @xcite , hate speech detection @xcite , and propaganda detection @xcite . while binary classifiers effectively identify propaganda, they often lack transparency, making interpretation difficult @xcite . @xcite showed that qualitative reasoning aids deception detection, while atanasova (2024) emphasized explanation generation for better interpretability. yet, explicit prediction reasoning for propaganda detection remains under-explored, particularly in multilingual settings. our work addresses the gap by developing a multilingual explanation-enhanced dataset and proposing a specialized llm. ## conclusions and future work in this study, we introduce a multilingual dataset for propaganda detection and explanation, which is the first large dataset accompanied by explanations for the task. for arabic, we have created a new propaganda-labeled dataset of size 13k samples, consisting of tweets and news paragraphs. using openai o1, we generated explanations for this dataset, as well as for arpro (consisting of 8k instances), and for english starting from the semeval-2023 dataset. to ensure quality, we manually evaluated the explanations and found they can serve as gold-standard references. we propose an explanation-enhanced llm based on llama-3.1 (8b) that matches strong baselines in performance while providing high-quality explanations. for future work, we plan to extend it to multilabel classification and span-level propaganda detection.