Multilingual Propaganda Detection: Exploring Transformer-Based Models m BERT , XLM - R o BERT a, and m T 5 (2025)

Abstract: This research investigates multilingual propaganda detection by employing transformer-based models, specifically mBERT, XLM-RoBERTa, and mT5. The study utilizes a balanced dataset from the BiasFigNews corpus, annotated for propaganda and bias across five languages. The models were finely tuned to generate embeddings for classification tasks. The evaluation revealed mT5 as the most effective model, achieving an accuracy of 99.61% and an F1-score of 0.9961, followed by mBERT and XLM-RoBERTa with accuracies of 92% and 91.41%, respectively. The findings demonstrate the efficacy of transformer-based embeddings in detecting propaganda while also highlighting challenges in subtle class distinctions. Future work aims to enhance cross-lingual adaptability and explore lightweight models for resource-constrained settings.

## introduction propaganda detection is a critical step in combating the spread of misinformation and biased content designed to manipulate public opinion. propaganda content often relies on subtle linguistic cues, making its detection a complex task even in monolingual contexts. while recent advancements in natural language processing (nlp), particularly transformer-based models, have significantly improved propaganda detection in english, multilingual detection remains a challenging frontier. this challenge stems from linguistic diversity, contextual variability, and the scarcity of high-quality annotated datasets for non-english languages, which limits the generalizability of existing models. previous studies have focused on monolingual approaches, leveraging machine learning and deep learning models to identify and categorize propagandistic content. transformer-based architectures, such as bert and its multilingual variants, have demonstrated strong performance in capturing complex linguistic patterns. however, these approaches often face limitations in multilingual settings due to imbalanced datasets, inconsistent performance across languages, and challenges in distinguishing nuanced propaganda categories. existing models also struggle to balance precision and recall across diverse classes, particularly in low-resource languages. in this paper, we address these limitations by integrating transformer-based models-mbert, xlm-roberta, and mt5-into ensemble frameworks designed to enhance multilingual propaganda detection. we aim to improve classification robustness and performance across languages. our approach leverages the strengths of individual models while mitigating their weaknesses through ensemble learning, offering a more balanced and effective solution for multilingual classification tasks. the rest of the paper is organized as follows: section 2 shows related work on the propaganda classification problem. section 3.1 describes the dataset and preprocessing techniques used in this study, section 3.2 details the transformer models and ensemble frameworks implemented. section 4 presents the experimental results, including individual model performance and ensemble outcomes. finally, section 5 discusses the findings, limitations, and potential directions for future research. ## related work in this section, we review recent advancements in related fields, focusing on propaganda detection, political bias detection, extremism detection, and multilingual misinformation detection. these studies provide a foundation for understanding the challenges and methodologies in multilingual classification tasks, highlighting limitations in existing approaches and motivating the contributions of this work. the safari @xcite study evaluates cross-lingual political bias and factuality detection using media-and article-level datasets. ensemble models with soft voting and multilingual pre-trained language models (mplms) like mbert and xlm-r performed best on english datasets, achieving f1 scores of 84.96% for factuality and 84.95% for political bias. multilingual datasets showed weaker results due to limited training data, with the best f1 score for political bias reaching 29.05%. article-level models performed strongly on english-distant supervision datasets (f1: 82.62%) but less so on expert-annotated and multilingual data, highlighting challenges in crosslingual transfer. joint modeling, combining political bias and factuality detection, achieved its highest f1 score of 83.81% using soft voting. large language models (llms), including mistral7b and llama27b, underperformed mplms in zeroshot settings, with f1 scores up to 46.84%. the study employed clustering techniques for data curation and evaluated models using f1 scores, accuracy, and recall. while ensemble methods and mplms proved effective, challenges remain in multilingual adaptation and fine-grained zero-shot learning. recent advancements @xcite ) in propaganda detection have utilized diverse methods, including fine-tuned transformers, few-shot gpt prompting, and classical machine learning. studies focused on a dataset of tweets by diplomats from china, russia, the u.s., and the eu in english and spanish, tackling binary to finegrained multilabel classification tasks. among these, xlm-roberta (xlm-bi) emerged as the top-performing model, excelling in multilingual and spanish tasks, while roberta (rob-en) demonstrated strong performance on englishspecific tasks. few-shot prompting with gpt-3.5 and gpt-4o showed potential for binary classification, with gpt-4o outperforming gpt-3.5 but not reaching the accuracy of fine-tuned bert models. classical machine learning approaches, including lightgbm and xgboost with stylometrix linguistic features, offered competitive performance, particularly in english binary classification, where lightgbm's f1 score rivaled that of bert-based models. these findings highlight the strength of fine-tuned transformers for complex multilingual tasks, while also recognizing the effectiveness of classical machine learning and gpt-based methods in specific contexts. advancements in extremism and radicalization detection @xcite have ex-plored multilingual datasets and sophisticated classification techniques. recent research introduced a multilingual corpus for binary and multiclass classification tasks, encompassing texts on extremism and radicalization from diverse sources such as isis-related content and hate speech in languages like english, arabic, indian, korean, and kazakh. the dataset includes 17,100 samples for binary extremism, 5,000 for binary radicalization, and 11,400 for multiclass extremism. the study utilized preprocessing methods such as language-specific cleaning, tf-idf, and word embeddings, alongside machine learning models (e.g., l.svc, random forest) and deep learning approaches (bi-lstm, distilbert-multi). bi-lstm achieved high accuracy for binary classification (97.8% for radicalization, 96.81% for extremism), while transformerbased models excelled in multiclass classification with 91.07% accuracy. these results highlight the effectiveness of deep learning and transformer models for multilingual extremism detection tasks. recent efforts in political bias detection @xcite have introduced an annotated dataset of 1,388 hindi news articles and headlines from diverse sources, balanced across three categories: biased towards bjp, biased against bjp, and neutral. articles were annotated for coverage and tonality bias with a kappa score of 0.65, highlighting the subjective challenges in labeling neutrality. the dataset features class-specific averages in word and sentence counts, providing insights into linguistic characteristics. the study evaluated transformer-based models, including mbert, xlm-roberta, and indicbert, alongside traditional machine learning approaches like svm, logistic regression, and random forest. xlm-roberta achieved the best results with 83% accuracy and an f1-macro score of 76.4%, significantly outperforming traditional models, which scored below 60% in f1-macro. the findings emphasize the effectiveness of multilingual transformers in bias detection tasks and highlight challenges in accurately identifying neutral articles due to their subjective nature. recent research @xcite on misinformation detection has focused on multilingual datasets and transformer-based models. a study using tweets related to covid-19 from the nlp4if 2021 shared task investigated misinformation detection in english, bulgarian, and arabic. the dataset included binary annotations for seven questions assessing misinformation charac-teristics, with 451-3,000 training samples and up to 1,000 test samples per language. the study evaluated logistic regression, a transformer encoder, and bert-based models. english bert @xcite achieved the best results for english (f1: 0.729), while multilingual bert (m-bert) demonstrated strong cross-lingual generalization, achieving f1 scores of 0.843 for bulgarian and 0.741 for arabic. experimental setups, including zero-shot, few-shot, and target-only training, highlighted the potential of m-bert to perform well in low-resource settings with minimal target-language data. these findings emphasize the effectiveness of contextualized embeddings and multilingual transformers in detecting misinformation across diverse languages. gap analysis of related work: despite the advancements in multilingual propaganda detection, several gaps remain that highlight the need for further exploration: limited multilingual representation: while studies like safari and nlp4if have explored multilingual contexts, their datasets are often limited in linguistic diversity, focusing on highresource languages like english and spanish. lowresource languages, which are equally vulnerable to propaganda and misinformation, remain underrepresented, impacting the generalizability of existing models. overreliance on individual models: transformer models like mbert, xlm-roberta, and mt5 have shown strong standalone performance, but their reliance on pretraining data biases can limit robustness in real-world scenarios. ensemble methods are underexplored in mitigating these weaknesses, particularly in balancing class-specific performance. ## materials and methods this section describes the methodology for detecting multilingual propaganda using transformerbased embeddings and ensemble learning. the process involves balancing the dataset to ensure equal class representation and preprocessing steps like tokenization. transformer models-mbert, xlm-roberta, and mt5-were fine-tuned to extract multilingual embeddings, capturing linguistic and contextual nuances. these embeddings were then used in classification frameworks to evaluate the models' effectiveness in handling multilingual propaganda detection tasks. ## conclusion this study implemented and evaluated three transformer-based models-mbert, xlm-roberta, and mt5-to address the challenge of multilingual propaganda detection. by leveraging these models' multilingual capabilities and employing advanced preprocessing techniques, including data balancing, we conducted a comprehensive assessment of their performance. key evaluation metrics such as accuracy, precision, recall, f1-score, and loss values were used to determine the models' effectiveness across diverse propaganda categories. the mt5 model consistently outperformed its counterparts, achieving an outstanding accuracy of 99.61% and an f1-score of 0.9961, demonstrating its exceptional ability to handle multilingual content and detect propaganda with high precision. its text-to-text framework allowed it to effectively model linguistic nuances across multiple languages, making it the most reliable model for this task. the mbert model also showcased strong performance, achieving an accuracy of 92.0% and an f1-score of 0.92, excelling in the "not propaganda" and "not applicable" categories. despite these results, it showed room for improvement in more nuanced categories. the xlm-roberta model achieved a respectable accuracy of 89.51%, with an f1-score of 0.8934, but faced challenges with generalization, as evidenced by its higher validation loss compared to the other models. the results emphasize the transformative potential of transformer-based embeddings in multilingual propaganda detection. while mt5 emerged as the most effective model, mbert demonstrated computational efficiency and solid performance, making it a viable choice for practical applications. on the other hand, xlm-roberta highlighted areas for future improvement, particularly in adapting to complex multilingual tasks. limitations and future work: these future directions aim to refine and enhance the capabilities of multilingual propaganda detection, expanding the models' adaptability, accuracy, and generalization across various contexts and languages. hyperparameter optimization and fine-tuning: future work could explore the fine-tuning of hyperparameters such as learning rate, batch size, and number of epochs for each model, especially mbert and xlm-roberta. optimizing these parameters could lead to improved performance, particularly in terms of accuracy, precision, and recall across different propaganda categories. exploring advanced ensemble techniques: while individual models like mt5, mbert, and xlm-roberta demonstrated strong performance, future research could investigate the use of more advanced ensemble methods, such as stacking and boosting, to combine the strengths of multiple models. this could help in improving performance, especially in identifying subtle propaganda elements that individual models may miss. cross-lingual transfer learning: one promising direction is to explore cross-lingual transfer learning by leveraging pre-trained multilingual models to better handle low-resource languages.