S ina AI at S em E val-2023 Task 3: A Multilingual Transformer Language Model-based Approach for the Detection of News Genre, Framing and Persuasion Techniques (2023)

Abstract: This paper describes SinaAI’s participation in SemEval-2023 Task 3, which involves detecting propaganda in news articles across multiple languages. The task comprises three sub-tasks: (i) genre detection, (ii) news framing,and (iii) persuasion technique identification. The employed dataset includes news articles in nine languages and domains, including English, French, Italian, German, Polish, Russian, Georgian, Greek, and Spanish, with labeled instances of news framing, genre, and persuasion techniques. Our approach combines fine-tuning multilingual language models such as XLM, LaBSE, and mBERT with data augmentation techniques. Our experimental results show that XLM outperforms other models in terms of F1-Micro in and F1-Macro, and the ensemble of XLM and LaBSE achieved the best performance. Our study highlights the effectiveness of multilingual sentence embedding models in multilingual propaganda detection. Our models achieved highest score for two languages (greek and italy) in sub-task 1 and one language (Russian) for sub-task 2.

## introduction semeval 2023 task 3 aims to evaluate the ability of computational models to detect three important aspects of news articles: genre, news framing, and persuasion techniques. the provided dataset includes news articles from various sources, such as newspapers and online media, which have been annotated with labels indicating the framing, genre, and persuasion technique. participants are required to submit models that can accurately identify these labels for each article. this task is significant as it provides a means to automatically analyze and classify news articles, which can be useful in various applications such as content filtering and recommendation systems @xcite . a better understanding of how news articles are framed, what genre they belong to, and what persuasion techniques are used can improve our ability to analyze and interpret news content. this has important implications for the field of media studies and society as a whole, as it can help us better understand the role of the media in shaping public opinion @xcite . we approached the news article classification task by fine-tuning multilingual language models such as xlm, labse, and mbert, using ensembling and data augmentation techniques. our experimental results show that xlm outperforms the other models in terms of f1-micro and f1-macro. moreover, the ensemble of xlm and labse achieved the best performance. our models achieved the highest score in two languages (greek and italian) for sub-task 1, and in one language (russian) for sub-task 2. ## results the results of the models for sub-tasks 1, 2, and 3 on the validation set are presented in tables 2, 3, and 4, respectively. in sub-task 1, the xrl model with data augmentation achieved the highest macro-f1 of 0.65 and a micro-f1 of 0.72. in subtask 2, the ensemble model of xrl and labse outperformed other models based on the validation set. in sub-task 3, the xlm single model achieved a micro-f1 of 0.60 and a macro-f1 of 0.13. the performance of the models in all sub-tasks over the official test set is reported in tables 5 , 6 , 7, and 8. however, the official results of subtasks 2 and 3 were not reported due to a minor error in output formatting at the submission time. in sub-task 1, xrl with augmentation achieved the best score among all teams for two languages, greek and italian. in sub-task 2, the performance of three ensemble models was reported since their performance on the validation set was close to each other. the ensemble of xrl and labse without augmentation outperformed other models in the russian language. ## discussion and conclusions semeval 2023 subtask 3 aimed to classify news articles in multiple languages in three different subtasks: news framing, genre detection, and persuasion technique identification. our dataset comprised news articles from various domains and nine different languages, with labeled instances of news framing, genre, and persuasion techniques. we showed that xlm outperformed the other language models in f1-micro and f1-macro scores in subtask 1 when used alone but performed even better when combined with mbert or labse in an ensemble approach in task 2. consequently, we used the most effective models to train for different tasks, analyzed their performance across languages and propaganda techniques, and explored data augmentation techniques to improve their performances. our study highlights the effectiveness of multilingual sentence embedding models in detecting multilingual propaganda and provides crucial insights into different models and data augmentation techniques. however, due to limited computational resources, we had to limit the number of experiments we ran for hyper-parameter optimization which may have affected our performances.