T eam A mpa at S em E val-2023 Task 3: Exploring Multilabel and Multilingual R o BERT a Models for Persuasion and Framing Detection (2023)

Abstract: This paper describes our submission to theSemEval 2023 Task 3 on two subtasks: detectingpersuasion techniques and framing. Bothsubtasks are multi-label classification problems. We present a set of experiments, exploring howto get robust performance across languages usingpre-trained RoBERTa models. We test differentoversampling strategies, a strategy ofadding textual features from predictions obtainedwith related models, and present bothinconclusive and negative results. We achievea robust ranking across languages and subtaskswith our best ranking being nr. 1 for Subtask 3on Spanish.

## introduction task 3 at semeval-2023 @xcite promotes research on propaganda detection by providing a shared task with annotated data for detecting framing (subtask 2) and persuasion techniques (subtask 3) in online news from various languages; namely, english, french, german, italian, polish, and russian. in addition, three additional surprise languages were included in the test phase: spanish, greek, and georgian. we have submitted predictions for each of the nine languages on both subtask 2 and subtask 3, with both subtasks being multi-label classification problems. we obtain robust results across the different languages by fine-tuning roberta models @xcite . our best rankings on subtask 3 are nr. 1 on spanish and nr. 2 on french and russian. on subtask 2, we rank nr. 2 on english and greek. in our paper, we study how to get good performance on multi-label classification using finetuning of transformer-based models on both subtasks and across languages. more concretely, we conduct experiments and report results on the following three questions. first, we explore when to use an english backbone with less annotated data and when to use a multilingual backbone to exploit a larger set of multilingual annotated data. second, we study different oversampling strategies. third, we try to directly include textual features in training for persuasion detection of the broader categories of the fallacy of ethos, pathos and logos. we report both positive and negative results. we made our final training scripts available on github foot_0 . ## overview of our approach our approach is to fine-tune models on the transformer architecture @xcite , more concretely, the roberta architecture @xcite . roberta is an optimized pretraining of the bert architecture @xcite . a roberta model is preferred over a bert model regarding performance on downstream-tasks in english shown by @xcite , and @xcite shows that the multilingual xlm-roberta is prefered over mbert. we, therefore, use the english pre-trained model roberta large @xcite , and the multilingual pre-trained model xlm-roberta (xlm-r) @xcite in our experiments. the english roberta is pre-trained on 160gb of text from five different datasets. the multilingual xlm-r is trained on a total of 2.5 tb of data from filtered common-crawl data including 100 different languages with 301gb of english text data, 278gb of russian, 67gb of german, 57gb of french, 45gb of polish and 30gb of italian. hence, we notice a difference for our target languages in how much specific language data the backbone model is trained on. we also note a difference in model size between xlm-r with 550m parameters and roberta large with 355m parameters @xcite . in our experiments, we examine when to use the english backbone with less annotated training data available and when to use the multilingual backbone with more annotated data available. we notice a large difference in the number of training samples for the different languages in the multilingual xlm-roberta. in our experiments, we test the difference between oversampling the language-specific data for either the language with the highest presence in the pretraining versus the language for the matching target data at hand. we also oversample the labels with a low count in the training data in an attempt to account for the class imbalance in the training data. when oversampling, we include every sample with the respective label twice in the training data. we test a model on the english data, which first uses the models from @xcite to predict the fallacies of ethos, pathos and logos on subtask 3 paragraphs. we then include these predictions as features in the text input before fine-tuning on subtask 3. we use the textual features from ethos as a 'credibility attack', from pathos as 'emotional appeal' and from logos as 'logical fallacy'. in this setup, the new text input becomes: "[credibility attack] [emotional appeal] [logical fallacy]: paragraph text", where the [] input depends on whether the respective prediction is positive or not. ## results on test set we report our test results from the official leaderboard. the task organizers released the dev labels with the encouragement to include them in the training of the final model. we thus retrained our models before submitting to the test leaderboard. all models are trained using the oversampling strategy few. on subtask 2, we train a single multilingual model and use this for predictions across all languages. on subtask 3, the predictions in the official test phase come from different models: the submitted predictions for fr, ge, it, po and ru come from individual xlm-r models trained with oversampling of the target language. the predictions for the three new languages (gl, es and ka) come from a xlm-r model with oversampling of english. for english, they come from a combination of a roberta model and a xlm-r model (oversampling english) (see subsection 4.2 for abbreviation and explanations) 4 . the results for the leaderboard are replicated in after the official end of the test phase, we evaluate the prediction for all languages on subtask 3 made by an xlm-r model with oversampling few -the counterpart of the submitted model for subtask 2 -to study the impact of opting for the simplicity of using a single model per subtask for all languages. in ## discussion we have tried to account for label imbalance by oversampling. we see a small positive effect on f1macro scores in the experiments in in addition, lastly, we note as limitation of the above analysis for the f1-macro in the dev set, that some classes have very few samples, which makes the results and their discussion less certain. ## conclusion we experiment with fine-tuning different roberta models for multi-label classification to achieve robust results on both subtasks and across languages. we noted that small changes in the learning-rate could lead to unstable results where the models either scored high or zero depending on the seed. we tried to mitigate class imbalance in the training data. however, we notice that some classes have so few training instances in comparison to their semantic complexity, that finetuning roberta models is not a good method fit for these classes. we explored different oversampling strategies and tried to include textual features from other models in the training. we got inconclusive results for the first and negative results for the second. however, we ended up getting robust results on both subtasks.