Appeal for Attention at S em E val-2023 Task 3: Data augmentation extension strategies for detection of online news persuasion techniques (2023)

Abstract: In this paper, we proposed and explored the impact of four different dataset augmentation andextension strategies that we used for solving the subtask 3 of SemEval-2023 Task 3: multi-label persuasion techniques classification in a multi-lingual context. We consider two types of augmentation methods (one based on a modified version of synonym replacement and one based on translations) and two ways of extending the training dataset (using filtered data generated by GPT-3 and using a dataset from a previous competition). We studied the effects of the aforementioned techniques by using theaugmented and/or extended training dataset to fine-tune a pretrained XLM-RoBERTa-Large model. Using the augmentation methods alone, we managed to obtain 3rd place for English, 13th place for Italian and between the 5th to 9th places for the other 7 languages during the competition.

## introduction the subtask 3 of semeval-2023 task 3 @xcite focuses on detecting the persuasion techniques used in online news, in a multilingual context. this task is significantly important as it may enable one to identify attempts of manipulation and misinformation, attacks on reputation, persuasive language and logical fallacies in textual data. the data offered by the organizers of the competition covered nine languages: english, french, italian, spanish, german, georgian, greek, polish and russian. for three of the languages @xcite only the test data was available, to promote the development of languageagnostic solutions in a zero-shot learning scenario. to solve this subtask, we used a pre-trained multi-lingual transformer-based model: xlm-roberta-large @xcite . we further fine-tuned this model on an augmented version of the dataset which encompassed the provided training data for the six languages. we used two augmentation techniques that had the purpose to balance the amount of data associated either with each persuasion technique or with each language. the first augmentation method consisted in creating new articles by translating the data from one language to all of the other languages using a translation api. the second technique involved employing a special procedure to balance the data corresponding to each persuasion technique; in this case, the augmentation consisted in creating similar articles by replacing words from the paragraphs with synonyms from the nltk wordnet corpus 1 . by using data augmentation, we managed to score 3 rd on the english language, and between 5 -13 th places on all the other languages on the official semeval-2023 test set leaderboard. post-competition, we considered extending the training dataset in two ways: by using a dataset from a similar previous competition @xcite and by using data generated with gpt-3 @xcite . we performed experiments with all four techniques of dataset extension and augmentation and summarized which methods determined our model to obtain the highest f1 score for each language, providing also a comparative analysis with our results during the competition. while most novel approaches focus on the architecture of the used model(s), for this subtask we tried to emphasize the importance of using augmented and/or additional training data in influencing a model's predictions and performance. the complete code for our proposed solution, including the data generation and augmentation steps, can be found on github foot_1 . ## background 2.1 related work to set a starting foundation for our work, we analyzed the state of the art techniques presented at the semeval's previous proceedings on similar (sub)tasks: the detection of persuasion / propaganda techniques in textual datasets @xcite , @xcite . it is important to mention that the provided data for these previous competitions was exclusively in english. we observed that the most performant solutions were ensemble based, combining multiple types of pretrained transformers: @xcite , @xcite , or combining fine-tuned transformer models with classical machine learning models, such as logistic regression and decision trees in the case of @xcite . we discovered that many contestants @xcite employed augmentation techniques to balance the given datasets and to improve the performance of their models. back-translation was one of the most commonly utilized augmentation methods to enrich the provided monolingual datasets. authors of @xcite chose as intermediate languages for translation dutch and russian, while german and russian was used in @xcite . in @xcite it was stated that repeated back translation could have produced many similar data instances, hindering the learning process for the complex proposed models. eda (easy data augmentation techniques for boosting performance on text classification tasks) @xcite was another popular utilized augmentation method, encompassing a series of four operations: synonym replacement (sr), random insertion (ri), random swap (rs) and random deletion (rs). however, the authors introducing eda mentioned that the techniques might bring negligible improvements when using pre-trained models (including transformer based models such as bert @xcite ). in @xcite , the authors explicitly stated that eda brought no significant improvements for the task of detecting persuasion techniques in texts. starting from these observations, we decided to tackle this year's current task by experimenting with various combinations of data augmentation and data extension techniques that might increase the performance of the transformer-based models. analyzing the results from the paper @xcite , we decided to try two augmentation techniques: synonym replacement based on part-ofspeech tags and translations from one language to all the other languages. we also considered extending the provided dataset with data generated and annotated by gpt-3 @xcite and with the dataset from a similar previous competition @xcite . ## experimental setup in our experiments, we utilized the google colab foot_4 and kaggle foot_5 platforms to fine-tune our xlm-roberta-large model. we cumulated the given train and dev splits, using 80% of it for training and 20% for validation. due to the dimensions of our model and the limited memory in our used environments, we performed a manual hyper-parameter tuning. we se-lected as final hyperparameters the ones that enabled our model to obtain the highest validation f1 micro score. to fine-tune the model on this multilabel classification task we used the m ultilabelclassif icationm odel class provided by the simple transformers library 9 . the official evaluation metric used for this subtask was the micro f1 score; however, the macro f1 score was also computed. ## conclusion in this paper, we propose and study the effect of four data augmentation and extension techniques to solve the problem of identifying persuasion techniques used in online news in a multi-lingual setup (subtask 3 of task 3 from the semeval 2023 competition). in case of augmentation, we utilize two methods: a variation of the classic synonym replacement operation that takes into account the words' part of speech and levenshtein distances and a translation based techniques that creates new articles by translating the data from one language to all of the other languages. we also consider extending the training dataset with data from a similar previous semeval task and with synthetic data generated with gpt-3. we explore the impact of these techniques by using the augmented and/or extended training dataset to fine-tune a pretrained xlm-roberta-large model. we manage to achieve good results on all 9 languages proposed in the competition. our results indicate that the quality and quantity of the training data have a major impact on a model's performance and that we can obtain major improvements by using suitable augmentation and extension techniques.