UMUT eam at S em E val-2023 Task 3: Multilingual transformer-based model for detecting the Genre, the Framing, and the Persuasion Techniques in Online News (2023)

Abstract: In this manuscript, we describe the participation of the UMUTeam in SemEval-2023 Task 3, a shared task on detecting different aspects of news articles and other web documents, such as document category, framing dimensions, and persuasion technique in a multilingual setup. The task has been organized into three related subtasks, and we have been involved in the first two. Our approach is based on a fine-tuned multilingual transformer-based model that uses the dataset of all languages at once and a sentence transformer model to extract the most relevant chunk of a text for subtasks 1 and 2. The input data was truncated to 200 tokens with 50 overlaps using the sentence-transformer model to obtain the subset of text most related to the articles’ titles. Our system has performed good results in subtask 1 in most languages, and in some cases, such as French and German, we have archived first place in the official leader board. As for task 2, our system has also performed very well in all languages, ranking in all the top 10.

## introduction this working note describes the participation of the umuteam in the semeval 2023 shared task 3 @xcite , focused on identifying different aspects of news articles, including document category, framing dimensions, and persuasion technique in a multilingual setup. online media monitoring in multiple languages is an essential capability for organizations and companies that analyze and report on specific topical issues in different countries. however, due to the large amount of data to be processed and the interest in comparing in depth how topics of interest are treated and presented in different countries and media, the automation of media analysis processes is necessary. automating media analysis processes can help reduce time and effort while providing a more complete and detailed overview of media coverage in different languages and geographies. it can also help identify trends, patterns, and deviations that would otherwise be difficult to detect. therefore, this task focuses on recognizing different complementary aspects of a persuasive text, such as genre and persuasive techniques used to influence the reader. advances in deep learning have enabled the development of very sophisticated and deep machine learning models. these include transformer-based models, a neural network architecture that has a weighty impact on nlp, making them suitable for tasks such as text classification, automatic translation, and sentiment analysis @xcite . in addition, they are trained on large datasets and are able to learn complex features in different languages. multilingual transformer-based models are a type of model that has been trained in several languages and therefore have the ability to deal with the linguistic task involving more than one language. thus, our approach for subtasks 1 and 2 is to fine-tune a multilingual model called xlm-roberta using a dataset of all languages at once and with particular pre-processing techniques that will be explained in more detail in section 3. our system has achieved good results in subtask 1 in most languages, such as french and german, in which we have achieved first place in the official ranking. as for task 2, our system has also performed very well in all languages, ranking in all the top 10. the remainder of this paper is organized as follows. section 2 provides a summary of important details about the task setup. section 3 offers an overview of our system for two subtasks. section 4 presents the specific details of our systems. section 5 discusses the results of the experiments, and finally, the conclusions are shown in section 6 609 2 background the semeval-2023 task 3 challenge consists mainly of the following sub-task: it is a single-label task at the article level. it is a multi-label task at the paragraph level. we have only used the dataset given by the organizers, which consists of a set of news articles in 6 languages (english, french, german, italian, polish, and russian). the dataset has been compiled from 2020 to mid-2022, revolving around a fixed set of widely discussed topics, such as covid-19, global climate change, abortion, migration, preparations leading up to the russian-ukrainian war and events related to and triggered by it, as well some country-specific local events, such as elections, etc. the media selection has covered both mainstream media in each country, such as alternative news and web portals. some news aggregation engines were used, such as google news or europe media monitor (emm), a large-scale, multilingual, near-real-time news aggregation and analysis engine. moreover, online services were used, such as newsguard and mediabiasfactcheck, which rank sources according to their likelihood of spreading misinformation or disinformation. it is worth mentioning that task's organizers divided the corpus of all languages into three parts: training, development, and testing. however, the size of the training and development datasets is relatively small, as shown in ## system overview for solving the subtask 1 and 2 proposed, we build a system, which architecture is depicted in figure 1 . in a nutshell, our system works as follows. first, the training and evaluation set of all languages was merged, and a preprocessing stage was performed to clean the dataset. second, a sentence-transformers model was used to extract the part of the news item most related to the title. third, we fine-tuned the xlm-roberta model for different classification tasks. finally, we evaluated the model with test datasets for each language. ## experimental setup finally, the dataset for training has a total of 1234 news items and 258 for development for both subtask 1 and subtask 2. to obtain the classification model for each task, we conducted a hyperparameter optimization stage using raytune ( @xcite ) with a tree of parzen estimators (tpe) to select the best combination of the hyperparameters over 10 trials. the hyperparameters evaluated, and their interval range are: (1) weight decay (between 0 and 0.3), (2) training batch size @xcite ), (3) number of training epochs ([1-20]), and (4) learning rate (between 1e-5 and 5e-5). for subtasks 1 and 2, the best subset of hyperparameters for xlm-roberta is 15 epochs, 0.01 in weight decay, a batch training of 8, and a learning rate of 1e-5. the input size of the text is also an important factor that significantly affects the performance of our system, so we tested taking fragments of different sizes with the phrase transformer model, such as 200, 300, 400, and 500 tokens and found that the best performance was obtained with fragments of 200 tokens with 50 overlap as input. the primary evaluation metric used in the task is the f1 scores. for sub-task 1, the macro-f1 is the official evaluation measure. for sub-task 2, the evaluation metric is micro-f1, in this case, being a multi-label classification, the seqeval 3 framework has been used as the evaluation framework when performing hyperparameter optimization and micro-f1 comparisons. ## results our system was evaluated using the validation split and trained with the set of all languages at once. ## conclusion in this working note, we have described the participation of the umuteam in the shared task 3 of semeval 2023. in this shared task, the participants were required to detect different aspects of news articles and other web documents, such as document categories, framing dimensions, and peras for task 2, our system has also performed well in all languages, ranking in all the top 10. in the future, we hope to explore other multilingual models and fine-tuning a sentence transformer model using the news dataset to improve the model performance and perform an ablation study on fragment size to analyse deeper the impact of this parameter. besides, we want to incorporate and to extend the spanish split of the evaluation dataset to the news related to the financial domain @xcite , hate-speech detection @xcite and political ideology @xcite .