MLM odeler5 at S em E val-2023 Task 3: Detecting the Category and the Framing Techniques in Online News in a Multi-lingual Setup (2023)

Abstract: System Description Paper for Task 3 Subtask 1 and 2 of Semeval 2023. The paper describes our approach to handling the News Genre Categorisation and Framing Detection using RoBERTa and ALBERT models.

## introduction popular mass media comes together and shapes the perceptions of people and communities on numerous subjects of our society @xcite . monitoring mass media in different languages and understanding and analyzing niche as well as broad domain topics is essential in this day and age. with the increase in the amount of data being generated and processed, and the proliferation of news as well as information around us in recent times, it is essential to perform in-depth comparisons of how data is being put in front of the public by different mass media of different countries. there is also a need to promote and support the automation of this media analysis process @xcite . the main drive behind this task is to foster the development of methods and tools to support the analysis of online media content in order to understand what makes a text persuasive: which writing style is used, what critical elements are highlighted, and which persuasion techniques are used to influence the reader's mind and thinking. there is an essential need to identify and develop methods that identify what techniques are being used in media content. in a polarized media environment, partisan media outlets intentionally frame news stories in a way to advance specific political agendas @xcite . "even when journalists make their best efforts to pursue objectivity, media framing often favors one side over another in political disputes, thus always resulting in some degree of bias" @xcite . hence, a news framing analysis is helpful because it not only tells us whether a news article is left-or right-leaning (or positive or negative), but also reveals how the article is structured to promote a specific side of the political spectrum. framing is a phenomenon primarily studied and debated in the social sciences, where, for example, researchers explore how news media shape debate around policy issues by deciding what elements of an issue to emphasize, and what to exclude. theories of framing posit that these decisions give rise to thematic sets of interrelated ideas, imagery, and arguments, which tend to cohere and persist over time @xcite . in a widely cited definition, @xcite argues that "to frame is to select some elements of a perceived reality and make them more salient in a communicating text, in such a way as to promote problem definition, causal interpretation, moral evaluation, and/or treatment recommendation for the item described." the task focuses on extending the analytical functionalities of media analysis solutions to: automated detection of framing dimensions and persuasion techniques, and visualization of related statistics, etc. the three subtasks cover several complementary elements of what makes a text persuasive -the genre: opinion, report or satire, the framing: what critical elements highlight the rhetoric and the persuasion techniques that are used to influence the reader @xcite . this paper focuses on subtask 1 -news genre categorisation and subtask 2 -framing detection. this paper presents our system for shared task on detecting the category, the framing, and the persuasion techniques in online news in a multi-lingual setup @ semeval 2023. two bert-based architectures -roberta and albert -have been implemented for both subtask 1 and subtask 2. following are the links to the code for subtask 1 and subtask 2 1. github link for subtask 1 2. github link for subtask 2. ## task description the shared task focuses on annotating given new articles by: (a) new category, (b) framing techniques, and (c) persuasion techniques contained in the text. we have been provided with a multilingual dataset in the following 6 languages -english, french, italian, german, polish and russian -as training and validation data. we were also provided with 3 more languages -spanish, greek and georgian along with the previously mentioned languages as testing data. the aforementioned tasks are mentioned below. ## data description the input for all tasks were news and web articles in a plain text format. each article appears in one '.txt' file. all the articles had titles on the first row followed by an empty row. the content of the articles starts from the third row. articles in six languages were provided, that revolved around a fixed range of topics such as covid-19, climate change, abortion, migration, the build-up leading to the russo-ukrainian war, and events related and triggered by the aforementioned war, and some country-specific local events such as elections, etc. the media selection covers both mainstream media and alternative news and web portals, a large fraction of which were identified by fact-checkers and media credibility experts as potentially spreading mis-/disinformation. articles whenever possible were retrieved with the trafilatura library or other similar web-scraping tools, and otherwise were retrieved manually. news aggregation engines like google news or the europe media monitor were used for large-scale multi-lingual news aggregation, and online services such as newsguard and mediabiasfactcheck were used for the purposes of fact checking and all the annotations done to the articles to provide us with the labels were done by the organizers. each language provided in training and validation data has a folder with the articles and the labels associated with each article. the testing data doesn't include the labels. ## experimental setup and results in this section, we will discuss the effects of the roberta and albert model on subtask 1 and subtask 2. the parameters used by the overall model for both subtasks are different and will be mentioned in their specific subsections below. ## conclusion through this paper we describe our experiment and results over the semeval-2023 task 3 dataset consisting of news articles in six different languages. we have proposed the implementation of two strategies -roberta and albert. roberta out performed albert across subtask 1/2 and across both the metrics f1 macro and micro. the usage of nlpaug for 'text augmentation' and class weights to handle the unbalanced nature of the data had a vast improvement on the metrics. the best value of subtask 1 -0.66077 f1 macro -was achieved post semeval evaluation using roberta and class weights 2 on the dataset that included numbers and the best value of subtask 2 -0.47692 f1 micro -was achieved during the semeval evaluation using roberta and text augmentation 1 parameters. in the future, better data pre-processing techniques, fine-tuning of the nlp augmentation method, including increasing the numbers of words being substituted or the number of sentences being formed in the augmentation process, and fine-tuning the hyperparameters of roberta and albert models should be considered for better and more accurate results.