MELODI at S em E val-2023 Task 3: In-domain Pre-training for Low-resource Classification of News Articles (2023)

Abstract: This paper describes our approach to Subtask 1 “News Genre Categorization” of SemEval-2023 Task 3 “Detecting the Category, the Framing, and the Persuasion Techniques in Online News in a Multi-lingual Setup”, which aims to determine whether a given news article is an opinion piece, an objective report, or satirical. We fine-tuned the domain-specific language model POLITICS, which was pre-trained on a large-scale dataset of more than 3.6M English political news articles following ideology-driven pre-training objectives. In order to use it in the multilingual setup of the task, we added as a pre-processing step the translation of all documents into English. Our system ranked among the top systems overall in most language, and ranked 1st on the English dataset.

## introduction the news genre categorization subtask of semeval-2023 task 3 @xcite is a classification task that aims to identify the genre of a news article by considering three classes: opinion, reporting, satire. news genre categorization is important because it helps readers to quickly identify the type of content they are reading. it also helps to ensure that readers are not misled by the content they are consuming. the organizers provided articles in six languages for the training phase (english, french, german, italian, polish, and russian) and three surprise languages were revealed for the evaluation on the test sets @xcite . it is therefore a multilingual task that includes small datasets during training and zero-shot classification for surprise languages. our main strategy relies on the fine-tuning of politics @xcite , a state-of-the-art language model for political ideology prediction and stance detection, pre-trained on more than 3.6m english news articles. based on this, a wide variety of strategies were tested, including a sliding window to cover long documents, pooling methods, adapted loss functions, data splitting and hyperparameters tuning. moreover, as politics is monolingual, and to take advantage of the largescale pre-training in the multilingual setup, we propose as a pre-processing step to translate all articles from other languages into english. other interesting strategies have also been considered and tested but have been less successful on the development set, including inherently multilingual models such as xlm-roberta @xcite or longsequence transformers @xcite . we ranked 1st on the english test set with a macro-f 1 score of 78.43 (+16.8 points above the second ranked system), but were less successful on the other languages with an average of 6th place (out of about 20 systems, depending on languages). we draw several observations from our participation in this task: -the dominance of large-scale pre-trained and domain-specific language models over more adapted but not massively pre-trained approaches; -the need to further develop multilingual language models and datasets rather than having to rely on low performing translation systems for complex tasks; -the importance of considering the whole text in the case of long documents, whereas most language models are limited to the first 512 subtokens. ## background news genre categorization is a multi-class, singlelabel classification problem that aims to predict the type of content of a news article according to 3 classes @xcite : opinion an article is an opinion piece when it expresses the writer's opinion on a topic. it is usually written in a persuasive style and is meant to influence the reader's opinion on the subject. reporting news reporting is the genre that focuses on providing factual information. it aims to inform readers about the world around them and to provide an objective account of events. satire satire is a form of writing that uses exaggeration, absurdity and obscenity to mock and ridicule people, organizations or events. it is not meant to deceive, but to expose and criticize behavior that is wrong or immoral. satirical pieces often mimic real articles, using irony to provide humor. ## system overview our main strategy is based on @xcite 's domain-specific pre-trained language model poli-tics which has recently demonstrated great success on a variety of news articles ideology and stance prediction datasets. it has several advantages for this task: (i) it was massively pre-trained on more than 3.6m english news articles, (ii) it relies on the comparison of articles on the same story written by media of different ideologies, (iii) it demonstrated its robustness in few-shot learning scenarios. although predicting political ideology is not the same as detecting the genre of an article, our assumption is that these two notions overlap as in both cases there is a linguistic shift in the way information is conveyed. we can also observe similarities between the genre reporting and articles from the center political class, or between the genre opinion and left or right leaning articles. ## experimental setup first, the train, dev and test split articles for each language are translated using google translate. next, we aggregate the train and dev splits for each language to create an augmented english dataset containing 1234 articles in the training split and 358 in the development split, on which we train a single english model. we built on @xcite 's implementation, 4 and the huggingface transformers library @xcite . hyperparameters were set using grid search on the augmented development split for each of the models considered ( regarding alternative approaches, we fine-tuned xlm-roberta-large foot_3 using a sliding window on the same set of hyperparameters as poli-tics ( the official evaluation measure used is macro-f 1 . micro-f 1 is also given as a secondary evaluation measure on the official leaderboard. ## results these results should be taken with a grain of salt for the following reasons: test sets are small (30-70 instances) hence the huge variance between models and across languages. this problem is compounded by the chosen evaluation : macro-f 1 scores equalize the contributions of all classes to the final scores, meaning that one instance from the minority class classified one way or the other could swing the evaluation disproportionately. it would probably be informative to evaluate accuracy, and per class metrics, but gold labels were not provided for the test sets. this also makes it hard to estimate the distribution shift between train, development and test sets, although it is already apparent there are large differences between train and dev sets. ## conclusion we propose to take advantage of in-domain pretraining for detecting the genre of news articles. we fine-tuned the large-scale english language model politics using sliding windows on the multilingual corpus translated into english. our various experiments have shown the effectiveness of indomain pre-training as well as the importance of approaches adapted to the processing of long documents. we ranked 1st on the english dataset while being among the top systems overall.