FTD at S em E val-2023 Task 3: News Genre and Propaganda Detection by Comparing Mono- and Multilingual Models with Fine-tuning on Additional Data (2023)

Abstract: We report our participation in the SemEval-2023 shared task on propaganda detection and describe our solutions with pre-trained models and their ensembles. For Subtask 1 (News Genre Categorisation), we report the impact of several settings, such as the choice of the classification models (monolingual or multilingual or their ensembles), the choice of the training sets (base or additional sources), the impact of detection certainty in making a classification decision as well as the impact of other hyper-parameters. In particular, we fine-tune models on additional data for other genre classification tasks, such as FTD. We also try adding texts from genre-homogenous corpora, such as Panorama, Babylon Bee for satire and Giganews for for reporting texts. We also make prepared models for Subtasks 2 and 3 with finetuning the corresponding models first for Subtask 1.The code needed to reproduce the experiments is available.

## introduction non-topical text classification includes a wide range of tasks aimed at predicting a text property that is not connected directly to a text topic, so it is not simply detected via keywords, for example, predicting a text style, politeness, difficulty level, the age or the first language of its author, etc. automatic genre identification @xcite is one of the standard problems of non-topical text classification. it can be applied in many areas such as information retrieval, language teaching or linguistic research to find texts of specific genres for any topic. semeval shared task 3 consists of 3 subtasks linked to the concept of non-topical text classification. subtask 1 (news genre categorisation) is aimed at predicting expression of opinions, objective news reporting, or satire on the text level. subtask 2 is on prediction of text framing with the possibility of multiple labels per article. subtask 3 is about detecting the kind of persuation techniques on the paragraph level. nlp shifted dramatically when large pre-trained models emerged. multilingual bert @xcite shows impressive results on a variety of languages and often it even manages to beat the monolingual models. xlm-roberta @xcite is a more powerful multilingual model. it is trained on more than 100 languages and uses a vocabulary with a higher number of tokens -more than @xcite . a problem with many non-topical classification setups concerns robustness. when a classifier sees a text that differs drastically from any text in the train set, it tends to misclassify it. and quite often, it is accompanied by a low probability of the max probable class. a recent study by @xcite represents a metric of confidence of prediction that estimates how certain the classifier is on the given text. it could be helpful to understand whether the label from classifier is trustworthy and, if not, gives signal to replace it. in some tasks, it is rational to replace the uncertain predictions with the majority class label. even the best classifiers have their own flaws that could result in poor performance on some set of texts. in order to reduce the flaws, it could be helpful to use ensembles of diffirent architectures. a recent study @xcite shows that ensembles not only have higher accuracy on diverse test sets, but also give more confident predictions. these are some of the ideas we tested by applying them to the shared task @xcite . ## subtask 1: news genres we experiment with all the languages and report our results on the dev sets. in all the models we use, the input size is restricted to k tokens (in most cases, @xmath0. to tackle it, we take first k tokens 1 549 in any text we forward to the model. we do not make any additional data splits and use the dev sets from the task statement to validate and compare our approaches. the weights for the ensembles are also selected on the corresponding dev sets. we first train separately the individual models. then we select the coefficients for the models and make ensemble as a linear combination of the class probabilities. ## subtask 2 in this subtask, we use the architectures that show the best results on subtask 1 among the individual models. it is mbert for russian, english, french and italian, polish bert for polish, german elec-tra for german. unlike in subtask 1, we have not fine-tuned the models on the additional data. in multilabel classification it is crucial to select the optimal thresholds for the classes. we experimented with a range of thresholds from 0.1 to 0.6 and selected the best one on the dev set. in order to simplify selection of the optimal hyperparameters, we use the same threshold all the labels. however, of course, it could be potentially improved by choosing different thresholds for each label. we found out the following threshold to be the best: 0.3 -for italian, french and russian, 0.2 -for polish and german, 0.1 -for english. ## subtask 3 we use the best architectures pre-trained on subtask 1 and fine-tune them further on the training set for subtask 3. it improves the f1 score compared to the baseline, though the difference is not dramatic. for instance, the macro f1 score for the english language rises from 0.22 to 0.25. the process of selection of the optimal thresholds is identical to that for subtask 2. ## conclusion in our study, we have tried multiple techniques including 1. usage of additional relevant corpora for finetuning, 2. usage of bert-based classifiers pre-trained on a different classification task, 3. computing of confidence of prediction and replacing the predictions with low condifence, 4. ensembles of classifiers. the available training corpora were not sufficient. when an additional relevant corpus is available (either with additional data in a specific genre, such as news, or a general purpose corpus, such as ftd), it always helps. we show that upsampling also improves the metrics for every language in the dataset. this implies that in tasks with few labeled texts available it is crucial for training data to be balanced. another important conclusion is that for most languages multilingual models are better than monolingual models. the only exceptions are german and polish. we also show that for most languages (4 out of 6) ensembles give higher performance than the individual models.