Hitachi at S em E val-2023 Task 3: Exploring Cross-lingual Multi-task Strategies for Genre and Framing Detection in Online News (2023)

Abstract: This paper explains the participation of team Hitachi to SemEval-2023 Task 3 “Detecting the genre, the framing, and the persuasion techniques in online news in a multi-lingual setup.” Based on the multilingual, multi-task nature of the task and the low-resource setting, we investigated different cross-lingual and multi-task strategies for training the pretrained language models. Through extensive experiments, we found that (a) cross-lingual/multi-task training, and (b) collecting an external balanced dataset, can benefit the genre and framing detection. We constructed ensemble models from the results and achieved the highest macro-averaged F1 scores in Italian and Russian genre categorization subtasks.

## introduction as we pay more and more attention to the socially influencing problems like covid-19 and the russo-ukrainian war, there has been an increasing concern about infodemic of false and misleading information @xcite . in particular, cross-lingual understanding of such information is becoming more important due to polarization of political stances, economical decoupling and echo chamber effect in social media. to that end, @xcite put together semeval-2023 task 3 "detecting the genre, the framing, and the persuasion techniques in online news in a multi-lingual setup." the shared task aims to analyze several aspects of what makes a text persuasive and to foster development of building blocks for multilingual media analysis. creating annotated data for media analysis is time consuming thus we cannot assume that we can obtain training data of enough quality and quantity. to tackle the problem, we investigated and compared strategies for multilingual media analysis under a low-resource setting. through ex- * equal contribution tensive experiments, we found that (a) cross-lingual/multi-task training, and (b) collecting an external balanced dataset, can benefit the genre and framing detection. we constructed ensemble models from the results and participated in genre categorization (subtask 1) and framing detection (subtask 2) in six languages, where we achieved the highest macro-averaged f1 scores in italian and russian subtask 1. subtask 1: news genre categorization given a news article, a system has to determine whether it is an opinion piece, it aims at objective news reporting, or it is a satire piece. this is multi-class document classification and the official evaluation measure is macro average f1 score (macro-f1) over the three classes. subtask 2: framing detection given a news article, a system has to identify what key aspects (frames) are highlighted the rhetoric from 14 frames (see @xcite for the taxonomy and definitions). this is multi-label document classification and the official evaluation measure is micro average f1 score (micro-f1) over the 14 frames. ## external data for english genre categorization in a preliminary analysis of the english subtask 1 dataset, we found that label distribution is quite unbalanced and it is different in the training and the development data. therefore, we did not make any assumption about the distribution of the test data and decided to increase the number of rare labels in order to create a new, balanced dataset for english genre categorization. first, we undersampled articles from the training dataset for subtask 1 such that the numbers of articles for each label are equal, i.e., ten articles for each label. we referred to a survey on fake news detection datasets (d' @xcite and checked a total of 27 datasets to see if they can be converted to subtask 1 dataset format using the following criteria: label similarity we checked whether the labels ## cross-lingual multi-task transformers we utilized pretrained language models (plms) in a simple sequence classification setup @xcite . we employed xlm-roberta large foot_0 (conneau et al., 2020) and rembert 1 @xcite . for english, we also utilized roberta in multi-task training we simply took sum of losses from two tasks. since there exist articles that only have either of subtask 1 or 2 labels, we ignore predictions for missing labels from loss calculation. for cross-lingual training, we concatenate all articles. all parameters are shared for the same subtask in different languages. for preprocessing, we concatenated all sentences from each article and tokenized them using the default tokenizer for each plm. we truncated articles whose tokens do not fit onto each model's maximum context size. ## exploring cross-lingual multi-task strategies it is empirically known that further fine-tuning a model trained in a multi-task or cross-lingual setting on the target downstream task/language improves its performance @xcite . also, models tend to require different hyperparameters for different training paradigms or languages. hence, we decided to explore different multi-task and cross-lingual strategies through a series of random hyperparameter searches (figure 1 ). first, we ran a random hyperparameter search in cross-lingual and/or multi-task settings (stage i). regarding the resulting stage i models as an additional hyperparameter, we ran another random hyperparameter search to optimize the choice of the pretraining paradigm along with other hyperparam-2 @xmath0, and @xmath1. finally, we construct an ensemble for each language-subtask pair from all models in stage i and ii using their performance in the development dataset. unlike more sophisticated hyperparameter search methods, this approach has an advantage that we can compare and evaluate different training paradigms post hoc. the choice of the subtask 1 english datasets (section 3) is also incorporated as an additional hyperparameter. the hyperparameter search spaces are shown in appendix a.2. we used the official development dataset for each language-subtask pair in order to calculate and compare the performance of each model. ## related work there exist several studies on multilingual and/or multi-task learning in the context of media analysis. @xcite applied cross-lingual training to multilingual false information detection and showed that cross-lingual training results in on par performance to monolingual training. @xcite annotated tweets in multiple languages with multiple choice questions regarding covid-19 disinformation. they fine-tuned mbert model @xcite in a crosslingual setup and in a multi-task setup, and found that the results are mixed in terms of their benefits. the semeval 2023 task 3 dataset @xcite exhibits significantly different properties to the datasets previous works used and we found that cross-lingual and/or multi-task learning help in this dataset. we also carried out more thorough experiments than the previous works, which might have been the key to the improvement. there also exist different approaches for multilingual and low-resource settings in broader domains. @xcite proposed a method for training multilingual sentence embeddings and demonstrated its effectiveness in 50+ languages. in a low-resource setup, @xcite used a data augmentation approach that retrieves additional data from related datasets by automatically labeling them. we only explored bert-based approach in this work, but we wish to explore other approaches for cross-lingual multi-task learning in the future. ## conclusion in our participation to semeval-2023 task 3, we investigated different strategies for multilingual genre and framing detection. through the extensive experiments, we found that collecting an external balanced dataset can help genre categorization. we also find that cross-lingual and multi-task training can help both genre and framing detection and found that cross-lingual training is more beneficial for framing detection. we constructed ensemble models from the results and achieved the highest macro-f1 in italian and russian genre detection. for future work, we will investigate the effect of cross-lingual multi-task training on zero-shot language transfer (greek, spanish and georgian subtasks that we did not participate), as well as the effect on and benefit from training models on persuasion techniques detection (subtask 3). the model pools and the postprocessing methods were also considered as an option and we compared all the combinations of the ensemble methods, the