Team QUST at S em E val-2023 Task 3: A Comprehensive Study of Monolingual and Multilingual Approaches for Detecting Online News Genre, Framing and Persuasion Techniques (2023)

Abstract: This paper describes the participation of team QUST in the SemEval2023 task3. The monolingual models are first evaluated with the under-sampling of the majority classes in the early stage of the task. Then, the pre-trained multilingual model is fine-tuned with a combination of the class weights and the sample weights. Two different fine-tuning strategies, the task-agnostic and the task-dependent, are further investigated. All experiments are conducted under the 10-fold cross-validation, the multilingual approaches are superior to the monolingual ones. The submitted system achieves the second best in Italian and Spanish (zero-shot) in subtask-1.

## introduction task 3 @xcite expects the participants to develop algorithms to automatically detect the news genre, framing and persuasion techniques in a multilingual setup as shown in one of the major challenges of this task is that the model must have the ability to tackle the issues of imbalanced and insufficient training data. in subtask-1, the number of samples for labels 'reporting', 'opinion' and 'satire' are 269, 878 and 87 respectively. with only few 'satire' samples relative to other majority classes such as 'opinion', the training model will easily overfit on majority classes and not learn enough from minority ones. for example, many batches will have no 'satire' samples, so the gradients will be less informative during the training phase and result in poor performances in predicting the minority class. mean-while, previous studies @xcite demonstrate that the scale of training data could influence the model performance significantly. however, the total training samples are 1234 in subtask-1, which is not enough to effectively train a deep learning model from scratch. to address the above issues, traditional machine learning and deep learning methods are first evaluated with the under-sampling of the majority class in a monolingual setting. then, the pre-trained multilingual model is fine-tuned to learn the crosslanguage features between languages. specifically, fine-tuning is a common approach for utilizing the pre-trained language model (plm) on many downstream tasks. it typically replaces the original output layer with a new task-specific layer and then fine-tunes the complete model. to model the features representation of news articles across different languages, the xlm-roberta (xlm-r) @xcite is fine-tuned since it can processes all the languages existing in task 3, and typically outperforms other models such as mbert @xcite . in addition, this study also calculates the sample weights and class weights to combat the data imbalance. the sample weights enable a training batch that contains represents a good spread of the dataset. the class weights are able to penalize the misclassification made from the minority class by setting a higher class weight and reducing the weight for the majority class at the same time. two types of fine-tuning approaches, taskagnostic and task-dependent, are conducted on three subtasks. a task-agnostic strategy is a standard approach that fine-tunes the plm on each subtask individually. this study also conducts a task-dependent strategy that fine-tunes the plm from subtask-1 to subtask-3, since all the training samples have a similar content except the labels which are different. article-id (para-id) text labels t1 t2 t3 710376094 (4) ...the asteroid named 3200 phaethon ... is classified as "potentially hazardous" by the minor planet center... opinion security and defense, quality of life repetition 23114 (17) les études de l'insee et de l'ined confirment en outre que l'ascenseur social fonctionne...chez les natifs (quatrième génération et plus)... reporting fairness and equality loaded language 2672 (3) ventisei milioni di cittadini chiusi in casa. reclusi. ai domiciliari. gli abitanti di shanghai... sono sottoposti a un lockdown severissimo satire policy prescription and evaluation, quality of life loaded language, name calling-labeling ## system description the submitted system is a standard fine-tuning approach by adding a linear classifier to project the [cls] embedding to an unnormalized probability vector over the output classes. to address the imbalanced data issue, the class weights enable more weights to be assigned to the minority class in the cross-entropy loss. therefore, the assigned weights are able to provide a higher penalty to the minority class and the algorithm could focus on reducing the errors for the minority class. furthermore, the 'satire' samples can be found with only 7% of the training set. this might result in only few or even no minority class samples appearing in the training batch. to tackle this, the sample weights aim to calculate each sample's weights over the entire dataset, and then apply a weight to each sample to ensure that the majority class will have a smaller weight, or vice versa. the sample weights are also calculated in subtask-2 and subtask-3 except for the class weights. after training with all the positive strategies, early stopping is applied to save the best model in each fold. eventually, the top 3 best models from the 10-fold cv are ensembled for the final prediction in each subtask. ## conclusion this paper extensively describes the monolingual and multilingual approaches in detecting news genre, news framing and persuasion techniques. to address the issue of imbalanced data, the majority classes are first under-sampled at the early stage of the task. the traditional machine learning models and deep learning models are evaluated in a monolingual setup. in order to learn the shared information between all languages, a multilingual model is built in the later stage of the task. the xlm-r is fine-tuned with the pre-calculated class weights and sample weights to combat the imbalanced data. in addition, two types of fine-tuning strategies, the task-agnostic and the task-dependent, are evaluated respectively. the proposed method is the second best system in italian and in spanish (zero-shot) subtask-1. in future work, the minor-ity class sample can be back-translated from other languages to augment the data size.