S heffield V era AI at S em E val-2023 Task 3: Mono and Multilingual Approaches for News Genre, Topic and Persuasion Technique Classification (2023)

Abstract: This paper describes our approach for SemEval- 2023 Task 3: Detecting the category, the fram- ing, and the persuasion techniques in online news in a multilingual setup. For Subtask 1 (News Genre), we propose an ensemble of fully trained and adapter mBERT models which was ranked joint-first for German, and had the high- est mean rank of multi-language teams. For Subtask 2 (Framing), we achieved first place in 3 languages, and the best average rank across all the languages, by using two separate ensem- bles: a monolingual RoBERTa-MUPPETLARGE and an ensemble of XLM-RoBERTaLARGE with adapters and task adaptive pretraining. For Sub- task 3 (Persuasion Techniques), we trained a monolingual RoBERTa-Base model for English and a multilingual mBERT model for the re- maining languages, which achieved top 10 for all languages, including 2nd for English. For each subtask, we compared monolingual and multilingual approaches, and considered class imbalance techniques.

## introduction with the rise of opinion-manipulating news and misinformation surrounding covid-19, elections and wars, the task of propaganda and hyperpartisan detection has received much attention over the last five years. since 2019, various semeval tasks have addressed detecting hyperpartisan @xcite , sarcasm @xcite , and persuasion techniques in textual and multimodal data @xcite . this task @xcite can be seen as an extension of the latter two tasks, suggesting an expanded ontology of persuasion techniques and addressing other related aspects of persuasion, such as satire, opinionated news, and framing detection. the three subtasks presented in this shared task are the detection of: 1) genre: opinion, objective reporting or satire; 2) framing techniques: 14 multilabel frames; 3) persuasion techniques: 23 multilabel techniques, which can be grouped into 6 high-level classes. the data consists of labelled training and development sets in english, french, german, italian, polish and russian, and unlabelled test sets in the same languages plus three zero-shot languages: spanish, greek and georgian. the main contributions of this paper are twofold: 1) evaluation of the viability of monolingual versus multilingual models for each of the subtasks; and 2) presentation of the models which ranked first in four subtask-language pairs, top three in 16 subtasklanguage pairs, and were within the top 10 for all. our approaches for the three subtasks differ, therefore we present each subtask separately in sections 3-5 respectively. an overview of the techniques used in each subtask is shown in ## background fine-grained propaganda technique classification was first introduced by @xcite , who suggested a multi-granularity network, where the lower and higher granularity tasks refer to the fragment and sentence-level classification respectively. other state-of-the-art approaches to this task used an ensemble of roberta models with class weighting, where some models perform a semi-supervised task of span detection @xcite and an ensemble of 5 different transformer models @xcite , namely bert @xcite , roberta @xcite , xlnet @xcite , deberta @xcite and albert @xcite . framing detection specifically has been addressed primarily for political news, with the models exploring unsupervised probabilistic topic models combined with autoregressive distributed-lag models @xcite , finetuning bert @xcite and multilingual bert (mbert) @xcite . the latter system is the closest to our task since it explores the multilabel multilingual setting and the effect of translating texts for use in monolingual models. however, it uses article headlines instead of the full texts as the classification data. the authors found that english bert base uncased trained on translated data and tested on the data in the target language often outperforms the multilingual model. we perform similar comparison experiments for all three subtasks of this shared task. @xcite performed a series of experiments comparing monolingual and multilingual approaches for hate speech detection and sentiment analysis and found that different tasklanguage combinations favour either monolingual and multilingual settings. the authors also concluded that data augmentation in the form of translation and task-adaptive pretraining (tapt) @xcite helps to further improve the results. another important task addressed in fake news detection is satire detection, with the methods ranging from convolutional neural networks (cnns) @xcite to adversarial training @xcite and bert-based architectures with long-short-term memory (lstm) @xcite and @xcite layers on top. bottleneck adapters adapters @xcite represent a family of techniques aimed at improving parameter efficiency in finetuning by freezing a pretrained model and inserting low-dimension adapter modules within each layer. houlsby et al. found that, despite training only 3.6% of the parameters compared to a full model, performance only decreased by 0.4%, while bapna and firat found that adapters produced comparable, or in some cases better, results. particularly relevant for our task is @xcite 's finding that adapter-based tuning of llms is particularly effective for low-resource and crosslingual tasks. for our system, we used two different configurations of the bottleneck adapter modules: 1) the original houlsby et al. bottleneck configuration, which places adapter modules after the multi-head attention block and feed-forward block of each transformer layer; 2) the pfeiffer et al. configuration, which places adapter modules only after the feed-forward block of each layer. chalkidis et al. (2021) also found that for xlm-r on the multieurlex dataset, training bottleneck adapters outperforms traditional full finetuning and improves zero-shot cross lingual capability. similarly to this task, the multieurlex dataset was used for multilingual multilabel classification, though it is significantly larger than the data in this task (covering 23 languages and classifying hierarchically from 21 to 567 labels). ## system description for subtask 1 3.1 system overview the system consisted of an ensemble of four models, comprising 1) three mbert models each finetuned using the organiser training set and 2/3 of the development set; 2) one frozen mbert model with a finetuned houlsby adapter foot_0 . the ensemble predictions were decided by majority vote, with rare tie cases handled by selecting the model with the best validation performance. full finetuning mbert base @xcite was finetuned on a shuffled combination of all languages. different to previous approaches @xcite , we chose the epoch with the best validation performance per-language, instead of overall, since the best overall epoch is not necessarily the best for a given language 3 . by using three identicallyconfigured models in the ensemble, the data sacrificed for model selection can be rotated between them, so overall no data is truly unseen. adapter model a houlsby bottleneck adapter was applied to a pretrained mbert base model, with a reduction factor of 8 (i.e. @xmath0, using the adapterhub @xcite framework. the mbert model parameters were frozen, so only the adapter and classification head parameters were trained. data preprocessing since task data is obtained from webpages, it often contains unwanted content, such as hyperlinks, account handles, dates and author biographies. we applied the preprocessing described in appendix b to remove this content. ## conclusion we presented three systems aimed at solving three subtasks within semeval-2023 task 3. our systems applied a variety of state-of-the-art techniques including adapters and tapt, and consistently achieved a high rank across all available languages, including zero-shot low-resource languages. we additionally presented an analysis of the viability of monolingual vs multilingual approaches for each subtask, and found that the results of the comparison vary depending on the subtask. for subtask 1, multilingual transformer models demonstrate better average performance than monolingual models with translations. a similar effect was observed for subtask 2 and subtask 3 where multilingual settings achieved better performance than monolingual ones in all languages except english. we found the impact on bottleneck adapters to be unpredictable across tasks -despite performing on average better for monolingual models in subtask 1, they were more beneficial for multilingual models in subtask 2 (and hindered monolingual performance). finally, we presented post-competition findings, which suggest that subtask 2 would have benefited from a zero-shot prediction using multilingual models, while subtask 1 could have achieved much better results with the 'translate-test' approach. further analysis of this will be possible when test labels are released.