Hitachi at S em E val-2020 Task 11: An Empirical Study of Pre-Trained Transformer Family for Propaganda Detection (2020)

Abstract: In this paper, we show our system for SemEval-2020 task 11, where we tackle propaganda span identification (SI) and technique classification (TC). We investigate heterogeneous pre-trained language models (PLMs) such as BERT, GPT-2, XLNet, XLM, RoBERTa, and XLM-RoBERTa for SI and TC fine-tuning, respectively. In large-scale experiments, we found that each of the language models has a characteristic property, and using an ensemble model with them is promising. Finally, the ensemble model was ranked 1st amongst 35 teams for SI and 3rd amongst 31 teams for TC.

## introduction this paper shows our proposed system for the semeval-2020 task 11: detection of propaganda techniques in news articles @xcite . the goal of the task was to design a model to detect and classify propaganda. to this end, there are two subtasks: span identification (si) for predicting propaganda spans and technique classification (tc) for predicting propaganda technique types used for a given span. si can be assumed as a sequence labeling problem and tc as a multi-label classification problem. recent studies such as @xcite proposed employing bert @xcite , a pre-trained language model, for propaganda detection. since propaganda detection tasks require highly semantic understanding, leveraging such strong pre-trained language models is promising. however, new state-of-the-art pre-trained language models, such as xlnet @xcite and roberta @xcite , are being proposed rapidly, and there are no sufficient studies on them in the research on propaganda detection. revealing the ability of capturing propaganda semantics with state-of-the-art models could contribute to discussing future applications. therefore, we investigate state-of-the-art pre-trained language models (plms) for propaganda detection. we employ not only bert @xcite but other plms such as gpt-2 @xcite , roberta @xcite , xlm-roberta @xcite , xlnet @xcite , and xlm @xcite . the plms are fine-tuned by our proposed si and tc models with various hyperparameters. we also propose an effective ensemble method with stacked generalization, which is generally better than a naive average ensemble. our ensemble model was ranked 1st in si and 3rd in tc, showing that leveraging state-of-the-art plms is promising. we also empirically gained the following insights as described later. propaganda generally intends to promote an agenda or point of view with specific information such as biased or misleading descriptions. since detecting propaganda in text could be useful for further applications such as detecting fake news, research on propaganda detection has been attracting much attention. @xcite proposed a model for assessing the "level" of propaganda in an article @xcite focused on more fine-grained analysis, proposing a model for detecting propaganda spans and classifying their techniques. some recent studies @xcite @xcite @xcite utilized plms such as bert @xcite to detect fine-grained propaganda. our work is related to the studies of @xcite , which employed ensemble models with plms. different from these studies, we further investigate the number of plms. ## pre-trained language models (plms) in this paper, six types of transformer @xcite based plms were used. the reason behind this is to unify our implementation and evaluation. we provide a brief description of each plm: bert @xcite is the epoch-making transformer-based masked language model. we employ a pre-trained model called bert-large-cased-whole-word-masking. gpt-2 @xcite ) is a model that followed open ai gpt @xcite . gpt-2 has achieved state-of-the-art results in a zero-shot setting. roberta (liu et al., 2019) is a carefully fine-tuned bert-based model, where the authors investigated hyperparameters and training data size. roberta has achieved state-of-the-art results on major text benchmarks. we employ a pre-trained model called roberta-large. xlm-roberta (conneau et al., 2019) is a cross-lingual version of roberta. xlm-roberta has outperformed the cross-lingual bert. we employ a pre-trained model called xlm-roberta-large. xlnet (yang et al., 2019) employs a generalized autoregressive pre-training in transformers. xlnet has outperformed bert in major tasks. we employ a pre-trained model called xlnet-large-cased. xlm (lample and conneau, 2019) is a cross-lingual language model, introducing an unsupervised cross-lingual learning. we employ a pre-trained model called xlm-mlm-en-2048. ## propaganda detection models given a split sentence foot_0 , our si model predicts propaganda spans. the tc model in turn predicts propaganda techniques for given spans in a sentence. 2 the technique labels include loaded language, which uses words with strong emotional implications, name calling or labelling, which describe an object or something that the audience fears or sees as undesirable, doubt, which questions the credibility of something, and so on @xcite . refer to @xcite for more details. the following subsections give details on the proposed si and tc models.