JUST at S em E val-2020 Task 11: Detecting Propaganda Techniques Using BERT Pre-trained Model (2020)

Abstract: This paper presents the submission to semeval-2020 task 11, Detection of Propaganda Techniques in News Articles. Knowing that there are two subtasks in this competition, we have participated in the Technique Classification subtask (TC), which aims to identify the propaganda techniques used in a specific propaganda span. We have used and implemented various models to detect propaganda. Our proposed model is based on BERT uncased pre-trained language model as it has achieved state-of-the-art performance on multiple NLP benchmarks. The performance results of our proposed model have scored 0.55307 F1-Score, which outperforms the baseline model provided by the organizers with 0.2519 F1-Score, and our model is 0.07 away from the best performing team. Compared to other participating systems, our submission is ranked 15th out of 31 participants.

## introduction the high rate of using social media and the spread of digital news and online blogs have enabled the massive amount of data to reach a wide range of audience as well as allowed non-journalist to disseminate false news, misinformation, hoax and propaganda to mislead people and deceive them @xcite @xcite . moreover, the main way of accepting this news by society is the mass media and digital news @xcite . the reasons behind disseminating false information and news could be for financial and political purposes or to mislead the readers and influence their opinion negatively as well it has been argued to influence elections and threaten democracies @xcite . propaganda is defined as"efforts by special interests to win over the public covertly by infiltrating messages into various channels of public expression ordinarily viewed as politically neutral @xcite ". it aims to mislead audiences by influencing them toward a particular political or social agenda in news media @xcite . therefore, several propaganda techniques and tools are designed to propagate certain ideologies. these techniques usually appeal to audience emotions and reach of their desires @xcite . recently, a group of researchers @xcite proposed a fine-grained propaganda corpus and organized a shared task on fine-grained propaganda detection, which includes both sentencelevel and fragment-level sub-tasks. semeval task 11 is an extension to this task, which aims to predict propaganda techniques in news articles with two sub tasks: in this paper, we present our participation of the just team at semeval2020 task 11, and more precisely, we have participated in technique classification (tc) subtask. we have used a set of deep learning models with different word embeddings to detect propaganda techniques. however, the final submission was based on bert's uncased pre-trained language model that achieved a significant performance. the rest of the paper is organized as follows.section 2 describes the related work.section 3 presents the dataset and per-processing.section 4 presents the model and architecture. section 5 presents the experiments and section 6 discusses the results.finally, section 7 draws conclusions and sketches of future work. ## related work it is not surprising that people have been knowing and using propaganda for centuries @xcite . for example, during world war one in 1914, propaganda was used on a global scale with the rise of the nazi propaganda machine @xcite to mobilize hatred against the enemy. detection of propaganda has gained massive interest in the research community in recent years. therefore, the automatic detection of propaganda is studied as part of the propaganda analysis project in @xcite . the researchers provided the first propaganda detection system, which is available publicly and is called poppy.this system is a real-world and real-time monitoring system to unmask propagandistic articles in online news. in @xcite , the researchers discussed the problem of identifying propaganda in online news content. therefore, they performed several neural network architectures, such as long short-term memory (lstm), hierarchical bidirectional lstm (h-lstm) and convolutional neural network (cnn) in order to classify the text into propaganda and non-propaganda. effective techniques were used for text pre-processing as well and different word representation models including word2vev, global vectors(glove), and tf-idf. moreover, the models performed to data provided from twitter to the internet research agency. the data was about the relevant activities of the ira from september 1 to november 15, 2016. however, the results showed that cnn with word2vec representation outperformed other models with accuracy equals to 88.2 %. the researchers in @xcite released the shared task on fine-grained propaganda detection as part of the nlp4if workshop at emnlpijcnlp 2019 that focused on detecting propaganda and specific propagandistic technique in news articles at sentence and fragment level, slc and flc tasks respectively. the wining system in slc task @xcite was based on using an attention transformer using the bert language model where the final layer of the model replaced with a linear softmax layer. however, to obtain multi-head results they have used ensemble attention neural networks with 12 attention heads and 12 transformer blocks.as well, the teams @xcite fine-tuned bert to tackle slc task.another team @xcite presented their proposed model to detect propaganda in the slc task. furthermore, they have experimented with various combinations of deep learning models including xgboost @xcite , bilstm and bert cased and uncased with a set of features including effective words, lexical features as well as word embeddings based on glove @xcite . moreover, their final model was based on an ensemble of xgboost, bilstm, and bert-case and uncased where they achieved a 0.6112 f1 score. for the flc task; a team @xcite achieved the best results by applying 20-way token level classification, each token in the input article classified into 20 token types. moreover, they fined tune bert uncased base model to classify the tokens by adding a linear head to the last layer of bert. ## dataset and preprocessing the corpus which is provided by @xcite includes 350 articles for training and 75 articles for development. in training articles, there are 6129 propaganda spans and in the development set, there are 1063 propaganda spans. figure 1 shows the distribution of classes in the training set, and as can be seen, the classes are imbalanced; loaded language is the most frequent class with 2123 samples whereas bandwagon, reductio_ad_hitlerum is the least frequent class with 72 samples. text preprocessing has been performed for each span of training and development set that includes: removing punctuation, tokenization, cleaning text from special symbols, and cleaning contractions. all preprocessing steps were performed using the nltk library. ## system overview we have used bidirectional encoder representations from transformers (bert) @xcite , which is one of the most powerful pre-trained language models that achieved state-of-the-art results in a wide variety of nlp tasks. there are two types of pre-trained models, bert-base and bert large, however, we have used the base model as it needs less computational resources. in each type, there are five pre-trained models, moreover, we have used uncased and cased. we have performed the pre-trained bert models for multi-class classification, and retrieve bert-uncased, which contains 12 transformer layers and 110 million parameters as well bert-cased with 12 transformer layer, 768-hidden, 12-heads, 110m parameters. bert models have been downloaded from tensorflow hub 1 . the architecture of the model is described in figure 2 . the cleaned span is represented using a special token [cls] that is added in front of every span and another token [sep] is added at the end. the input feeds the transformers' layers to generate the prediction for each class. accordingly, the models are trained for 15 epochs using a learning rate of 2e-5, a maximum sequence length of 128, and a batch size of 128. transformer layer 2 transformer layer 2 raw data data pre-processing cleaned data cls are touting the president is order predictions [pad] additionally, we have experimented with several deep learning models based on learning representations (embeddings) that have been performed on the dataset to detect propaganda techniques. the input to the models is represented using a pre-trained sentence and word level embeddings to encode the input into the embedding vector. we used the word2vec embedding that is trained on google news @xcite , where the sentence is encoded to embedding layers, which is a lookup table that consists of 300-dimensional pre-trained vector to represent each word. it is worth to mention that we have also experimented glove @xcite and fasttext @xcite embeddings, but the results were not promising. for sentence embeddings, we have performed universal sentence encoder (use) @xcite that generates 512-dimensional representation for each sentence. the following deep learning models have been performed to the dataset: the models are implemented using keras framework foot_0 .use loaded from tensorflow hub foot_1 .for training, we have used the adam optimizer foot_2 with @xmath0.001, and categorical cross-entropy as the loss function. the batch size is set to 32 and the number of epochs to 25. ## results and discussion we have directly evaluated all the models on the development set, and the best model is chosen to generate predictions of the test data. in the test stage, since we are only allowed to submit a single run on the test set, we choose the model with the highest f1 score on the development set (0.5766) to generate predictions, which is bertuncased. the evaluated results on the test set are listed in table2, the model yielded a test f1 score of 0.55307. table3 shows the class-wise scores. accordingly, the model performs well on propaganda techniques that appear frequently in the article such as "load language"and "namecalling, labeling "that achieved 0.71958 and 0.64727 f1 score respectively on the test set, while bert acts poorly on some propaganda types, such as "exaggeration, minimisation", "bandwagon ", "reductio_ad_hitlerum"and "repetition". however, some propaganda techniques are challenging due to the way that the span is shaped or the number of words in the span, for example, "flag-waving"can be shaped in different ways, and "repetition"depends on the occurrence of a word (or more) and the repetition of it in the article. therefore, it is not enough to only look at the span to make the prediction, more information needs to be given to the model such as article context and word counts for each word in the span across the article thus including the whole article as the context is needed. due to the limited time, we didn't experiment with the effect of adding these features. finally, we have noticed that bert has a strong performance in detecting some complex propaganda techniques while may perform poorly on other techniques and that because of the problem of imbalanced classes, which are impacting the performance of the minority class and leading the model to favor predicting the majority class. ## conclusion in this paper, we described our solution in the propaganda technique classification subtask at semeval-2020 task 11. we have investigated several models such as cnn, bilstm, and nn with word and sentence embeddings including word2vec and use to detect propaganda techniques. however, our final solution was based on the bert-uncased language model, which showed significant performance. the evaluations were performed using the dataset that was provided by semeval2020 task11 organizers. our proposed model is ranked in 15 th place among 31 teams. moreover, the f1-score that is achieved is 0.55307, which outperformed the baseline model (0.25196). the results confirmed that pre-trained language models(like bert) are clearly a step forward for nlp and it has a strong performance in detecting propaganda techniques.