Team D i S aster at S em E val-2020 Task 11: Combining BERT and Hand-crafted Features for Identifying Propaganda Techniques in News (2020)

Abstract: The identification of communication techniques in news articles such as propaganda is important, as such techniques can influence the opinions of large numbers of people. Most work so far focused on the identification at the news article level. Recently, a new dataset and shared task has been proposed for the identification of propaganda techniques at the finer-grained span level. This paper describes our system submission to the subtask of technique classification (TC) for the SemEval 2020 shared task on detection of propaganda techniques in news articles. We propose a method of combining neural BERT representations with hand-crafted features via stacked generalization. Our model has the added advantage that it combines the power of contextual representations from BERT with simple span-based and article-based global features. We present an ablation study which shows that even though BERT representations are very powerful also for this task, BERT still benefits from being combined with carefully designed task-specific features.

## introduction the purpose of propaganda is to use communication to foster predetermined agendas, or to achieve a response that furthers a desired outcome @xcite . prior research has focused on creating machine learning models that label whole news articles or even entire news outlets as propagandistic @xcite . to increase the granularity of these coarse models, a new data set was developed in a study by @xcite , which enabled models to jointly identify fragments of propaganda within a document, while also classifying their respective propaganda techniques @xcite . this paper presents our solution (disaster, finishing at 11th place) to the technique classification (tc) sub-task at semeval 2020 task 11 "detection of propaganda techniques in news articles". 1 tc is a multi class classification problem in which a system needs to identify the propaganda techniques of a given span of an article. for instance, when given the span "stupid and petty" the system should classify it as loaded language. our system is an ensemble model based on stacked generalization @xcite which enables the incorporation of both traditional engineered features @xcite and the transformer @xcite based language model bert @xcite . ## related work in addition to formulating the original problem of fine-grained propaganda identification and creating the corpus needed to solve the task, @xcite also designed a multi-granularity neural network. this model outperformed several strong bert baseline models in the high granularity fragment-level classification by using information from low granularity classification (e.g. document-level) to drive higher-granularity classification (e.g. paragraph-level). as the tc sub-task of this competition does not require span detection, a multi-granularity approach is not necessary. instead, our model is inspired by a project by @xcite , in which they outperformed bert baseline models by combining a bert model with linguistic features. ## data the provided training data for this competition contains 371 articles in which all fragments of propaganda are annotated with one of the 18 different propaganda techniques described @xcite . however, due to a low frequency of some of the techniques, similar underrepresented techniques were merged into a superclass, while one of the techniques was eliminated completely. thus, the tc task was a 14-class classification problem, where two of the classes were superclasses representing several techniques each @xcite . ## model overview our simplest baseline model worked by always predicting highest prior probability. the label with the highest prior probability in the training set was loaded language. we compare the results of this baseline model with our final model in section 5, we tackle the problem of propaganda technique identification as a classification task where we combine figure 2 : normalized matrix of labels co-occurring in the same article. the diagonal line indicates that articles tend to contain spans with the same label. the ids correspond to the classes listed in ## experiments to test the importance of the different components, a feature ablation study was performed. a 10-fold stratified cross-validation was then performed on the training set and the micro-average f1 score was recorded. the models used in this experiment were implemented in python using a pytorch framework @xcite . the features were created using a mixture of spacy and nltk @xcite , whereas for bert we use the huggingface library @xcite . all results obtained from the ablation study are summarized in ## discussion as evident from the ablation study (