NLFIIT at S em E val-2020 Task 11: Neural Network Architectures for Detection of Propaganda Techniques in News Articles (2020)

Abstract: Since propaganda became more common technique in news, it is very important to look for possibilities of its automatic detection. In this paper, we present neural model architecture submitted to the SemEval-2020 Task 11 competition: “Detection of Propaganda Techniques in News Articles”. We participated in both subtasks, propaganda span identification and propaganda technique classification. Our model utilizes recurrent Bi-LSTM layers with pre-trained word representations and also takes advantage of self-attention mechanism. Our model managed to achieve score 0.405 F1 for subtask 1 and 0.553 F1 for subtask 2 on test set resulting in 17th and 16th place in subtask 1 and subtask 2, respectively.

## introduction nowadays, many publishers provide articles online to reach their audience faster. this shift from traditional journalism via printed press enabled more extensive misuse of objective news for own agenda using different propaganda techniques. propagandistic articles try to convey their message using different techniques. propaganda is the spread of purposefully shaped information in media to promote some sort of agenda @xcite . this kind of information is produced by increasing number of biased news outlets in attempt to sway or manipulate public opinion. however, there is also a connected problem to propaganda, online spreading of fake news and misinformation in general @xcite . however, nowadays the widespread use of internet and access to multiple sources of information has caused traditional techniques of propaganda, like spreading of disinformation, to become far less effective. along with this trend, propaganda techniques are naturally getting more complex to maximize their impact. use of logical fallacies and appealing to the emotions of the audience is getting much more prominent because these techniques are significantly harder to spot at first glance and require much deeper analysis to get revealed. research on detecting propaganda has focused primarily on news articles @xcite . however, analysis of articles produced by propagandistic news outlets indicates that these sources can often publish objective articles in attempt to increase their credibility @xcite . recently the attention has moved towards fragment and sentence level propaganda detection. a new problem has been introduced -to detect the use of specific propaganda techniques in articles along with large corpora to assess this problem @xcite . to tackle this problem many different neural model architectures were proposed, such as fine-tuned bert pre-trained model @xcite , ensemble of bert, bi-lstm, and xgboost @xcite , or ensemble of logistic regression, cnn and bert @xcite . the main goal of this semeval task was to develop tools for automatic detection of propaganda and consisted of two subtasks @xcite ): in this paper, we present our proposed neural network architecture consisting of different types of neural layers. we experimented with different word representations, such as elmo, glove, and different encoder layers, such as gru, lstm, and also with transformer model bert. we report description and results for multiple model architectures for both subtasks in the following sections. ## model we experimented with multiple model setups for each subtask. we tried out multiple pre-trained embeddings, which fed bi-lstm layers with word representations @xcite . for subtask 2, we also experimented with a fairly big pre-trained transformer model. the general architecture portraiting a structure of our proposed model is shown in figure 1 . preprocessing preprocessing can be considered as an important step in each natural language processing task. the dataset for this task consist of news articles thus the input text is not as much noisy as the texts from social media but still contains significant number of tweet footers, timestamps, web surveys, hyperlinks, and advertisements. in attempt to reduce number of input tokens that are very likely not propaganda, we decided to partly or completely remove these samples. in addition, we also removed all non-latin characters, such as hebrew, emoji characters, and also twitter mentions. we also substituted unicode quotation marks, apostrophes and hyphens with their ascii equivalents @xcite . word representation to be able to feed input samples into models, we need to use appropriate word representations. in our experiments, we used different word representations to represent those samples. first, we used deep contextualized word representation also known as elmo @xcite along with its available pre-trained model. we also experimented with an unsupervised learning algorithm for obtaining vector representations for words -glove @xcite . in subtask 2, we also experimented with transformer model known as bert @xcite and its pre-trained model @xcite . encoder layer word representation produced by pre-trained embeddings are fed to encoder layer. after initial experiments with both, unidirectional and bidirectional gru @xcite and lstm layers @xcite , we decided to continue with only bidirectional lstm @xcite in our experiments. we also tried to tweak the size of a hidden state (sizes of 128, 256, 512 and 1024 were used), but it resulted in negligible performance change. for both subtasks, we also experimented with different number of stacked encoder layers (2 and 3 stacked layers were considered), but the most stable results was produced by single encoder layer. to enhance performance of lstm layers for long input samples we also experimented with the self-attention mechanism on top of the encoder layer. decoder layer we used standard linear layer to decode output representation of both setups of recurrent layers without and with self-attention mechanism for both tasks. we also experimented with conditional random fields (crf) for subtask 1. the results especially in combination with glove were highly unstable regarding to the run and we decided not to continue in further experiments with crf. loss function as a loss function, we used standard cross-entropy loss. in addition, we also experimented with weight setup for classes in both subtasks, but unfortunately this yielded slightly worse results especially in subtask 1, due to massive class imbalance of propaganda tokens versus non-propaganda tokens. ## evaluation in this section, we briefly provide information about the dataset, experimental settings used for our models and performance results of the proposed models. ## conclusions we proposed a neural model architecture for automatic detection of propaganda techniques in news articles. for both subtasks of the semeval-2020 task 11, we used elmo embeddings, which fed word representations to bi-lstm encoder. for subtask 2, we also utilized the self-attention mechanism on top of the bi-lstm layer. we experimented with multiple stacked bi-lstm layers, however single layer proved to perform the best for both subtasks. our experiments with embeddings showed significantly better performance of elmo over glove for both subtasks. for subtask 1, we tried to employ crf on top of bi-lstm with its perfomance coming closest to our proposed model. for subtask 2, we experimented with bert pre-trained model that showed potential to outperform our proposed model, but due to high memory requirements it could not be fully utilized. we tried to improve performance of our proposed models by using model ensemble, but unfortunately our strategy seems not to have a significant impact on performance. code for this submission is available via github repository foot_1 . to obtain better results for subtask 1 we would probably try to fine-tune bi-lstm model with crf and further experiment with ensemble of this model and bert. as we mentioned, the biggest pre-trained bert model utilized to its full potential could outperform our model for subtask 2. another interesting possibility would be ensemble of bert and bi-lstm with the attention.