LTI at CMU at S em E val-2020 Task 11: Incorporating Multi-Level Features for Multi-Granular Propaganda Span Identification (2020)

Abstract: In this paper we describe our submission for the task of Propaganda Span Identification in news articles. We introduce a BERT-BiLSTM based span-level propaganda classification model that identifies which token spans within the sentence are indicative of propaganda. The ”multi-granular” model incorporates linguistic knowledge at various levels of text granularity, including word, sentence and document level syntactic, semantic and pragmatic affect features, which significantly improve model performance, compared to its language-agnostic variant. To facilitate better representation learning, we also collect a corpus of 10k news articles, and use it for fine-tuning the model. The final model is a majority-voting ensemble which learns different propaganda class boundaries by leveraging different subsets of incorporated knowledge.

## introduction propaganda @xcite is "the deliberate and systematic attempt to shape perceptions, manipulate cognition, and direct behavior to achieve a response that furthers the desired intent of the propagandist" (jowett and @xcite . the rise of digital and social media has enabled propaganda campaigns to reach vast audiences @xcite , influencing or undermining the democratic processes, amplifying the effects of misinformation, and creating echo-chambers. as a tactic of manipulation, propaganda is considered most effective when it is veiled: the readers are not able to identify it, but their opinions are shaped according to the propagandist's hidden agenda @xcite ; it is thus very difficult to identify propaganda automatically. however, automatic identification and analysis of propaganda in news articles and on social media are essential to understand propaganda at scale and develop approaches to countering it @xcite @xcite . prior research on propaganda detection focused primarily on identifying propaganda at a document level, due to the dearth of finer-grained labelled data @xcite . this has resulted in classifying all news articles from a particular source as propaganda (or hoax, disinformation, etc.), which is often not the case, and which can obfuscate a finer-grained analysis of propagandistic strategies @xcite . recently @xcite carried out a seminal task of fine-grained propaganda detection and curated a dataset consisting of about 550 news articles. the dataset contains word-span level annotations provided by high-quality professionals along with additional information about the propaganda technique employed in the span. the span identification (si) sub-task of semeval 2020, task 11, @xcite employs this aforementioned corpus and requires participants to detect propaganda spans in news articles. in this paper, we describe our solution to the si task. we propose a bert-bilstm based multigranularity model, inspired from @xcite . the model's parameters are jointly optimized on the tasks of sentence-level and word-level propaganda detection. we fine-tune bert-bilstm end-to-end on the joint-loss to obtain competitive scores on the si task. furthermore, we explore the benefits of incorporating word, sentence and document-level syntactic and affective features extracted from dictionaries like liwc (tausczik and pennebaker, 2010) and empath @xcite . we find that incorporating these features leads to significant improvements in performance (discussed in section 3). we also leverage articles from multiple well-known propaganda news sources to fine-tune bert in an unsupervised manner. furthermore, in order to tackle the problem of unbalanced data, we use a weighted cross-entropy loss. we present our model and additional features in section 2, and show our results and ablation analysis in section 3. ## model in this section, we describe the modeling aspects we have adopted for the task. the overview of our model is given in figure 1 . ## experimental setup we used the bert model from hugging-face library which was then fine-tuned on the corpus. since the test-set was not made available during the competition, we hold out a small part of the training data as dev-set which was used to tune the models. the submitted model was chosen based on its span-level normalized f1-score on the validation set provided by the competition organizers. 5 the hyperparameter choices are provided in table a1 in the appendix. we run all our experiments on nvidia geforce gtx 1080 ti gpus with 12gb memory. ## results in this section, we present our method's results on the official validation set. we also analyze the importance of word, sentence and document-level features (as detailed in section 2.2) to our model. for brevity, we only present the results on multi-granularity bert-large-uncased bisltm (mgu-lstm) and bert-large-cased bilstm (mgc-lstm) models due to their superior performance. initial few rows in our experiments with sentence (s) and document-level (d) features ( finally, we present the contributions of the weighted cross-entropy loss (w) and the unsupervised finetuning (f) performed on articles from propaganda news sources in ensemble we created a majority-voting ensemble of a subset of the model variants discussed above. we observe that we obtain the best results when the ensembled models are the most dissimilar. this pattern has also been shown to be beneficial in stacking where the same machine-learning problem is tackled with different types of learners. we find that an ensemble of 7 models namely bert-lc-axn-s-d-w (seed 3), mgc-lstm-ax-s-w (seed 3), mgu-lstm-ax-s-w (seed 1), mgc-lstm-axn-s-w (seed 3), mgc-lstm-axn-s-w-f (seed 1), and mgc-lstm-axn-s-w-f (seed 3) performs substantially better than individual models improving the performance on validation-set to 48.86. we illustrate an example of detecting propaganda spans by our model in appendix. the spans are highlighted in bold. we observe that our model is not only able to detect short spans like loaded language or named calling but also large spans corresponding to repetition and slogans that rely on contextual information. ## conclusion this paper describes our 4 th place submission to the span identification (si) subtask of semeval 2020 task 11. our approach is based on a multi-granularity bilstm with bert embeddings. we explore the contributions of several word, sentence and document-level features concatenated with token and pooled embeddings output from bert. our work also highlights the importance of tackling class-imbalance in the corpus and learning domain-informed representations through unsupervised fine-tuning of bert on latest news articles. we submit a majority-voting ensemble of multiple models with potentially dissimilar decision boundaries to make robust predictions on the official validation and test-set. table a1: hyper-parameters b example of propaganda detection tudor simionov was killed when he tried to prevent a gang from gatecrashing a party in mayfair. the page one headline in the evening standard said it all: 'first stab victims of the new year.' welcome to 2019. ... he died at the scene, in park lane, after being attacked by up to ten men â ȃť including, allegedly, the 26-year-old son of finsbury park firebrand captain hook, the islamist hate preacher currently serving life ... but a visible police presence may have deterred the gang who killed tudor simionov. ... no doubt londonâ ȃ źs two-bob chancer of a mayor sadiq khan would blame the absence of police on the streets on â ȃÿausterityâ ȃ ź or the â ȃÿtory cutsâ ȃ ź. maybe if he hadnâ ȃ źt splurged âč2.3 million of taxpayersâ ȃ ź money on a vainglorious new year fireworks display, he could have diverted more money to the met to put extra bobbies on the beat. some of this largesse was frittered away turning the london eye into a blue and yellow eu flag, as part of a cynical, self-serving propaganda stunt aimed at bolstering the stop brexit campaign and burnishing the image of the mayor himself.... whatâ ȃ źs most likely to deter immigrants from coming here is not khan slandering leave voters as knuckle-scraping, racist bigots, but the news that someone can be stabbed to death on one of londonâ ȃ źs busiest, best-known and most upmarket streets .... and if that chilling reality doesnâ ȃ źt put people off moving here from overseas, then perhaps they may be dissuaded by the paralysis on londonâ ȃ źs roads ... rough sleepers and aggressive beggars are a permanent fixture in the west end. yet while innocent blood runs in the gutters, and police budgets are restrained,