syrapropa at S em E val-2020 Task 11: BERT -based Models Design for Propagandistic Technique and Span Detection (2020)

Abstract: This paper describes the BERT-based models proposed for two subtasks in SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles. We first build the model for Span Identification (SI) based on SpanBERT, and facilitate the detection by a deeper model and a sentence-level representation. We then develop a hybrid model for the Technique Classification (TC). The hybrid model is composed of three submodels including two BERT models with different training methods, and a feature-based Logistic Regression model. We endeavor to deal with imbalanced dataset by adjusting cost function. We are in the seventh place in SI subtask (0.4711 of F1-measure), and in the third place in TC subtask (0.6783 of F1-measure) on the development set.

## introduction propaganda exists in social media content and can subvert and distort public deliberation @xcite . natural language processing (nlp) researchers develop computational techniques that automatically detect propaganda in the content. in this paper, we develop two systems for two subtasks in the semeval-2020 task 11: (1) span identification (si) (2) technique classification (tc) in news articles respectively. the si subtask focus on identifying fragments in a given plain-text document which contain at least one propaganda technique; while tc subtask aims to classify the applied propaganda technique given a propagandistic text fragment @xcite . for si, we interpret the task as to detect a span in a context, which is the smallest detect unit. we base our system on spanbert @xcite , and combines three jointly trained classifiers: sentence, start, and end classifiers. specifically, the start and end classifiers detect the start and end position of the span, while the sentence classifiers provide sentence-level information for the start and end classifiers. for tc, we come up with a hybrid system combining the two bert models with different training methods, a logistic regression model, and extra rules to classify a propagandistic fragment into one of the 14 techniques. for si, we find that the segment approach of a context affects the result. while for tc, emotion features extracted from nrc lexicon are not effective to distinguish 14 classes. however, features such as text length, tf-idf, occur times in a document, superlative form, question words, hashtags and supplement are useful in distinguishing different propaganda techniques. overall, we are in the seventh place in si subtask (0.4711 of f1-measure), and in the third place in tc subtask (0.6783 of f1-measure) on the development set. ## related work it is believed that news media plays an active and major role in producing and distributing propaganda and there can be a tool box that helps detect propaganda in news articles @xcite . @xcite annotated 18 propaganda techniques in news articles. the researchers later merged the similar underrepresented techniques, resulting in 14 techniques @xcite . the model trained on the sentence level task (whether the input sentence is propagandistic) is found to be effective for the fragment level task (detect the propagandistic fragment). @xcite designed a multi-granularity and multi-tasking neural architectures to jointly perform both the sentence and fragment level propaganda detection. da san martino et al. (2019b) designed a multi-granularity neural network that includes document-level, paragraph-level, sentence-level, word-level, subword-level and character-level task to detect fragments in news articles. different from their studies, our si model utilizes a sentence-level classifier which predicts whether a context contains a span or not, and embeds the hidden representations from it into the other two classifiers. we also explore different approach of concatenation of other higher representations. previous researches explore the effectiveness of hybrid model. for instance, al-omari et al. ( 2019 ) use several submodels including bilstm, xgboost, bert model to predict if a sentence is propagandistic. our tc model is also an hybrid model of three submodels (bert, cost bert, lr † ‡) that combine the partial results based on their learning capacity of different categories. we compare the unweighted and weighted cost function in our approach and find out that the latter outperforms on the minority classes such as whataboutism/straw men/red herring, thought-terminating cliches and bandwagon/reductio ad hitlerum. ## tc system description our system includes three individual sub-models combined together with extra rules, which outperforms any of the sub-models itself. ## experimental setup we use the training, development and test datasets provided by @xcite , which contains news articles from around 50 news outlets. for si, the evaluation function gives credit to partial matching between two spans @xcite . we base our model on the pretrained cased spanbert-base model and fine-tuned them on development dataset with the following configuration: sequence length of 128, learning rate of 1e -5, batch size of 4. we choose 64 for the hidden size of l k+1 s and l k+1 e . in addition, α sent , α start , α end equal to 0.25, 0.5 and 0.5. for tc, we evaluate our model on by micro-averaged f1-measure. we base our bert model on the pretrained uncased bert-base model and fine-tuned with the following configuration: sequence length of 128, learning rate of 1e -5, batch size of 4. we utilize the solver of lbfgs, penalty of l2, c of 1.0 and "balanced" mode in logistic regression. ## conclusion this paper develops a spanbert-based model for span identification (si) and a hybrid model for propaganda techniques classification (tc). as for si, our paper explores different segmentation of contexts from news articles. based on span-bert, we facilitate the detection by a deeper model and a sentence-level representation. the start and end boundary are conceptually independent to each other, therefore, obtaining best indexes from both the start and end classifiers achieve the best performance comparing to that of any one of them. our model is not restricted on semantics integrity, but remaining a high ratio of span-annotated data is essential especially for a small size of training data. our experiment of tc offers several insights. first, we find that emotion features extracted from nrc lexicon is not effective to distinguish 14 classes. second, we find that the cost-weighted learning approach is effective in addressing the imbalance issue of the training dataset. third, features such as text length, tf-idf, occur times in a document, superlative form, question words, hashtags, and supplement are useful in distinguishing different propaganda techniques. in the future, we will explore more on how to segment a context in the training dataset and how different context affect the results. in addition, our model lacks the ability to detect the multiple spans in one context. we will conduct a fine-grained analysis to examine whether a context contains a span and if so, how many spans are included and the exact start and end boundary of them. we also have two suggestions for future work of tc. first, the use of part of speech (pos) tags in correcting mislabeled data shows a good improvement of the performance. it would be interesting to further explore its use and representations in the model. second, while the hybrid model achieves improvement from sub-models, it'd be interesting to investigate a single model that differentiates 14 classes at the same time in the future.