YNU taoxin at S em E val-2020 Task 11: Identification Fragments of Propaganda Technique by Neural Sequence Labeling Models with Different Tagging Schemes and Pre-trained Language Model (2020)

Abstract: We only participated in the first subtask, and a neural sequence model was used to perform the sequence tagging task. We investigated the effects of different markup strategies on model performance. Bert that performed very well in NLP was used as a feature extractor.

## introduction propaganda is a form of information promoting or publicizing a particular cause or point of view, especially one of a biased or misleading nature. propaganda usually uses psychological and rhetorical techniques to meet its purpose. in today's information age, propaganda is widely used in politics, business, science and technology, and other fields. and there are increasing propaganda techniques that are becoming more diverse and difficult to identify @xcite . we can search for the use of specific propaganda techniques via analyzing the language of an article, which is useful to find out the aims of propagandist news articles. in the mass of propagandist news articles, how to quickly and accurately identify the use of propaganda techniques is a challenging task. semeval-2020 task 11 -detection of propaganda techniques in news articles aims to develop automatic tools to detect propaganda techniques. and there are two subtasks: the shared task provides a news article corpus, as well as 18 predefined propaganda techniques that have been annotated correspondingly. more detailed information can be seen in @xcite . the rest of the paper is divided into the following sections: section 2 describes the previous works and the relationship between coarse-grained and fine-grained tasks. section 3 reports the final approach and those we tried . section 4 details the results for the experimented models followed by error analysis of the best model. ## related work some previous works have proposed various methods for propaganda detection. lstm is used for propaganda detection @xcite . @xcite used the maximum entropy classifier with different features to deal with the same work of @xcite . however, their works are all at the document level, which labels the whole article as propaganda. more fine-grained propaganda analysis has been proposed by @xcite , which avoids noisy gold labels affect the quality of model training and also makes it possible to illustrate to the user why an article was judged as propagandistic. this is also the case in other areas, where more fine-grained research that can increase performance and interpretability on coarse-grained tasks has quite a few advantages for further follow-up studies. @xcite presented a community-based weighted graph model to predict valence-arousal ratings of affective words. future research can benefit from such useful lexical resources to extend current valence-arousal prediction work from the word-level to sentence-levels and document-levels. we treat this si subtask as a sequence tagging task, and previous jobs have been handled in this way. da san martino et al. (2019) proposed a multi-granularity neural network model based on bert to deal with both sentence-level classification and fragment-level classification. tayyar madabushi et al. (2019) presented a method of cost-sensitivity bert to address sentence-level classification in imbalanced data and provide a measure of corpus similarity to determine the difference between the training and the development or test sets. ## methodology we only participate in the si subtask, and our approach is a neural sequence labeling model along with different tag schemes. our model used some large pre-trained language model as the encoder, and other network structures as the output of the encoder. ## experiment and result our experimental implementation is based on the pytorch framework and uses the transformers package @xcite to get the pre-trained language model. in the data preprocessing stage, the 0 20 40 60 80 100 /hqjwkriiudjphqw 0 200 400 600 800 1xpehuriiudjphqw figure 2: the distribution of the span of propaganda technique length of the training set in the data preprocessing stage, we clean the text and remove the special symbols. the number of words in all sentences is analyzed, as shown in figure 1 , and the maximum length is 150. from the figure, we can find that sentences less than 80 tokens in length are 99.15%, and less than 100 tokens are 99.72% of the total. in the experiment, we set the sentence length as 80,100 and 150 respectively, and find that the sentence length as 100 is the best. the si is a token-level task, which is supposed to preserve all tokens in the sentence. but the too big sentences length setting will result in a lot of irrelevant padding in short sentences, which will create a lot of data noise and affect the performance of the model. the bert model is used to evaluate the sentence length parameters. the experimental results are displayed in a few discontinuous i-propaganda labels appear in the predictions when the iob tagging scheme is used, which may be caused by the large length of the span of propaganda technique. we calculate the length of the fragment of the propaganda technique, as shown in figure 2 , and the mean length is 8.8203 tokens. the fragments are longer than the usually named entity. 2 , compared with the iob tagging scheme, the iobes does get a performance improvement. however, due to the large length of the propaganda fragment, the labeling method of ner task is difficult to accurately identify its boundaries because it was found in the experiment that the tag set did not form an accurate boundary of the propaganda fragment. we believe that the propaganda fragment has longer semantic dependence, while its syntactic dependence is not clear. therefore, it may be better to use the tagging scheme without emphasizing the boundary of propaganda fragment. and we use the simple binary tagging scheme. the experiments show that the simpler binary tagging scheme achieves the best results. after using bert as the feature encoder, we attempt to use roberta and xlnet as the pre-trained language model, and the decoder also use the softmax layer. we fine-tune the learning-rate, min-batch-size and num-epochs hyperparameters, and obtain the results with three different models on development data set, as shown in ## conclusion and future work we focus on the span identification task and develop an automated system with three tagging schemes and pre-trained language models in this paper. we analyze the sentence length and fragment length of the data set in detail and use experimental methods to verify the impact of the tagging scheme on fragment recognition in sentences, which can serve as a reference for future research. we select different pre-trained language models as feature encoders and compare their results. however, we do not make too much effort into the design of the decoder, which can be a bottleneck for the whole system. we believe that using a complex decoder may further improve system performance, which is one of the future research directions. at present, the sequence annotation task has three main types of decoders, namely softmax, crf, and seq2seq. softmax can decode tags in parallel, but cannot model dependencies between tags, which results in poor performance. crf decodes the whole sentence tag by learning the transfer matrix between tags, which can model the dependency of adjacent tags. however, using viterbi decoding, crf can not be parallel and the decoding speed is slow. seq2seq can model the long-distance dependence of tags, but it has a problem that the false prediction of the previous tag will lead to the error of the subsequent tags. for the task of propaganda techniques detection, because the detection fragment is long, the model needs to be able to model the long-distance dependence. but now there is not much research on decoding, which will be a worthy research goal.