UTMN at S em E val-2020 Task 11: A Kitchen Solution to Automatic Propaganda Detection (2020)

Abstract: The article describes a fast solution to propaganda detection at SemEval-2020 Task 11, based on feature adjustment. We use per-token vectorization of features and a simple Logistic Regression classifier to quickly test different hypotheses about our data. We come up with what seems to us the best solution, however, we are unable to align it with the result of the metric suggested by the organizers of the task. We test how our system handles class and feature imbalance by varying the number of samples of two classes (Propaganda and None) in the training set, the size of a context window in which a token is vectorized and combination of vectorization means. The result of our system at SemEval2020 Task 11 is F-score=0.37.

## introduction propaganda is a complex phenomenon that was studied in psychology @xcite , sociology @xcite , theory of communication @xcite , pedagogy @xcite , history @xcite , linguistics @xcite , and other sciences. naturally, the task of automatic propaganda detection has been set and approached in different ways. proppy, an on-line service, detects propaganda in news articles and clusters them according to an index of propaganda @xcite . it daily analyzes emerging texts, identifies events described in them, discards near-duplicates, and, lastly, computes propaganda index on the basis of n-gram features, vocabulary, its richness, style, readability, and news landscape (nela) @xcite . in the propaganda analysis project, the propaganda detection is more focused on locating propaganda within a text. at the hack news datathon 1 task 3 was to detect a fragment containing propaganda. however, it was paired with classifying it according to 18 techniques 2 , same as the fragment level classification at the shared task of "fine-grained propaganda detection" (emnlp2019) @xcite . the competition @xcite evaluates these two stages separately. although it is hard to map results of previous competitions on the current one, the former revealed some life hacks that can be of use in the latter. for example, as noted by h.t. madabushi (2019), extracting fragments of propaganda is similar to the task of named entity recognition, in that they are both span extraction tasks. their system propergander was partially based on the bert @xcite solution for ner. however, probably, the main lesson learned from the first two events was about the necessity of using bert (or at least a deep-learning neural network) to achieve a state-of-the-art result. in the current article, we proudly present a no-bert and even no-deep-learning solution for the task of span identification. our choice not to utilize contextualized vectors such as bert is grounded by the following reasons. (1) in the annotation to "poor man's bert: smaller and faster transformer models" @xcite , it is noted that bert-based models require certain laboratory facilities, and this work is licensed under a creative commons attribution 4.0 international licence. licence details: http:// creativecommons.org/licenses/by/4.0/ . the paper is organized as follows: we first describe our idea of class and feature imbalance in propaganda; we, then, outline the basics of our classifier; we devote a paragraph to testing hypotheses about our data and conclude about the usability of our system. ## propaganda in news articles the verbal propaganda in a news article is a stretch of text written with an intent to influence the reader's opinion foot_0 with arguments that lead to conclusions advantageous to the author. the communicative intent is to control and manipulate the reader. the tools are cognitive, logical, emotional and other means to confuse the reader (verbal fouling), and the verbal representation adds up information to what has to be said as a fact in a news article as a genre: the known facts and the believed-to-be truth foot_1 . hence, we expect that propaganda enlarges the size of a textual unit where it appears. this unit is usually a sentence: according to @xcite , propaganda takes about half a sentence. and the sentences with propaganda tend to be longer, as reported by creators of the pig system at the hack news datathon foot_2 . the attribute of secrecy requires that propaganda looks like its non-propagandist context. so, it should not differ much from the context in its lexical, grammatical or stylistic expression: there should be a difference, but slight. @xcite look at four kinds of news articles (reliable, hoax, propagandistic and satiric) and find that they have linguistic features that are more frequent in fake news (the difference is statistically significant): pronouns 'i' and 'you' and their forms, modal, action, manner adverbs, words semantically related to swear, sexual, see, negation foot_3 , strong and weak subjective etc. 7 at the same time they enlist some features which are not in the fake news, but these features are fewer. as mentioned, propaganda should look like news, but with a bit of additional "foul" content. hence, the method of semantic vectors (with due regard for other methods) must be useful in propaganda detection to extract semantic features of the class "propaganda", but it will not be as useful in defining the class of "non-propaganda". to sum up, there are linguistic features that are more common in propaganda than in non-propaganda, and there are nearly no linguistic features of non-propaganda that are uncommon in propaganda (feature imbalance). further, propaganda is less common than non-propaganda (class imbalance). finally, propaganda takes about half a sentence; sentences with propaganda tend to be longer; propaganda is emotionally colored. ## system architecture detection of propaganda as a fragment of an article (fragment or span identification) presupposes that the minimum textual unit that holds it is a word form. however, if it neighbors with punctuation marks, numbers, etc. the latter can also be attributed to the fragment. hence, the per-token approach is very common in solutions to tasks organized by the propaganda analysis project: yoosuf and yang (2019) use bert to classify each token into 20 classes (18 propaganda techniques, none and "auxiliary"); @xcite create a token vector from a combination of features, including an embedding vector of a word and sentence; @xcite combine a vector of a token from three vectors of contextualized embeddings (elmo @xcite , bert @xcite , grover @xcite ); @xcite and @xcite use not only word but also character embeddings, as some morphemes, e.g. "-ist", can be frequent in propaganda. as mentioned, in this project we decided to focus on a fast solution that would allow us to test many hypotheses about our data. as spans should not contain just words, we chose a token as a minimum unit of a span, regardless whether it is a word form, a punctuation mark, etc. with tokens that are not actual words, it is important to rely on the context of each token -this is what we base our model on. a context window is a chunk of text that is within n tokens to the left and n tokens to the right from the given token; the adjustable parameter of the context window is its size n. if the token has too few tokens to the left or right, the context window simply shrinks. we tokenize texts with spacy @xcite : the encoreweblg model foot_5 . the machine learning algorithm that we finally chose is the logistic regression (lr) implemented in scikit-learn @xcite . the reasons for that were, first, the fastness of lr and, second, it proved to be one of the most efficient classifiers for the data models that we tested at the preliminary stage of research. also, lr can be considered a shallow neural network and, hence, a baseline to test how neural networks treat the data model. ## conclusion we have described our system of automatic propaganda detection at semeval-2020. the system is designed so as to allow us to quickly test many feature parameters. we call it a kitchen solution in the title: coming in small, but handy and utilizable blocks. in google colab, our classifier takes 141 seconds (without accelerator) and 128 seconds (with gpu accelerator) to train and calculate the result on the test data. the idea of feature imbalance has led us to using an unbalanced combination of propaganda and none token vectors in the data set. also, we have experimented with the size of a context window in which propaganda becomes most discernible and ended up with size 7, the number which peculiarly correlates with data on human short-term memory: "the 7-item limit" @xcite . it seems to us that this correlation is suggestive of the idea that, in the end, what we, as readers, are able to call propaganda, is propaganda.