C havan K ane at WANLP 2022 Shared Task: Large Language Models for Multi-label Propaganda Detection (2022)

Abstract: The spread of propaganda through the internet has increased drastically over the past years. Lately, propaganda detection has started gaining importance because of the negative impact it has on society. In this work, we describe our approach for the WANLP 2022 shared task which handles the task of propaganda detection in a multi-label setting. The task demands the model to label the given text as having one or more types of propaganda techniques. There are a total of 21 propaganda techniques to be detected. We show that an ensemble of five models performs the best on the task, scoring a micro-F1 score of 59.73%. We also conduct comprehensive ablations and propose various future directions for this work.

## introduction the advent of social media has enabled people to view, create and share information easily on the internet. such information can easily be accessed and viewed by a very large number of people in surprisingly short periods. moreover, most social media websites have few restrictions over what the users choose to post and lack preemptive techniques to censor posts before they are uploaded. this has enabled the free flow of information from various strata of society which might have been restricted due to the lack of access to proper news sources. however, this has also led to a stark increase in the spread of propaganda through the internet. information propagated through social media posts presents an individual's personal opinions, and hence is often biased and lacks rigorous fact-checking. such problems are less frequently found in the original media sources of newspapers and tv news channels where their posts are subjected to a higher level of scrutiny. the presence of propaganda online poses a serious threat to society as it can often polarize the * equal contribution majority opinion and lead to violent events. a wave of misinformation-based propaganda during the time of the covid-19 pandemic @xcite was observed. however, the problem of propaganda detection is much more complicated than it appears. the biggest challenge in propaganda detection is that the bulk of propaganda information is partially based on truths, but is presented in a manner that might be misleading or unnecessarily polarizing. it is also observed that propaganda posts are written professionally and are compelling which makes most of the readers believe the information to be authentic. all of these problems make it difficult to train a model to detect propaganda, and much more difficult to interpret the results of such models. the purpose of the shared task @xcite , a multi-label classification problem, is to come up with efficient methods for detecting propaganda on a dataset containing arabic tweets. transformer-based models have achieved great success in text classification tasks. additionally, ensemble-based models also outperform these individual models. thus, we explore individual as well as ensemble of models for this task. furthermore, we experimented with oversampling where we repeat the samples having minority labels. we also pretrained the dehatebert model on 1 million tweets to study the effect of domain-specific pretraining on downstream performance. we report the results of all these experiments and thereby propose an ensemble-based method for this task. ## data the dataset consists of 504 training examples, 52 validation examples, and 52 testing examples. our models were finally evaluated on a separate testing dataset, which consisted of 323 examples. each example can have one or more of the 20 propagandist techniques foot_0 . thus, it was a multi-label dataset. the number of label occurrences is illustrated in figure 2 . as shown in the figure, we see a skewed distribution. this shows that there is an imbalance. given this problem of multi-label classification with a high class imbalance, we experimented with several architectures and found that dehatebert performed the best on the dataset. a full account of all of our successful experiments, as well as failed experiments, is given in sections 5 and 6. we try multiple methods to mitigate this imbalance, as elaborated in section 6. since the dataset is a multilabel dataset, used one-hot encoding for each label to denote the ground truth labels. furthermore, we make some key observations about the number of labels per example in figure 1 . we observe that most examples have one label per example. we see that the number of examples having more than one label diminishes quickly, with only one example having 7 labels. we use basic preprocessing to minimize the noise in the inputs. firstly, we remove all links in the tweet. then we remove the user mentions and hashtags (denoted by "@" and "#" followed by a string respectively). finally, we replace underscores ("_") with space. this way, the separated words contribute to the semantics of the sentence. note that we retain the emojis in the sentence since they also carry significant meaning and can aid the model to better detect sentiment. ## system given this problem of multi-label classification with a high class imbalance, we experimented with several architectures and found that dehatebert performed the best on the dataset. a full account of all of our successful experiments, as well as failed experiments, is given in sections 5 and 6. we tried several models, namely arabert v1, v02 and v2, marbert, arbert, xlmroberta @xcite , araelectra @xcite . note that the difference between arabertv2 and arabertv02 is that the former uses presegmented text whereas the latter uses the farasa segmenter @xcite to segment the text since arabic is a language which requires its words to be segmented before being fed into the tokenizer. we used a specific variant of dehatebert, which is initialized from multilingual bert and fine-tuned only on arabic datasets. we found that this particular variant performed the amongst the best, in terms of micro-f1 on the test split of our dataset. our model training is fairly straightforward. we train dehatebert on our multi-label dataset for 30 epochs and the best performing epoch is chosen based on validation micro- the dataset has a significant class imbalance. to overcome this, we tried to augment the dataset by oversampling. for oversampling, we duplicated the samples containing less frequent target classes. thus we obtained a larger dataset containing duplicate samples but overall having lesser class imbalance. however, this did not yield better performance. we discuss this in detail in section 6. ## results the official scoring metric for the shared task is the f1 micro score. we present the results of the various models we tried in we can however see that arbert outperforms all other single models. another key observation is that arbert outperforms marbert, which in turn outperforms all variants of arabert. an explanation for this is that arabert variants are trained on far less data than arbert and mar-bert. in the case of arbert and marbert, arbert is pretrained on a wide variety of sources as opposed to marbert and thus has better performance than marbert. we can also speculate that the high performance of the ensemble is because the constituent models are pretrained on different datasets. this enables the ensemble to capture a wider array of semantic vocabulary and hence is better at predicting classes. the hard voting mechanism ensures that the ensemble will not predict too many classes for each sample and thus limits the number of false positives. ## discussion we conducted several experiments apart from our best-performing model. specifically, we tried pretraining on a large arabic sentiment analysis tweet dataset as well as oversampling the classes having few samples. we retrained the dehatebert model on 1 million tweets from the large arabic twitter data for sentiment analysis dataset using the masked language modeling technique. we found that pretraining on the sentiment analysis tweet dataset did not result in any gains to the model. we speculate this is primarily because the number of tweets we pretrained the model on is less than the size the model was originally pretrained on. in another attempt, we implement oversampling in the dataset, where we repeat samples of less frequent classes. we calculate the average number of examples for each class. then, we get the oversampling factor, that is the number of times the examples must be repeated to reach the average number of samples. we further clip this factor to 10. note that, since this is a multi-label scenario, we need to be careful not to use examples with the most frequently occurring classes, in which case the process will have no effect. currently, we use hard voting for choosing the final output of the ensemble. we believe better results can be obtained by having a more sophisticated method like using an svm instead of hard voting. ## conclusion this paper aims to articulate our approach for the wanlp 2022 shared task. we experimented with multiple transformer-based models, namely arabert, arbert, marbert and others. we also present ablations with monolingual pretraining, oversampling, and ensemble of the aforementioned transformer-based models. we show that the ensemble consisting of models pretrained on various sources of data has the best performance, with a micro-f1 score of 59.73%. we foresee several possible future directions. one line of work can be to improve the ensemble mechanism as well as to better handle the class imbalance in multi-label setting. another line of work can be to study the effects of domain-specific pretraining on downstream classification tasks like multi-label classification.