NLPIITR at S em E val-2021 Task 6: R o BERT a Model with Data Augmentation for Persuasion Techniques Detection (2021)

Abstract: This paper describes and examines different systems to address Task 6 of SemEval-2021: Detection of Persuasion Techniques In Texts And Images, Subtask 1. The task aims to build a model for identifying rhetorical and psycho- logical techniques (such as causal oversimplification, name-calling, smear) in the textual content of a meme which is often used in a disinformation campaign to influence the users. The paper provides an extensive comparison among various machine learning systems as a solution to the task. We elaborate on the pre-processing of the text data in favor of the task and present ways to overcome the class imbalance. The results show that fine-tuning a RoBERTa model gave the best results with an F1-Micro score of 0.51 on the development set.

## introduction the purpose of this task was to identify propaganda in electoral campaigns where memes are one of the most popular types of content used to carry out disinformation campaigns. they are most effective on social media platforms since they can easily reach many users. semeval-2021 task 6 @xcite refers to propaganda whenever information is purposefully shaped to foster a predetermined plan. propaganda uses psychological and rhetorical techniques to achieve its objectives. such techniques include the use of logical fallacies and appealing to the emotions of the audience. logical fallacies are usually hard to spot since the argumentation, at first sight, might seem correct and objective. however, a careful analysis shows that the conclusion cannot be drawn from the premise without the misuse of logical rules. another set of techniques uses emotional language to induce the audience to agree with the speaker only based on the emotional bond that is being created, provok-ing the suspension of any rational analysis of the argumentation. the oxford dictionary defines a meme as 'an image, video, piece of text, etc., typically humorous in nature, that is copied and spread rapidly by internet users, often with slight variations'. they generally consist of an image superimposed with text. the role of the image in a deceptive meme is either to reinforce/complement a technique in the text or to convey one or more persuasion techniques. a total of 3 subtasks were defined: for subtasks 1 and 2, only the meme's textual content was given. the former was a multilabel classification problem while the latter was a multilabel sequence tagging problem, each with 20 labels of techniques. for subtask 3, both textual and visual content of the memes were given. the goal was to identify which of the techniques are used for a given text/image. in this paper, we describe a system to address the subtask 1 for which, a training dataset of 600 sentences with their labels was given. the test set consists of over 200 sentences. we evaluated the performance of all the models using the micro and macro f1 scores. we make use of 13 different models to approach the problem at hand, experimenting with different data preprocessing techniques to finally document 23 sets of results. we found that strong results can be achieved by fine-tuning a pre-trained roberta language model. however, we also discovered that some other models, when used as part of a voting classifier, gave decent results without requiring resources and time needed to train a roberta model. the rest of the paper is organized as follows. section 2 discusses the related work. section 3 talks about the methodology. section 4 describes the dataset for the task. section 5 provides the experimental setup. section 6 presents the systems implemented to address the task. section 7 shows the results and section 8 concludes the paper. ## related work the closest research to persuasion detection in computer science is perspective detection. @xcite discussed different types of machinelearning classifiers such as naïve bayes and support vector machines for the perspective detection. another topic, closely knitted with the detection of persuasion techniques is hate speech detection. @xcite worked on detecting personal insults, profanity, and user posts characterized by malicious intent. @xcite focused on vulgar language and profanity-related offensive content. @xcite further look into jokingly formulated teasing in messages that represent bullying episodes. @xcite specifically looked into othering language, characterized by an 'us' and 'them' dichotomy in racist communication. approaches to detecting hate speech on twitter using convolutional neural networks and convolution-gru-based deep neural networks are discussed in @xcite and @xcite respectively. persuasion technique detection is a multi-label classification problem. many attempts have been made in the literature to make a multi-label classification system @xcite @xcite @xcite . ## method in this paper, we make use of a one-versus-rest strategy to split a multi-label classification dataset into binary classification problems, splitting the data into one binary dataset for each class @xcite , called 'pairwise coupling'. we present results with various approaches for the persuasion techniques detection in the text data. we make use of pre-defined language models and tune the hyperparameter for best performance on the dataset. soft voting classifier and hard voting classifier @xcite are deployed to use traditional machine learning models for the task. a soft voting classifier uses the average of probabilities for each class to classify the test instance, whereas, in hard voting, the classifier assigns the class that was voted by a majority of the classifiers. in our case, since we implemented our own voting classifier, we did not always go with the majority but experimented with different threshold values. in some cases, we gave more weightage to the models which gave more promising results. various models, such as logistic regression, naive bayes, svm, rnn, bilstm, bert, naming a few, were implemented, which are discussed in the later sections. roberta model performed the best for the persuasion techniques detection task on the text data (section 7). ## dataset in this paper, we provide systems to address subtask-1. the input data for subtask-1 is the text extracted from the memes. the training, the development, and the test sets for subtask-1 were distributed as json files. an object of the json has the following type: { "id": "125" "labels": [ "loaded language", "name calling/labeling" ] "text": "i hate trump \n\nmost terror-ist do" } where below are the technique names with their frequency in the training dataset. ## experimental setup we train a wide range of different models for the task. as discussed in section 3, we used both a soft voting classifier and a hard voting classifier on the traditional machine learning models for better performance. however, it should be noted that a soft voting classifier cannot be used in all the models, and accordingly, models were categorized. in order to assign different weights to different models, as per the individual performance, we used our own version of the hard voting classifier by scikit-learn @xcite . most of the models that we implemented support a multi-class classification where one assumes that each sample is assigned to one and only one label. however, since we were in a multi-label classification scenario, where we were required to assign a set of target labels, we wrapped these models in a onevsrestclassifier. we converted the multi-label classification problem into a series of binary classification problems assuming that there was not any underlying correlation between any two labels and they were mutually exclusive. below is the list of the machine learning and deep learning approaches that we implemented to build the systems to identify persuasive techniques. evaluation measures: the given evaluation measure for the task was micro-f1. we also report macro-f1 in this paper as it is more favorable for multi-class classification problems. ## system description we split the training data into a training and validation set in the ratio 9 : 1, i.e., 90% of the corpus was used for training while the remaining 10% was part of the validation set. the preprocessing (section 6.3) for all the models was the same and once the best model was found, it was retrained including the validation set for training as well. the trained model was then used to predict labels for the development set. the model yielding the best accuracy on the earlier chosen validation set was used for the submission of the development set. we followed a similar approach for the submission of the results in the task on the test set using roberta model. we secured 15th rank in the task. ## results and analysis firstly, even though there is a visible gap between roberta and the rest of the models, the ensemble of models was almost 200 times faster on a cpu compared to roberta, which was trained on a gpu. this indicates that the former is a superior choice when quick nlp predictions are required using low-end systems. next, we observe the effect of tackling the data bias technique by choosing some specific augmented sentences for the training corpus. in 'all models together', even though a better result was reported by the system without the drop and the threshold value 0, overall, dropping the sentences gave better performance for different threshold values. one can note that the latter system even gave better f1-macro score universally for all thresholds. finally, we adjudge a rather interesting observation. most of the systems comprehensively work far better with the augmented data, but the deep neural network, which contained 1.5 million trainable parameters, showed a startling outcome. the score obtained after training on the base corpus was better than the one obtained after training on much larger augmented data. repeated back-translation might be the reason that we fed the similar type of data instances to the system. thus, the model was not able to learn enough, giving poor accuracy and faulty predictions. we didn't see the similar observation in traditional machine learning models or even mlp regressors because these models use fewer parameters. ## conclusion persuasion techniques detection is a multi-label classification problem. this paper presents and analyzes 13 machine learning and deep learningbased systems for persuasion techniques detection in the text data. persuasion data has an extensive range of techniques as labels (20 in our case) with high-class imbalance. we observe that the data augmentation technique, i.e., back translation, helps overcome class imbalance and produces more robust systems. besides, we present the essential data preprocessing for the task. results show that roberta, a deep neural language model, outperformed other systems by a significant margin. fine tuning the roberta model on the training data captures the sensitive features for persuasion techniques identification.