TUB at WANLP 22 Shared Task: Using Semantic Similarity for Propaganda Detection in A rabic (2022)

Abstract: Propaganda and the spreading of fake news through social media have become a serious problem in recent years. In this paper we present our approach for the shared task on propaganda detection in Arabic in which the goal is to identify propaganda techniques in the Arabic social media text. We propose a semantic similarity detection model to compare text in the test set with the sentences in the train set to find the most similar instances. The label of the target text is obtained from the most similar texts in the train set. The proposed model obtained the micro F1 score of 0.494 on the text data set.

## introduction social media has played a crucial role in recent year, having a great impact on different areas such as communication, entertainment, and politics. beside their positive applications, social networks became an easily accessible medium to spread disinformation and propaganda in recent years. based on hamilton (2021), propaganda differs from mis/disinformation in that it need not be false, but instead, it relies on rhetorical devices which aim to manipulate the audience into a particular belief or behavior @xcite . in this paper we present our proposed approach for sub-task 1 of the propaganda detection in arabic social media text shared task (wanlp 2022). wanlp 2022 shared task includes two sub-tasks; identifying the propaganda techniques in tweets as a multi-label text classification task, and identifying the propaganda techniques used in tweet together with the span(s) of text in which each propaganda technique appears as a sequence tagging task. we only submitted results for the first subtask of the shared task. in this sub-task one or more propaganda techniques have been assigned to arabic texts from social media @xcite . there are 21 propaganda techniques in the dataset that represent different approaches to manipulate the audience. more details about the different tasks can be found on the web-page of the shared task 1 . although the first sub-task (identify the propaganda techniques) in a multi-label text classification problem, we used semantic textual similarity (sts) methods to identify related propaganda techniques to the instances in the test set. based on the obtained results sts methods show competitive performance to the text classification models for the task. the rest of the paper is organized as follow; section 2 presents recent research on text based propaganda detection in social media. an overview of the data and the proposed approach are presented in sections 3 and 4, respectively. we briefly review the obtained result and discuss it in section 5. finally, we conclude the paper and the system in section 6. ## related work in this section we highlight some of the recent models for the task of propaganda detection using machine learning and natural language processing (nlp) methods. the task of propaganda detection could be analyzed from two different perspectives; text analysis and network analysis perspectives @xcite . here we focus on text analysis based models and review the recently developed models based on deep neural networks and pre-trained language models. @xcite proposed a model to detect propaganda in sentence level based on the bert @xcite and an bilstm model. they formulated the task of propaganda detection as a binary classification task in which the model should distinguish propaganda and non-propaganda contents. the proposed model includes fine-tuning a pre-trained model on the task of emotion classification and feeding the output into a bilstm architecture. the obtained results show that the model can significantly exceeds the baseline approach @xcite . in another study vorakitphan et al. (2021) developed "protect" model for propaganda detection in text. as a propaganda detection pipeline, "protect" extract the text snippets from the input text, and then classify the technique of propaganda. the text snippets extractor module uses bert pre-trained language model and feed the extracted text into the next step that is propaganda detection module. roberta @xcite pre-trained model is used for the classification task and propaganda detection. as another research on propaganda detection as a text classification task, @xcite proposed a binary text classifier based on different features includes readability level and writing style. they compared the performance of different supervised models such as logistic regression and svms for the task. in addition to only text based models, some multimodal models have also been proposed in recent year to detect propaganda not only on text but also on images. as one of these efforts, a data set and a model are developed by @xcite to propaganda identification in a multimodal setting. the compiled data set in this research contains 950 memes, each annotated with 22 propaganda techniques. it is collected from facebook in includes different topics such as vaccines, covid-19, and gender equality. they also proposed four different models, two unimodal and two multimodal models. for the unimodal setting, they used @xcite for the text-and imagebased models, respectively. the obtained results on the proposed data set show that the multimodal approach can outperform the unimodal training approaches. in the proposed propaganda identification approach in this paper, we developed a model based on semantic similarity techniques unlike the highlighted researches in this section that formulated the task as a text classification problem. ## data in this section we briefly describe the dataset for the first sub-task of wanlp 2022. the task organizers provided four data sets for sub-task 1 that includes train, development, dev_test and test sets. all the data sets are provided in json format that includes an id, the text of tweets and a list of labels for each instance. there are a total number of 21 propaganda techniques in the data sets @xcite . most of the techniques are presented and described in @xcite . the distribution of ten most frequent techniques in the train set is depicted in figure 1 . as it is highlighted in the figure, "loaded language" is the most frequent propaganda techniques, followed by "name calling/labeling". as the pre-processing step, we replaced twitter handles (i.e., the usernames) and urls with constant texts ("username" and "weblink" respectively). these pre-processing steps have shown promising impact on the overal performance of related tasks like hate speech detection @xcite . although replacing urls with the content of pages they refer to could be effective in related tasks like fake news detection (mohtaj and möller, 2022a), we decided to replace them with a constant text to prevent changing the length of input texts, drastically. the python regular expression package (re) has been used to replace the above mentioned texts in the data. text. although the first sub-task of the competition is a multi-label text classification task, we decided to use an sts based model to detect the most probable techniques for the instances in test set. from the provided definition for different propaganda techniques in @xcite and also from the provided data sets, lexicons play an important rule in a number of the techniques. for instance, "loaded language" as the most frequent technique in the data set is defined as "using specific words and phrases with strong emotional implications to influence an audience" in @xcite . here, emotion lexicons are the main indicator of this propaganda technique. considering the role of lexicons and keywords on these techniques, using lists of related lexicons for each propaganda technique could show promising results on the task. however, due to lack of access to such resources in arabic, we proposed an sts based approach to find the most similar instances in the train set to the unlabeled texts in the test set. word embedding models have shown promising performance on nlp tasks related to semantic textual similarity (e.g., plagiarism detection) @xcite . however, it has been shown that the state-of-the-art contextual word embedding models (e.g., bert) can outperform the traditional models in different nlp tasks like word similarity detection (gupta and jaggi, 2021) and hate speech and fake news detection (mohtaj and möller, 2022b). the overall proposed approach for wanlp 2022 to identify the most probable propaganda techniques for the instances in the test set is presented in figure 2 . in the proposed model we used the arabic bert model @xcite to convert all the instances in the train and test sets into contextual vectors. we averaged word vectors to obtain the vector representation of sentences which resulted a vector with the length of 768 for each instance in the data sets. as the next step, we computed the cosine similarity between each instance in the test set (i.e., target sentence) with all of the instances in the train set. we took n most similar instance to the target sentence where the similarity is higher than a threshold. we named them as candidate sentences. finally, top t frequent techniques in the candidate sentences have been chosen as the label of the target sentence. more details about the experiments are presented in section 5. ## results and discussion in this section we briefly present our results based on the above mentioned model and discuss the main findings in the experiments. as it is mentioned in the previous section, we tested three main hyper-parameters in our experiments; number of candidate sentences (n), minimum similarity threshold (threshold), and number of most frequent techniques to take from the candidate sentences (t). one possibility to improve the overall performance of the proposed model would be using more than one hidden layer to convert the input text into vectors. in some studies on similar tasks like hate speech and fake news detection, it has been shown that using more than one layer for embedding could improve the performance of classification models (mohtaj and möller, 2022b). ## conclusion in this paper we presented our model for the subtask 1 of the propaganda detection in arabic social media text shared task (wanlp 2022). we proposed a model based on semantic textual similarity to compare instances in the test set with the labeled instances in the train set. the label of the test sentences obtained from the most similar sentences in the train set. based on the results on the test set, the sts based model show a competitive performance compared to classification based models for the task. for the future work, one can use different pretrained language models to convert raw input texts into vectors. moreover, the hyper-parameters can be tuned in order to improve the overall performance of the model. the north american chapter of the association for computational linguistics: human language technologies, naacl-hlt 2019, minneapolis, mn, usa, june 2-7, 2019, volume 1 (long and short papers), pages 4171-4186. association for computational linguistics. dimitar dimitrov, bishr bin ali, shaden shaar, firoj alam, fabrizio silvestri, hamed firooz, preslav nakov, and giovanni da san martino. 2021. detecting propaganda techniques in memes. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 6603-6617, online. association for computational linguistics. prakhar gupta and martin jaggi. 2021. obtaining better static word embeddings using contextual embedding models. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing, acl/ijcnlp 2021, (volume 1: long papers), virtual event, august 1-6, 2021, pages 5241-5253. association for computational linguistics. kyle hamilton. 2021. towards an ontology for propaganda detection in news articles. in the semantic web: eswc 2021 satellite events, pages 230-241, cham. springer international publishing. kaiming he, xiangyu zhang, shaoqing ren, and jian sun. 2016. deep residual learning for image recognition. in 2016 ieee conference on computer vision and pattern recognition, cvpr 2016, las vegas, nv, usa, june 27-30, 2016, pages 770-778. ieee computer society. yinhan liu, myle ott, naman goyal, jingfei du, mandar joshi, danqi chen, omer levy, mike lewis, luke zettlemoyer, and veselin stoyanov. 2019. roberta: a robustly optimized bert pretraining approach. corr, abs/1907.11692. giovanni da san martino, stefano cresci, alberto barrón-cedeño, seunghak yu, roberto di pietro, and preslav nakov. 2020. a survey on computational propaganda detection. in proceedings of the twenty-ninth international joint conference on artificial intelligence, ijcai 2020, pages 4826-4832. ijcai.org. salar mohtaj and sebastian möller. 2022a. the impact of pre-processing on the performance of automated fake news detection. in experimental ir meets multilinguality, multimodality, and interaction -13th international conference of the clef association, clef 2022, bologna, italy, september 5-8, 2022, proceedings, volume 13390 of lecture notes in computer science, pages 93-102. springer. salar mohtaj and sebastian möller. 2022b. on the importance of word embedding in automated harmful information detection. in text, speech, and dialogue -25th international conference, tsd 2022, brno, czech republic, september 6-9, 2022, proceedings, volume 13502 of lecture notes in computer science, pages 251-262. springer. salar mohtaj, vinicius woloszyn, and sebastian möller. 2020. tub at hasoc 2020: character based lstm for hate speech detection in indo-european languages. in working notes of fire 2020 -forum for information retrieval evaluation, hyderabad, india, december 16-20, 2020, volume 2826 of ceur workshop proceedings, pages 298-303. ceur-ws.org. ali safaya, moutasem abdullatif, and deniz yuret. 2020. kuisail at semeval-2020 task 12: bert-cnn for offensive speech identification in social media. in proceedings of the fourteenth workshop on semantic evaluation, pages 2054-2059, barcelona (online). international committee for computational linguistics. george-alexandru vlad, mircea-adrian tanase, cristian onose, and dumitru-clementin cercel. 2019. sentence-level propaganda detection in news articles with transfer learning and bert-bilstm-capsule model. in proceedings of the second workshop on natural language processing for internet freedom: censorship, disinformation, and propaganda, pages 148-154, hong kong, china. association for computational linguistics. vorakit vorakitphan, elena cabrio, and serena villata. 2021. protect -a pipeline for propaganda detection and classification. in proceedings of the eighth italian conference on computational linguistics, clic-it 2021, milan, italy, january 26-28, 2022, volume 3033 of ceur workshop proceedings. ceur-ws.org.