Hidetsune at S em E val-2024 Task 4: An Application of Machine Learning to Multilingual Propagandistic Memes Identification Using Machine Translation (2024)

Abstract: In this system paper for SemEval-2024 Task4 subtask 2b, I present my approach to identifying propagandistic memes in multiple languages. I firstly establish a baseline for English and then implement the model into other languages (Bulgarian, North Macedonian and Arabic) by using machine translation. Data from other subtasks (subtask 1, subtask 2a) are also used in addition to data for this subtask, and additional data from Kaggle are concatenated to these in order to enhance the model. The results show a high reliability of my English baseline and a room for improvement of its implementation.

## introduction semeval 2024 task 4 @xcite calls for classification of memes into different persuasion techniques in textual content only (subtask 1) or in textual and visual content (subtask 2a), and identifying whether or not memes are propagandistic (subtask 2b). i participate in subtask 2b, which is a binary classification problem between "propagandistic" and "non_propagandistic". various memes are provided in english, bulgarian, north macedonian and arabic to determine whether the memes are propagandistic or not. my baseline is established so that it achieves fairly high accuracy in english. although it adopts a classical machine learning method with training, the training data are adjusted by being concatenated with additional dataset. after setting up my baseline, the model is implemented into other languages using machine translation. participating in this task allows me to test the ability of my model, achieving a fairly high score for its simplicity. in future studies, the model can be strengthened well enough with appropriately adjusted training data. as for the other languages, on the other hand, the implementation of my english baseline does not necessarily show consistent re-liability. although my baseline works relatively well for arabic to some extent, the scores go down drastically for bulgarian and north macedonian. one of the main reasons for this issue might be that accuracy of machine translation is not high enough, changing the original meanings of the memes and possibly making it more difficult for the english based model to identify propagandistic memes. my code is available on github foot_0 . ## background the subtask i participate in focuses on classification of memes. given a json file that has an id, an image name and text for each meme as input, the subtask requires assigning either "propagandistic" or "non_propagandistic" to the memes. in development phase, data in english were given and participants were allowed to test their solutions for the english data only. participants were told that they would also have unlabeled test data in three non-english languages as the evaluation phase starts, meaning that no information was released about non-english languages in the development phase. propagandistic memes on social media have become a growing issue in the past few years. as more and more people use social media platforms, many types of information including propagandistic one is spread to a number of people (bradshaw and howard, 2019). these days, issues caused by propaganda on social media have become worse than most people might think, as a study by @xcite gives a warning. in fact, these memes did change people's thoughts for elections influencing people's voting behaviors @xcite . therefore, it might become more and more important in the future for nlp to be able to recognize whether or not the information uses persuasion techniques with high accuracy. there are already many previous studies that adopt modern nlp techniques for propagandistic memes detection. a previous study by @xcite uses roberta, which is the stateof-the-art pre-trained language model at the time, resulting in a f1 score of 60.2%. another previous study by @xcite tries modern large language models including gpt-3 and gpt-4, also resulting in reliable scores. @xcite utilizes combinations of multiple deep learning models including bert, bilstm and xgboost with accuracy of around 0.67 in f1 score. on the other hand of these previous studies, my methodology intends to test classical machine learning approach rather than state-of-theart llms. during my methodology, spacy-v3 model @xcite is trained with over 20000 memes in english, showing high reliability with accuracy of 0.71353 in f1 macro. then, the trained model is implemented into non-english languages (bulgarian, north macedonian and arabic) by using google translate @xcite . my participation in this task reveals a high potential of applying classical nlp methods to detection of propagandistic memes with properly adjusted training data. the results reveal that a fairly high score can be achieved without state-of-the-art llms and complicated methods. although the direct implementation of my english baseline into other languages has a room for improvement, the accuracy might go up easily with better machine translation models and careful consideration of differences in topics behind the memes. this paper introduces both strengths and weaknesses of my approach, guiding a direction to future application of classical machine learning to a modern issue of propagandistic memes that requires automatic binary classification. ## system overview the main strategy of my system is a classical machine learning method for english baseline and implementation of it into other languages using machine translation. a quick overview of my algorithm is as follows. the system imports a prepared training csv file as a pandas dataframe, where all the data from task 4 (train and validation data of subtask 1, subtask 2a and subtask 2b) and additional data from kaggle 2 are concatenated to compose a large training data with 20774 rows. only first 20000 rows are used as for the additional data 2 due to its large data size, and all the rows that exceed the limitation of 11001 memes (for both "propagandistic" and "non_propagandistic") are eliminated from the concatenated dataset. memes in the data are all in english for the purpose of establishing a baseline that guarantees fairly high accuracy in english. then, spacy-v3 model @xcite ) is trained using the resulting data, and test data with unlabeled memes in english is imported as a json file. after that, the trained model is used to assign either "propagandistic" or "non_propagandistic" to each unlabeled meme. as for other languages (bulgarian, north macedonian and arabic), the memes are translated into english so that my english baseline can be implemented into them. google translate @xcite ) is adopted in this part, and the same trained model is used similarly to the english baseline after translation. my participation in this task allows for testing the classical nlp approach and simple implementation of it using machine translation. the english baseline, which uses classical machine learning methods, achieves its certain ability to identify propagandistic memes with a reliable score. on the other hand, the scores go down as for non-english languages. this might be because of slight changes in meanings in the translation process, which turns out to be the biggest weakness of my approach. classification of english memes is fairly difficult even for humans given a single utterance. for instance, "vote republican. they may not be perfect. but the other side is insane." can be easily classified as propagandistic, but sentences like "cry all you want...... he's doing exactly what i hired him for......" might not be that easy for the classification because their nuance might depend on situations (on sns or at work etc.) to some extent. since propagandistic memes classification is difficult in this way, possible changes of original meanings by machine translation might have resulted in a serious issue for classification of test data, lowering the scores of other languages dramatically. another possible reason for the lowered performance in non-english languages is that there are large differences in topics. words like "trump" and "russia" frequently appear in english memes, but "bulgarian" is one of the most frequently used words in north macedonian memes. since bulgaria and north macedonia have some diplomatic issues (kamberi, 2023), memes that have basis on them ("the bulgarians enter the constitution" for example) tend to appear frequently. this kind of differences in topics probably caused the lowered accuracy in non-english languages, revealing a new challenging problem of the approach to implement my english baseline into other languages. ## experimental setup before moving on to the actual training of the model, data preparation was an essential part of my solution. first of all, all the data of this task (including other subtasks) are imported as json files and converted into pandas dataframes. the dataframes have "text" and "propagandistic/non_propagandistic" as columns. training data and validation data from both subtask 1 and subtask 2 are composed of "propagandistic" only, so they are concatenated as "large_data1", which is a dataframe with 8307 rows and all "propagandistic" memes. also, data from kaggle 2 are imported as all "non_propagandistic" dataframe and processed so that it has no usernames in "text" column. validation data of subtask 2b are previously imported, but i decided to increase weight of training data of the reason why the additional data from kaggle 2 are chosen is that they are less likely to be propagandistic compared to many other existing datasets. the data are from customer support on twitter including applesupport and amazonhelp, so the topic there should be something related to their products or services. there are many other existing datasets extracted from social media platforms, but it takes too much time end effort to manually assign propagandistic and non-propagandistic to each utterances. for the purpose of getting nonpropagandistic utterances only, it might be one of the easiest and most realistic approach to find an existing dataset whose topic is clearly unrelated to politics and diplomacy as included in my solution. after these data preparation steps, the model @xcite is trained with the training dataset, and test data with unlabeled memes are imported as a json file. the memes in non-english languages are translated into english as stated in the previous section. the trained model assigns either "propagandistic" or "non_propagandistic" to each meme. the training data and test data are cleaned with new line removal and lower casing prior to use of them. ## results as for the non-english languages, the scores go down due to the potential reasons as stated in prior sections. even so, arabic has relatively high accuracy, which is lower than english by around 0.18 in f1 macro but higher than bulgarian by around 0.20. this difference might have been caused by accuracy of machine translation mainly. arabic is a widely used language with a total of about 372.7 million native speakers in the world. 3 there is a possibility that google translate @xcite , which i use for machine translation, has higher accuracy for widely used languages including arabic, maintaining the original meanings and nuances of the memes fairly correctly. ## conclusions to summarise, my methodology firstly focuses on establishing a baseline that guarantee fairly high accuracy for english memes. after that, the baseline is implemented into non-english languages by translating the memes into english using machine translation. the results show high reliability of my english baseline. the methodology for the baseline has its basis on classical machine learning, but my participation in this task reveals its fundamental abilities to deal with complicated classification task with properly adjusted training data. on the other hand, the results also show that the simple application of my english baseline has a room for improvement. the scores of non-english languages dramatically dropped although arabic has relatively reasonable accuracy compared to bulgarian and north macedonian. there can be many possible reasons for this including the accuracy of machine translation and changes in topics between memes in different languages. in future studies, it might be worthwhile to enhance the model with many more memes for my english baseline. collecting propagandistic memes might be a time consuming task, but nonpropagandistic memes, on the other hand, can be easily found and used by utilizing existing datasets whose topics clearly have nothing to do with politics and diplomacy. as for non-english languages, higher accuracy might be achieved by using better machine translation models and enhancing the baseline model with specific political or diplomatic topics in the countries.