UI at S em E val-2020 Task 8: Text-Image Fusion for Sentiment Classification (2020)

Abstract: This paper describes our system, UI, for task A: Sentiment Classification in SemEval-2020 Task 8 Memotion Analysis. We use a common traditional machine learning, which is SVM, by utilizing the combination of text and images features. The data consist text that extracted from memes and the images of memes. We employ n-gram language model for text features and pre-trained model,VGG-16,for image features. After obtaining both features from text and images in form of 2-dimensional arrays, we concatenate and classify the final features using SVM. The experiment results show SVM achieved 35% for its F1 macro, which is 0.132 points or 13.2% above the baseline model.

## introduction semeval-2020 task 8 memotion analysis consists three tasks, which are sentiment classification, humor classification, and scales of semantic classes foot_0 . for this research, we chose to do the task a because we have interest in sentiment analysis area. in task a: sentiment classification, we need to classify the sentiment of given english memes to three sentiment categories, which are positive, negative, and neutral. sentiment analysis is one of nlp task that conducted for extracting the sentiment, emotions, or judgement of reviews and classified it. mainly, sentiment analysis task requires dataset in form of text. however, the opinion that people express is not only in form of text, such as comments, reviews, but also in other form, for example, image with text, or can be called memes. according to @xcite , an internet meme is a piece of culture, typically a joke, which gains influence through online transmission. in social media, usually the memes are expressed in images that contain short text which express the author opinion towards object. this makes classifying the sentiment of memes a challenge since the opinion in memes can be expressed explicit or implicitly. in this study, we conducted sentiment classification task on memes by combining the features from images and text, then classified it as positive, negative, or neutral. by using memes as data, it is believed that we can obtained better sentiment classification results because the other data source such as images, can enhance the robustness of models @xcite . the data that used were obtained from semeval 2020 task 8: memotion analysis @xcite . as we want to see how traditional machine learning works in this case of classification, we compared four machine learning algorithms and a neural network. our system shows that it can achieve high scores in the testing set compared to the baseline scores even though the score was still considered low score. the rest of this paper is arranged as follows: in section 2, we review the related works with our study. section 3 talks about the data that we used, including training and testing data. we describe the research steps that applied in this work and the result from that we obtained in section 4. in section 5, we analysis the result and error. then we conclude our research in the section 6. 2 related work there are several researches that using text and images as the features for their research. for in-stance, a study conducted by sabat et al. (2019) for detecting the hate speech using multi-layer perceptron and calculating its scores. the results show highest scores were obtained by image and text features fusion compared to only using images or text features. next is the study conducted by audebert et al. (2019) about document classification using two document datasets that contain digitized documents. the features that used were document embedding combined features extracted using pre-trained model, mobilenetv2. the results show that the overall performances from combination of the features from text and images are better than only using image or text as features in both datasets. another study conducted by @xcite . they proposed a target-oriented multimodal using a pre-trained language representations, bert architecture, for detecting the sentiment. the study also shows the good performances while using images and text features for sentiment classification. in contrast, a research by @xcite about multimodal model to detect hate speech cannot outperform the textual model even though images are useful for hate speech detection. according to them, the reason can be caused by noisy data, complexity of multimodal relations, and small set of samples. in our research, we focus of our research is to see the performance of machine learning when classifying using the combination of text and visual features. we compared few algorithms before selecting the best model and use n-gram language model and a pre-trained model, vgg-16 @xcite for extracting text and images features respectively, before combining and classifying the sentiment. ## data in this part, we show the descriptions of datasets that we used. there are three datasets that provided by semeval-2020 task 8, which are trial dataset, training dataset, and testing dataset. however, in our research, we only used training data for model building, then predict the classes for testing data. ## system description this section will describe the flow of our system we made in this study that will be implemented in training and testing data. our system uses few libraries such as pandas @xcite and numpy @xcite for data processing, scikit-learn @xcite ) for text pre-processing, feature extraction for text, and classification models. we also use keras (chollet and others, 2015) for image pre-processing, and feature extraction for image. in classification step for training data, we compared three machine learning algorithms for classification which are multinomial naïve bayes, support vector machine @xcite , and random forest @xcite , and neural network multi-layer perceptron. after that, we selected the best model and predict the sentiment of testing data. the flow of our system classification model consists six stages that can be seen at figure 3 . since the data have two text columns, "text ocr" and "text corrected", we chose "text corrected" as our base text and made new "text" column from it. after that, we filled the null rows in our "text" column with "text ocr" and dropped the rows that still remained null in "text" column. then, we filtered out images that are not in data-frame list names, since there were images that are not listed in it. for the testing data, we did the same steps, but we did not drop the null rows from its dataframe and keep the remain null rows because we need to predict all the instances in the testing data. we also merged the positive and very positive labels as well as negative and very negative in this stage. after that, we flattened the result into 2-dimensional array. • stage 4 (feature concatenation): after obtained the features from both text and images in 2dimensional arrays, we concatenated them using numpy function. then, we used these concatenated features as final features for classifying the sentiment of memes. ## experimental result and analysis in this section, we show our experimental results in all datasets that provided. the first part shows the results using training data when classified using text features only, combination of text and images features, and images features only. for the second part, it shows the official scores from organizer that achieved by our model when predicting the testing data. ## conclusion in this research, we built ui, a traditional machine learning based system for classifying the sentiment of memes by utilizing the combination of text and images features. the text features were extracted by implementing n-gram language model while images features were using vgg-16. after that, we examined the performance of three machine learning algorithms using training data for classifying memes before selecting the final model. the algorithms are multinomial naïve bayes (mnb), super vector machine (svm), and random forest (rf), also a neural network multi-layer perceptron (mlp). by the comparison of their scores, we chose svm as our final algorithm, and it achieved 0.350 points or 35% for its f1 macro, which is 0.132 points or 13.2% than the baseline model.