CUET - NLP @ D ravidian L ang T ech- ACL 2022: Exploiting Textual Features to Classify Sentiment of Multimodal Movie Reviews (2022)

Abstract: With the proliferation of internet usage, a massive growth of consumer-generated content on social media has been witnessed in recent years that provide people’s opinions on diverse issues. Through social media, users can convey their emotions and thoughts in distinctive forms such as text, image, audio, video, and emoji, which leads to the advancement of the multimodality of the content users on social networking sites. This paper presents a technique for classifying multimodal sentiment using the text modality into five categories: highly positive, positive, neutral, negative, and highly negative categories. A shared task was organized to develop models that can identify the sentiments expressed by the videos of movie reviewers in both Malayalam and Tamil languages. This work applied several machine learning techniques (LR, DT, MNB, SVM) and deep learning (BiLSTM, CNN+BiLSTM) to accomplish the task. Results demonstrate that the proposed model with the decision tree (DT) outperformed the other methods and won the competition by acquiring the highest macro f_1 -score of 0.24.

## introduction over the years, sentiment analysis has grown to an influential research domain with widespread commercial applications in the enterprise. to date, a significant number of applications have already been used for classifying or analyzing textual sentiment, including customer feedback @xcite , recommendation systems @xcite , medicine analysis (rajput, 2019), marketing, financial strategies @xcite and so on @xcite @xcite . usually, people express their opinions, emotions, and ideas through text over the internet @xcite . however, the mode of communication is gradually shifting from unimodal to multimodal due to the rapid growth of all sorts of media content, including massive collections of videos (e.g., youtube, facebook, tiktok), audio clips, and images @xcite . classification of sentiment utilizing multiple modalities is becoming increasingly important and an exciting research topic. multimodal sentiment analysis can analyze public opinions based on the speaker's language, facial gestures and acoustic behaviours, and voice's intensity @xcite . in recent years, a few studies have been performed on unimodal sentiment analysis concerning low-resource languages (e.g., tamil, malayalam, bengali) @xcite @xcite @xcite . the most challenging task in categorizing movie reviews is the interpretation of the words as most of the time, words are anticipated to the elements of a movie, not the opinion of the reviewer @xcite . moreover, most language processing works mainly concentrate on high-resource languages like english, arabic, and other european languages, where standard datasets are not available for low-resource languages. this work addresses the multimodal sentiment analysis from movie reviews in tamil. tamil is a member of the southern branch of the dravidian languages, a group of about 26 languages indigenous to the indian subcontinent. it is also classed as a member of the tamil language family, which contains the languages of around 35 ethno-linguistic groups, including the irula and yerukula languages @xcite @xcite @xcite . tamil is an official language of tamil nadu, sri lanka, singapore, and the union territory of puducherry in india. significant minority speak tamil in the four other south indian states of kerala, karnataka, andhra pradesh, and telangana, as well as the union territory of the andaman and nicobar islands. it is also spoken by the tamil diaspora, which may be found in malaysia, myanmar, south africa, the united kingdom, the united states, canada, australia, and mauritius @xcite @xcite . tamil is also the native language of sri lankan moors. the term "old tamil" refers to the time of the tamil language from the 10th century bc to the 8th century ad. the earliest old tamil documents are small inscriptions in adichanallur dating from 905 bc to 696 bc. these inscriptions are written in tamil-brahmi, a variation of the brahmi script. the tolkppiyam, an early work on tamil grammar and poetics, is the first extended book in old tamil, with layers dating back to the late 6th century bc @xcite . ## related work with the rapid popularization of social media, people's eagerness to express their views or opinions on these mediums increases sharply. however, sentiment analysis in low-resource languages is still rudimentary due to the scarcity of standard corpora and limited language processing tools. few ml-based methods such as support vector machine (svm), logistic regression (lr), naive bayes (nb) have been used to analyze the textual sentiment in bengali @xcite . thavareesan and mahesan (2019) performed sentiment analysis on five different tamil text corpora using various ml techniques with bow and tf-idf features, which obtained the highest accuracy of 79% with extreme gradient boosting with fast-text. singla et al. (2017) experimented with nb, dt, and svm with the 10-fold cross-validation achieving 81.75% accuracy with svm. phani et al. @xcite used the sail corpus to assess the sentiment of tweets. they achieved the best perfor-mance in tamil with nb and hindi, bengali with lr. the performance of these models is not very impressive as they were unable to capture semantic and contextual information in the text. the major obstacles are the inherent ambiguity of the language, the computational complexity of exploring large amounts of content, resource-poor language problems, and the contextual understanding of natural language @xcite . different dl models were applied to malayalam tweets to classify them into positive and negative where gated recurrent unit (gru) achieved the highest accuracy @xcite . several approaches, including lexicon, supervised ml, hybrid, were experimented on tamil texts @xcite @xcite . @xcite proposed a joint structure that combines cnn and rnn layers along with glove embeddings for capturing long-term dependencies of twitter data. in another similar work, the sentiment lexicon is used to enhance the sentiment features, and then cnn-gru networks are combined to analyze the sentiment of product reviews @xcite . pranesh and shekhar (2020) presented 'memesem' where vgg19 is used for visual and bert for textual modality to analyze the sentiment of memes. memesem outperformed all the unimodal and multimodal baseline by 10.69% and 3.41% respectively. recently, the cnn + bi-lstm model @xcite has been employed to classify the sentiment of twitter data and gained the highest accuracy of 90.2% for binary classification (positive and negative). ## dataset description the dataset we have used for this task is provided by the shared task organizers foot_0 . it is a collection of videos, audios, and text accumulated from youtube and manually annotated. the dataset is divided into three sets (i.e., train, validation, and test) and annotated into five classes: highly positive, positive, neutral, negative, and highly negative. the dataset consists of a total of 134 videos, out of which 70 are malayalam videos and the remaining 64 are tamil videos @xcite . the length of the videos is between 1 minute to 3 minutes. this work employed textual features to address the assigned task. participants have the freedom to utilize unimodal data (i.e., video, audio, or text) or multimodal (i.e., a combination of any two or three modalities) features to perform the classification task. each text is provided in the *.docx file format. therefore, we extracted all the texts from documents before starting the experimentation and evaluation (section 4 provides a detailed description). ## methodology the objective of the task is to classify the underlying sentiments from movie reviews using video, audio, and text modalities. however, we have used only textual data to attain this goal. initially, texts were taken from the *.docx files and preprocessed for further use. subsequently, feature extraction techniques are applied to get the features. finally, the extracted features are utilized to develop ml and dl models to perform the classification task. figure 1 illustrates an abstract view of the sentiment analysis model. ## results and analysis the results demonstrate that the dt model outperformed the other ml and dl models. the dt models showed 86.6% of increased performance compared to the ml models and improved by 140% than the best dl model (cnn+bilstm). the other ml models, such as lr, svm, and mnb, were classified all test instances as the positive class. the bilstm with word2vec features predicts maximum reviews as neutral ones having a macro f 1 score of 0.07. however, after using the pre-trained word embedding with combined cnn and bilstm model, the macro f 1 -score has grown to 0.10. thus, the model shows an increase in performance using pre-trained word embedding. however, it cannot beat the dt model developed based on the tf-idf features. ## conclusion this paper investigated several ml and dl techniques to address the sentiment analysis task on a multimodal dataset in tamil. although the provided dataset included text, audio, and video modalities, this work considered the text modality. results indicate that the dt model outperformed the other ml and dl models obtaining the maximum macro f 1 -score (0.24). surprisingly, dl models showed poor performance compared to their ml counterparts. since the dataset is too small and crooked, data oversampling techniques or any open source large corpora can be used to create synthetic data to improve performance. the scarcity of training samples might cause lower scores. moreover, excluding the video and audio features might also hurt the model's performance. we aim to incorporate multimodal features (video, audio, text) and address the task with the recent transformerbased models (i.e., indicbert, mbert, xml-r, muril) in the future.