SADT ech@ D ravidian L ang T ech: Multimodal Sentiment Analysis of T amil and M alayalam (2023)

Abstract: We present several models for sentiment analysis of multimodal movie reviews in Tamil and Malayalam into 5 separate classes: highly negative, negative, neutral, positive, and highly positive, based on the shared task, “Multimodal Abusive Language Detection and Sentiment Analysis” at RANLP-2023. We use transformer language models to build text and audio embeddings and then compare the performance of multiple classifier models trained on these embeddings: a Multinomial Naive Bayes baseline, a Logistic Regression, a Random Forest, and an SVM. To account for class imbalance, we use both naive resampling and SMOTE. We found that without resampling, the baseline models have the same performance as a naive Majority Class Classifier. However, with resampling, logistic regression and random forest both demonstrate gains over the baseline.

## introduction sentiment analysis is the application of natural language processing techniques to identify, quantify, and examine the subjective attitudes and affective content of language. it has a rich history and many methods have been used in the past @xcite . while sentiment analysis tasks most frequently occur in the text domain, analysis of multimodal content has become an increasingly important task in recent decades as such content has become much more common. users of popular social media websites, for instance, have long grown accustomed to creating and interacting with content which has either a textual, auditory, or visual form, or some combination of these three modalities, e.g., a youtube video with subtitles features all three at once. sentiment analysis models trained for such contexts must reflect features of all modalities concerned. practically speaking, this task can be constructed in many ways: as a binary classification task (i.e. categorizing language into either positive or negative sentiment), an ordinal regression problem, or a multi-class classification problem. in our case, the task was the latter. we are classifying multimodal @xcite data in tamil and malayalam into 5 separate sentiment classes: highly negative, negative, neutral, positive, and highly positive @xcite . ## related work previous work on sentiment analysis on dravidian languages was done in last year's shared task where only one team submitted their sentiment analysis project @xcite . also, previous work on sentiment analysis on dravidian languages has used code-switched data (english and either tamil or malayalam) @xcite . our data is not code-switched, and contains only one language. like many tasks in natural language processing, different neural network architectures have been used to perform sentiment analysis @xcite . text vectorization synthesizing text documents into feature vectors for text classification is a difficult problem; common methods @xcite for doing so include bagof-words, n-gram, tf-idf, word2vec @xcite , glove @xcite , and sentence2vec/doc2vec @xcite . previous work has used features present in the orthography of low resource languages, especially dravidan languages @xcite . using pre-trained elmo @xcite or bert @xcite word embeddings has become increasingly popular, since they better encode semantic information @xcite . @xcite uses the last three hidden layers of the pretrained multilingual model xlm-roberta @xcite in conjunction with a convolutional neural network (cnn) to encode tamil-english, as well as malayalam-english code-mixed data into feature vectors. audio vectorization speech technologies have progressed a lot in the recent years, especially in multilingual and low-resource areas. the introduction of pseudo-labeling has improved the efficiency of semi-supervised deep neural network learning @xcite . multitask learning approaches in conjunction with deep neural networks has enabled improvements in speech recognition for low-resource languages @xcite . wav2vec2 utilizes transformer-based unsupervised pre-training @xcite which works well with little training data. it also does this pre-training via masked reconstruction loss which has benefits similar to data augmentation methods @xcite . m-ctc-t, xls-r (crosslingual speech representations), and unispeech, on the other hand, make use of supervised/selfsupervised pre-training methods which often allow for better accuracy @xcite . ## dataset description the shared task organizers have provided 54 training samples for tamil and 60 training samples for malayalam, as well as 10 test samples for both @xcite . each sample consists of an audio file of speech and a (sometimes partial) transcript of it. (they have also provided video files which we do not use.) every sample is a movie review collected from youtube and labeled by human annotators. they have also provided a train and dev split over the data set, which is further subdivided by language (tamil and malayalam). the 54 tamil samples are split 44/10 train/dev while the malayalam samples are split 50/10. however, instead of using the official split, we combine train and dev data and use k-fold validation. ## results to evaluate our classifiers, we pooled the official train and dev splits into one large training set, and then ran k-fold cross-validation. this allowed us to train our models on more training data, as there was not a lot of training data provided. for k-fold crossvalidation, we used 4 folds. to be able to compare the different models that we created, we used a deterministic algorithm to shuffle the data. this ensures that the same four train and dev splits were used for all the models, making the performance of our different models comparable. first, for reference, in we have selected the best-performing model in terms of the pooled dev macro-f 1 score, training just on text data, as well as training on text in conjunction with audio data. for this model, we report the validation pooled f 1 scores and accuracy aver-aged across five seeds. 7 for the test data, we report the same metrics averaged across the same seeds in we also report the dev and test results of the best model for any specific seed in terms of f 1 score on the test data. this "best model" was trained with only the text modality (across the board, the text audio models did worse due to greater feature sparsity). note that this single seed happened to perform especially well, but is not representative of overall model performance. ## conclusion we performed multimodal sentiment analysis on text and audio data from the dravidian languages tamil and malayalam. to perform sentiment analysis, we built an end-to-end system, trying different vectorization methods, resampling techniques, and classifiers. we evaluated the system's performance using only text data, as well as jointly using text and audio data. we created baseline systems using tf-idf vectors and a multinomial naive bayes classifier with no resampling and found that these baseline models performed no better than a majority class classifier. as discussed elsewhere in this paper, we found that significant challenges stemming from the small size of our data set and its highly imbalanced distribution of classes proved largely insurmountable