I ndi S entiment140: Sentiment Analysis Dataset for I ndian Languages with Emphasis on Low-Resource Languages using Machine Translation (2024)

Abstract: Sentiment analysis, a fundamental aspect of Natural Language Processing (NLP), involves the classification of emotions, opinions, and attitudes in text data. In the context of India, with its vast linguistic diversity and low-resource languages, the challenge is to support sentiment analysis in numerous Indian languages. This study explores the use of machine translation to bridge this gap. The investigation examines the feasibility of machine translation for creating sentiment analysis datasets in 22 Indian languages. Google Translate, with its extensive language support, is employed for this purpose in translating the Sentiment140 dataset. The study aims to provide insights into the practicality of using machine translation in the context of India’s linguistic diversity for sentiment analysis datasets. Our findings indicate that a dataset generated using Google Translate has the potential to serve as a foundational framework for tackling the low-resource challenges commonly encountered in sentiment analysis for Indian languages.

## introduction within the domain of natural language processing (nlp) research, a notable challenge is the scarce or limited availability of resources, particularly in the context of indian languages. this scarcity extends to the field of sentiment analysis as well. as nlp researchers and practitioners continue to explore innovative solutions to bridge the gap between resource-rich languages like english and resource-poor languages in the indian subcontinent, this study embarks on a critical inquiry into one such challenge: the support for low-resource languages within the indian context, specifically in the realm of sentiment analysis. a fundamental realization emerges -fulfilling the resource requirements for indian languages necessitates the creation of these resources. this entails manually creating linguistic datasets, including labeled text, grammatical structures, and socio-cultural and language specific sentiment indicators. while this approach is undoubtedly effective, it raises questions about its feasibility, especially when dealing with the extensive linguistic and cultural diversity found in india. given the multitude of languages spoken in india, manual resource creation for each language becomes a monumental and perhaps unfeasible task. considering the experience of the nlp researcher in the context of resource constrain, is it possible to supplement the resource gap by bootstrapping with partially correct english to indian languages machine translation tools such as google translate? various machine translation tools and apis, including google translate, deepl, microsoft translator, and openai's machine translation, are readily available. according to the documentation available for google translate @xcite , it supports over 1000+ languages, among which 22 are indian languages. motivated by this, and with the objective of assessing the feasibility and efficacy of machine translation in the context of india's linguistic diversity, particularly in the development of sentiment analysis datasets for indian languages, this paper considers sentiment140 @xcite which is considered to be one of the largest corpora for sentiment analysis task in general domain curated from twitter (currently x). our selection of the sentiment140 dataset is driven by its reputation as a significant dataset for sentiment analysis in noisy environments. its substantial size, balanced distribution, and the inherent challenges of tweets make it an ideal choice for evaluating sentiment analysis models in real-world scenarios. by incorporating this dataset into our study, we address resource constraints in indian languages while benefiting from a well-established benchmark for assessing sentiment analysis models in noisy environments. leveraging this widely used sentiment140 dataset, we generate parallel corpora, known as indisentiment140, by translating it into 22 indian languages supported by google translate. we utilize these generated parallel corpora to investigate their suitability for sentiment analysis by training two foundational neural models: the multi-layer perceptron (mlp) and the long short-term memory (lstm) models. despite the availability of more advanced neural and transformer-based models, our choice of these simpler models is driven by the aim to assess whether the translated dataset preserves both the sentiment characteristics of the original dataset and maintains the underlying sequential structure rather than the model aspect. if the dataset is suitable for these two models, it is likely to be suitable for advanced models. considering the socio-cultural and languagespecific nuances influencing sentiment, we not only leverage this translated corpus but also utilize a human-translated sentiment dataset, indic-sentiment @xcite , to evaluate the sentiment analysis model trained on the translated corpus, indisentiment140. our findings indicate that these translated datasets not only retain the sentiment characteristics of the original dataset but also maintain the sequential structure within the translated text across specific languages. this suggests that these translated datasets have the potential to serve as a foundational framework for addressing the low-resource issue that prevails in indian languages, offering a promising avenue for further nlp research and development in this context. the indisentiment140 dataset is available on https://www.iitg.ac.in/cseweb/ osint/resourcess.php . ## related work sentiment analysis in the context of indian languages presents a unique challenge due to the limited availability of resources. addressing this lowresource challenge, several studies have introduced sentiment datasets for indian languages. most of the existing sentiment analysis datasets are primarily designed for resource-rich languages like hindi, bangla, tamil, telugu, malayalam, among a few others. these datasets are typically small in size and rely on human annotations. here are some of the datasets specifically crafted for sentiment analysis in the context of indian languages. a manually annotated telugu dataset, sentiraama, for sentiment analysis in telugu, was presented by @xcite . this dataset integrates multiple domain sources to improve sentiment prediction, comprising 1k documents from various domains, including song lyrics, movie reviews, product reviews, and book reviews. for sentiment analysis in bengali. these data were gathered from diverse sources like bengali news articles, tv channel news dumps, books, blogs, sports portals, and manually annotated. @xcite introduced another dataset of 15k instances for analyzing sentiment in bangla text. this dataset was created by annotating comments on news and videos from different domains. a dataset, consisting of 15k manually annotated comments for sentiment analysis in sinhala, is presented by @xcite . more recently, indicsentiment @xcite has been introduced for sentiment analysis in 13 indian languages, including assamese, bengali, bodo, gujarati, hindi, kannada, malayalam, marathi, odia, punjabi, tamil, telugu, and urdu. this dataset comprises 1k reviews for each language, which were manually translated and annotated from english reviews collected from various e-commerce websites. ## proposed dataset we have considered sentiment140 @xcite , a publicly available large dataset for sentiment analysis as source dataset. this dataset contains 1.6 million tweets in english extracted using the twitter api. each of these tweets has been labeled either with 0 or 4 which represents negative and positive sentiment respectively. as this dataset was collected from twitter, a microblogging-based social media platform, the text predominantly employs informal and casual language, incorporating slang, fancy acronyms, emojis, urls, as well as hashtags and keywords. before translating this dataset to indian languages urls, emojis, and special symbols like # and @ have been removed. we have considered 22 indian languages i.e. assamese (as), bengali (bn), bhojpuri (bho), dogri(doi), gujarati (gu), hindi (hi), kannada (kn), konkani (gom), maithili (mai), malayalam (ml), marathi(mr), meiteilon (manipuri) (mni), mizo (lus), nepali (ne), odia (or), punjabi(pa), sanskrit (sa), sindhi (sd), sinhala (si), tamil (ta), telugu (te), and urdu (ur) for our study. we have created the dataset for each language listed above after translating the senti-ment140 dataset to corresponding language. the same sentiment labels are kept back after translating it to different indian languages. there are many machine translators (and or apis) available like google translate, deepl, microsoft translator, openai's machine translation, and many more. we have chosen google translate as it supports 1000+ languages @xcite out of which 22 are above mentioned indian languages. to understand the level of correctness of the translated text, we also evaluate the performance of google translate taking different publicly available parallel corpora in indian languages such as facebook's flores-200 dataset (costa-jussà et al., 2022) that contains parallel corpora of 24 indian languages out of which 19 languages are found to be relevant for our study, pmindia dataset (haddow and kirefu, 2020) which we have found 12 languages are relevant for our study and indicsentiment dataset @xcite . we have chosen 1000 random samples from each of these parallel corpora for evaluating the performance of google translate. we have taken the samples in english as the source sample and translated them to different indian languages available in the particular parallel corpus. after that the bleu (bilingual evaluation understudy) score has been calculated taking the 4-gram overlap and the same is tabulated in ## model training on translated data in order to evaluate practicality of using the translated dataset for sentiment analysis, we implement two basic machine-learning models for each language -a simple classifier employing multi-layer perceptron (mlp) and a sequential model-based classifier employing lstm ( long short-term memory). we have considered the simplest mod-els, as we are interested in investigating usability of the datasets, rather than the model design. the mlp model is composed of an input layer with dimensions matching the word embeddings, a single hidden layer with 256 nodes, followed by a dropout layer, and concluding with an output layer having 2 nodes for sentiment classification. similarly, the lstm-based model is tailored to process sequences, each with a length of 128 tokens. the architecture includes two stacked lstm layers, each with a hidden size of 64. subsequently, the model's output is concluded with an output layer having 2 nodes for sentiment classification. the choice of the lstm model serves the purpose of exploring whether the translated datasets effectively retain sequential information. to obtain text embeddings, we have used two distinct pre-trained word embedding models: (i) indicbert v2 @xcite , which was trained using the bert architecture, and (ii) indisocialft @xcite )-a multilingual fasttext @xcite based embeddings for indian languges. to obtain the embedding for a specific text(tweet), we have calculated the mean of all the embeddings generated by the pretrained model for each token. the text embedding generated by the indicbert v2 is of size 768 and that of indisocialft is 300. these text embeddings are directly supplied to the input layer of our mlp model. for each language in our dataset, we have trained separate classifiers, taking into account the embeddings from both indicbert v2 and indisocialft individually (for lstm model only indicbert v2 is used). we split each language dataset into an 80% portion for model training and a 20% portion for model testing. the statistics for the train and test splits for each language dataset are summarized in ## result and evalution we have evaluated the trained model considering two datasets -(i) the test split of the translated dataset mentioned in ## conclusion and future work in this paper, we have investigated the potential of machine translation as a solution to overcome the low-resource challenge faced by diverse indian languages, specifically in the context of sentiment analysis. our approach involved translating the sentiment-140 dataset into 22 indian languages using google translate and conducting comprehensive experiments to assess the usability of these translated datasets. our findings reveal that these translated datasets not only retain the sentiment characteristics of the original dataset but also maintain the sequential structure within the translated text across specific languages. this suggests that these translated datasets could serve as a foundational framework to address the low-resource issue prevalent in indian languages. as for future work, we plan to further improve our dataset by incorporating more machine translation tools and apis available for indian languages. we also plan to add more languages to this dataset.