A Novel Approach towards Cross Lingual Sentiment Analysis using Transliteration and Character Embedding (2022)

Abstract: Sentiment analysis with deep learning in resource-constrained languages is a challenging task. In this paper, we introduce a novel approach for sentiment analysis in resource-constrained scenarios using character embedding and cross-lingual sentiment analysis with transliteration. We use this method to introduce the novel task of inducing sentiment polarity of words and sentences and aspect term sentiment analysis in the no-resource scenario. We formulate this task by taking a metalingual approach whereby we transliterate data from closely related languages and transform it into a meta language. We also demonstrated the efficacy of using character-level embedding for sentence representation. We experimented with 4 Indian languages – Bengali, Hindi, Tamil, and Telugu, and obtained encouraging results. We also presented new state-of-the-art results on the Hindi sentiment analysis dataset leveraging our metalingual character embeddings.

## introduction sentiment analysis is a widely explored topic in the field of natural language processing, which focuses on classifying text into 3 sentiment classes: positive, negative, and neutral @xcite . for any text classification task, supervised approaches require an extensive and domain specific corpus in the corresponding language. however, sentiment annotated data, which is the primary resource for sentiment analysis, is limited in many languages. transfer learning can be used to learn sentence representations by pretraining on a large corpus and finetuning to a particular downstream task. but transfer learning often fails in adapting knowledge from one domain to another, and sufficient training samples across these domains may not be available for efficient finetuning. another method to deal with such situations is to create cross lingual tools between a resource rich and resource poor language. this method either uses machine translation between the language or bilingual dictionaries to overcome the language gap. however, it is an extremely challenging task to create an accurate machine translation system between two languages or create a bilingual dictionary pair in a resource constrained scenario @xcite . moreover, there are some studies @xcite @xcite that show that there is a significant loss of pragmatics in the translations produced by the stateof-the-art machine translation (mt) systems, which could adversely affect the performance of these downstream nlp applications that use unedited raw mt output. very little work has been done to use cross-lingual sentiment analysis without translation involved in any step, and even if they do, they require the presence of large sentiment annotated data in at least one language. another major problem is the unavailability of significantly large corpus to train the language-specific word embedding models in resource scarce languages. as a result, training word embeddings on those languages and reusing the pretrained embeddings is not feasible. literature suggests that using character embedding or subword embedding can be an efficient alternative, as the number of unique characters and alphabets for any language are well deterministic @xcite @xcite . the main advantage of using character embedding is that when we encounter a word which is not in the domain of the available corpus (i.e., an out-of-vocabulary word, or oov), we can still have a latent representation of the word for any nlp task. previously, dos santos and gatti (2014) used character embedding for sentiment analysis of short texts, however, very few work have been done on word level classification. there exist many such languages where the coverage of the corpus is too poor to effectively train the embedding weights for each word. even transfer learning with pretrained large models like m-bert @xcite , roberta @xcite , electra @xcite might encounter many unknown ([unk]) tokens since the vocabulary set of the tokenizers may not have many words in the testset. moreover, a dataset for the downstream task may not exist in the language for effective finetuning. therefore, we need methods that can help create sentiment lexicons effectively and perform sentiment analysis without using any training dataset of the target language. in this paper, we introduce a novel method to learn efficient latent representation by projecting data in multiple languages to a shared metadata representation. we propose to utilize the wx-notations as the transliteration approach for the projection. this enables us to leverage the collective training corpus to learn a deep learning system with higher confidence. we evaluate our approach on 7 sentiment analysis datasets (4 word-level, 2 sentence-level, and 1 aspect-level) across four indian languages -hindi, telugu, bengali, and tamil. moreover, we experiment in two setups -mono-lingual and cross-lingual. finally, we compare our model against various baselines and observe the performance to be better for the majority of the cases. convolutional neural network (cnn) provides the best result on aspect analysis and sentiment classification of sentences, while different models prove to be useful on different datasets for word-level sentiment analysis. contribution: the paper makes the following contributions. the main difficulty is cross alignments, due to word order/ syntactic differences in languages. @xcite presented an alternative approach to clsa using wordnet senses as features for supervised sentiment classification. but for many resource-constrained languages, wordnet does not exist. hence in the proposed methodology, we propose using transliteration as alignments are monotonic, ie, they do not cross each other. previously an attempt was made to transliteration for sentiment analysis. in oder to automatically classify sentiments of arabizi messages. @xcite transliterated their corpus into arabic and used classification models like support vector machines, naive bayes classifier and decision trees for sentiment analysis. however for our experiments we will use a transliteration scheme to obtain a common metalanguage that can be used cross-lingual embedding training, and classify words/ sentences using simple classification models without using any data of the resource-scarce language. ## proposed method training deep neural network requires huge amount of data, which is not feasible for low resource languages. to avoid this, we follow the principles of cross-lingual learning, where model trained on one language is reused on another language. this ensures that no data from the target language is used during training, and hence can be adapted in a low resource scenario. our approach is as follows: similar sentimentannotated datasets from other languages are leveraged to create a pool of data instead of a single source language data, which is subsequently transliterated into a common language (metalanguage) using a transliteration scheme. we use this combined metadata as our training data, which acts as a relatively bigger dataset that can be used for efficient training of deep learning-based text classification models. additionally, it reduces the dependency on a single language dataset. the target language dataset is also transliterated using the same transliteration scheme and this transliterated data is used only as the testing data and plays no part in training. we train our classification models on the transliterated data from other languages, and use the trained classification models to classify words/sentences of the transliterated test set of the target language. thus we build our sentiment classification model for a language without using any training data of that language or involving any translation. as shown in the later sections, this method yields in comparable results given by state-of-the-art models like bert in text classification in scarce-resource scenarios where very little or no training data is available. ## dataset we used the sentiwordnet foot_0 for indian languages @xcite @xcite for our experiments. this dataset contains sentiment polarity of words in four indian languages -bengali (bn), hindi (hn), tamil (ta) and telugu (te). for each language, the dataset contains four different files lc_pos foot_1 , lc_neg, and lc_neu and lc_ambi, listing the set of positive, negative, neutral, and ambiguous words, respectively. lc_ambi was not used in our experiments on word-level classification due to the negligible presence of ambiguous words compared to the rest of the classes. each word in this dataset is marked with pos category foot_2 . ## experimental setup for the classification task, we used various models -rnn, cnn, lstm, gru, self attention mechanism @xcite and a combination thereof. embedding length in each model is set to the number of unique characters. for the metalanguage experiments, since the wx format converts characters to uppercase and lowercase english characters, the embedding size is set to 52. each convolution layer has embedding @xmath0, number of input @xmath1, kernel @xmath2, and number of output @xmath3. for lstm and gru, the embedding dimension is equal to the input size which in turn is equal to the embedding length, number of hidden @xmath4, and a dropout rate of 0.01. each model ended with a dense layer with a softmax activation layer for classification. many of these layers were combined one on top of the other for sentiment classification of an individual dataset. it was tested with the following learning rates: 0.001,0.005,0.05,0.5 and the best results have been reported. since the data suffers from class imbalance, proportionate samples were taken from each class. for stacked layers, the hyperparameters were tweaked accordingly to adjust input and output dimensions. for the metalanguage experiments, adam optimizer was used with a learning rate of 0.001 and sparse categorical cross-entropy as the loss function. ## experiments and results we conducted experiments in 2 directions -(i) inducing sentiment polarity of words, and (ii) sentence-level sentiment analysis and aspect analysis. ## analysis of results results suggest that our proposed method performs reasonably well for different tasks and languages. the traditional cnn architectures captured structural features very well, which is evident from the fact that sentiment embedded vectors when incorporated in training produced state-of-the-art results on most of the datasets. we realize that using cnn will give us two main advantages: (i) learn hidden semantics from a metalanguage, and (ii) handling limited coverage of lexical resources. even when training data is not used, we achieved promising results which proves the effectiveness of the method in a resource-constrained scenario. an interesting observation was found in the review ah dataset, where the model gave different results based on the convolution window size. for example, the models performed better when the sentence comprised of the aspect term and four words from either side of the aspect term and used for sentiment classification than the situation when 2 words from either side were considered for training. this can be attributed to the fact that the information from the distant part of the sentences is sometimes attributed to the overall sentiment polarity of the aspect terms. however, the situation reversed when no training data was used, where sentences with 2 words from either side provided better classification results in comparison to sentences with 4 words from either side of aspect terms. this is because we used word-level classification models to classify these sentences, and hence these models captured the local features of the aspect terms more efficiently and gave better accuracy. ## conclusions in this paper, we introduced the novel task of inducing the sentiment polarity of words using character embedding-based deep learning models. we extended the task to inducing the sentiment polarity of words in a new language having no training data. we carried out experiments with 4 indian languages and obtained encouraging results. the cross-lingual approach proved to be an effective method in a resourceconstrained scenario. the same idea was followed to perform sentiment analysis and aspect analysis in hindi without using any training data in hindi. while using training data, our method outperformed the previous state-of-theart in sentiment analysis and aspect analysis in hindi.