E nglish- M alay Word Embeddings Alignment for Cross-lingual Emotion Classification with Hierarchical Attention Network (2022)

Abstract: The main challenge in English-Malay cross-lingual emotion classification is that there are no Malay training emotion corpora. Given that machine translation could fall short in contextually complex tweets, we only limited machine translation to the word level. In this paper, we bridge the language gap between English and Malay through cross-lingual word embeddings constructed using singular value decomposition. We pre-trained our hierarchical attention model using English tweets and fine-tuned it using a set of gold standard Malay tweets. Our model uses significantly less computational resources compared to the language models. Experimental results show that the performance of our model is better than mBERT in zero-shot learning by 2.4% and Malay BERT by 0.8% when a limited number of Malay tweets is available. In exchange for 6 – 7 times less in computational time, our model only lags behind mBERT and XLM-RoBERTa by a margin of 0.9 – 4.3 % in few-shot learning. Also, the word-level attention could be transferred to the Malay tweets accurately using the cross-lingual word embeddings.

## introduction sentiment analysis and opinion mining are used interchangeably to represent the task of classifying the sentiment polarity of opinionated text @xcite . on a coarse-grained level, the task is often a binary classification problem (positive or negative) @xcite . the neutral sentiment in addition to positive and negative could also be taken into consideration, as demonstrated in @xcite . beyond sentiment polarity, the text could be analysed at a finer-grained level to detect emotions, which is also known as emotion analysis. this could help narrow down the broad concepts of sentiment to better capture a person's emotional state @xcite . for instance, while anger and fear express negative sentiments, each semantically represents a different emotional state. anger is perceived as the possible driving force of collective action, whereas fear is viewed as an action inhibitor @xcite . regardless of the level of sentiment analysis, it had only been the privilege of languages with rich resources like english. most existing studies focusing on resource-rich languages have produced extensively annotated corpora and computational tools exclusive to these languages. however, the advent of cross-lingual sentiment analysis opens up the possibility of performing sentiment analysis on resource-poor languages by leveraging the resources from richer counterparts. with crosslingual sentiment analysis, resource-poor languages can be endowed with comparable computational ability in identifying sentiments. among the seven thousand languages documented across the world, only approximately 30 languages have been equipped with linguistically annotated resources @xcite . while malaysian malay is not the most spoken language globally, it is a language that is dominantly spoken in malaysia. nonetheless, malay still lacks linguistic resources for sentiment analysis, which poses a challenge in automatically identifying sentiments expressed in malay texts on a large scale, especially on social media platforms where almost everyone shares their personal and affective experiences. there is a need for sentiment analysis in malay to more accurately assess individual or public emotions expressed in the local language, particularly during natural disasters, pandemics or political instability in malaysia. ## related work the main challenge in cross-lingual sentiment analysis is how to bridge the language gap between the source (rich-resourced) and target (lowresourced) languages. the approaches adopted by prior studies had the element of machine translation of varying degrees. one approach uses direct translation. wan (2012) translated chinese reviews into english and classified the translated english reviews using a rule-based classifier or support vector machine. @xcite translated arabic social media posts to english using their in-house machine translation system and manual translation. the arabic-to-english translated posts were then classified automatically and manually. another approach tries to project annotations from the source language to the target language. @xcite annotated the english side of the english-romanian parallel corpus automatically using a rule-based classifier and a naïve bayes classifier from opinionfinder before projecting the annotations to the romanian-side corpus for training. @xcite translated english sentences into french, german and spanish, and the english-side annotations were then projected to their corresponding translated sentences to train a classifier respectively. the third approach uses joint learning. banea et al. (2010) translated the english corpus into five different languages. they then concatenated monolingual unigram from different languages as the features to train the model. @xcite created a synthetic multilingual training corpus by combining english movie reviews and corresponding translated malay movie reviews. they then trained multilingual word embeddings using a blended approach that made differentiating original english and malay words impossible. @xcite first translated english documents into the target language to obtain paired training documents in the training phase. they embedded every sentence in the documents with trained monolingual sentence representations and concatenated the document representations in both languages to train a classifier subsequently. the fourth approach uses alignment. abdalla and hirst (2017) adopted a simple vector space transformation for which the matrix was obtained using a closed-form solution to linearly map the spanish/chinese monolingual word embeddings to the english vector space. these word embeddings were later used to predict the affective norms for english words (anew) values of the word and form the sentence arrays of the reviews for sentiment classification. ahmad et al. (2020) adopted a similar approach but constrained the transformation matrix to be orthogonal. they then pre-trained a model using the english tweets and fine-tuned it using hindi reviews. @xcite adopted muse @xcite to construct english-arabic and english-spanish cross-lingual word embeddings. the transformation matrix was first learnt using adversarial training and subsequently refined using a synthetic parallel lexicon built from the shared embeddings space. @xcite constructed crosslingual word embeddings between english and each of the 17 low-resourced languages using vecmap @xcite . the initial bilingual word pairs were constructed without supervision by exploiting similarity matrices of each language. the transformation matrix was then iteratively refined using self-learning until the convergence criterion was met. @xcite exploited the structural similarity of wordnet bahasa and english wordnet to map synonyms in malay to the corresponding english counterparts using synset value and pos value. to classify documents, they aggregated the polarity scores of every word in the documents, but the classification was not satisfactory. @xcite also used a similar approach by considering code-switched tweets containing both english and malay words and classified the tweets by the sign of the total sentiment score. the fifth approach uses co-training or its variants. wan (2009) adopted a co-training approach by translating labelled english and unlabeled chinese reviews into the other language. two classifiers were trained on each view of the labelled reviews, and the classifiers were retrained iteratively by augmenting their training corpus with confidently predicted unlabeled reviews. hajmohammadi (2014) combined self-training and active learning in their study. they first trained a base classifier on english reviews to predict the unlabeled chinese/french reviews translated to english. the unlabeled reviews predicted with high confidence and human-annotated reviews were then selected to retrain the classifier. two studies on the malay language relied on lexicon-based approaches and reported less than promising results, while the third one relying on bilingual word embeddings was ambiguous. there have also not been any studies on finer-grained sentiment analysis in the malay language. this study aims to improve cross-lingual sentiment analysis in the malay language using a better but also less computational expensive approach at a finer-grained level on informal corpora, and our approach is similar to that by @xcite . malay evaluation tweets are a random subset of the tweets available on malaya documentation (husein, 2018), previously labelled using a rulebased classifier. we hired and trained three native speakers to validate the emotions using majority voting. the malaya documentation corpus contains both malaysian malay and indonesian malay tweets. therefore, we adopted a hybrid approach (google's language detector followed by human detection) to remove the indonesian malay tweets from our corpus. we performed similar pre-processing steps as in the english training tweets. contractions in malay were first normalised and then spell-checked according to the context. for example, msg2 foot_0 were expanded to masing-masing (individually or respectively), and x was expanded to tidak (no). we then performed stratified sampling to select 1000 malay evaluation tweets as the test set (malay test set) and the remaining 843 tweets (after removing tweets shorter than 3 words) as the fine-tuning set (malay fine-tuning set) to finetune our model. ## conclusion and future work we evaluated the quality of the existing set of malay-english bilingual word pairs as part of the experiments in this paper and discovered that its quality could be further improved. apart from this, we demonstrated that malay words could benefit from their semantically and sentimentally similar english counterparts through refined cross-lingual word embeddings that were mapped using our bilingual lexicon after fine-tuning. most importantly, our model is better than monolingual malay bert and at par with mbert but utilises significantly less computational power. even though xlm-r-base shows slightly better performance than our model by 4.3% in few-shot learning, our model is still competitive as the amount of finetuning and computational time can be reduced by 6 -7 times. this provides us with a more cost-effective alternative to predict emotions in malay tweets on a large scale more efficiently and possibly generalise to other languages with limited training corpora. unlike english, malay remains a low-resource language with no standard malay emotion corpus. thus, we could not evaluate our model on other test sets to obtain a more unbiased judgement. our malay emotion corpus may contain some bias as the emotion labels were verified from the malaya documentation corpus as part of our effort to build upon existing language resources, and not annotated from scratch. nonetheless, we hope our study can serve as the benchmark for future research, especially in english-malay cross-lingual emotion classification using a higher quality goldstandard malay emotion corpus we have created. our malay emotion corpus can be expanded in the future to include more emotion annotations. as we only performed word-level mapping and refinement, we would like to explore sentencelevel mapping and refinement in future work to investigate if this will lead to further improvement. also, we would like to evaluate our model on standard malay emotion corpora to compare the performance of our model in formal and informal use of the malay language. in the future, we also plan to explore semisupervised and unsupervised approaches such as muse and vecmap in creating cross-lingual word embeddings. these approaches have shown to be promising for other language pairs. therefore, it is a possible direction to explore in building more computationally efficient cross-lingual models particularly for english-malay that can compete with or even outperform multilingual language models.