Sentiment Analysis Using Aligned Word Embeddings for U ralic Languages (2023)

Abstract: In this paper, we present an approach for translating word embeddings from a majority language into 4 minority languages: Erzya, Moksha, Udmurt and Komi-Zyrian. Furthermore, we align these word embeddings and present a novel neural network model that is trained on English data to conduct sentiment analysis and then applied on endangered language data through the aligned word embeddings. To test our model, we annotated a small sentiment analysis corpus for the 4 endangered languages and Finnish. Our method reached at least 56% accuracy for each endangered language. The models and the sentiment corpus will be released together with this paper. Our research shows that state-of-the-art neural models can be used with endangered languages with the only requirement being a dictionary between the endangered language and a majority language.

## introduction most of the languages spoken in the world are endangered to one degree or another. the fact of being endangered sets some limitations on how modern nlp research can be done with such languages given that many endangered languages do not have vast textual resources available online, and even with the resources that are available, there is a question about the quality of the data resulting from a variety of factors such as fluency of the author, soundness of spelling and, on the lowest level, inconsistencies in character encoding (see @xcite . this paper focuses on the following uralic languages: erzya (myv), moksha (mdf), @xcite and @xcite . unesco classifies these languages as definitely endangered @xcite . in terms of nlp, these languages have fsts @xcite , universal dependencies treebanks @xcite (excluding udmurt) and constraint grammars available in giella repositories @xcite . for some of the languages, there have also been efforts in employing neural models in disambiguation @xcite and morphological tasks @xcite . out of these languages, only erzya has several neural based models available such as machine translation models @xcite , a wav2vec model and a stanza model @xcite . in this paper, we present a method for translating word embeddings models from larger languages into the endangered languages in question. furthermore, we fine-tune the models with language specific text data, align them and show results in a sentiment analysis task where no training data is provided in any of the endangered languages. we have made our data and models publicly available on zenodo foot_0 . ## related work apart from the work described earlier in the context of the endangered languages in question, there has been a lot of previous work on multilingual nlp where a model is trained in one language to sentence classification and then applied in the context of other languages. in this section, we will describe some of those approaches together with sentiment analysis approaches. a recent paper demonstrates sentiment analysis on 100 languages @xcite . the authors use roberta-xlm to extract feature vectors. these are then used in training a bidirectional lstm based classifier model. another line of work (liu and chen, 2015) compares several different multilabel classification methods on the task of sentiment analysis showing that rakel @xcite gave the best performance on raw token input. a recent paper @xcite demonstrated promising results in french sentiment analysis on a model that was trained in english, italian, spanish and german. the approach relied on a multilingual bert @xcite . @xcite suggests that lexicon based approaches, while viable for endangered languages, are not particularly suitable for sentiment analysis. in the context of cross-lingual nlp, there is work on pos tagging. for instance, @xcite propose a new model that does not require parallel corpora or other resources. the model uses a common blstm for knowledge transfer and another blstm for language-specific representations. it is trained using language-adversarial training and bidirectional language modeling as auxiliary objectives to capture both languagegeneral and language-specific information. another line of work by @xcite focuses on cross-lingual transfer of word embeddings, which aims to create mappings between words in different languages by learning transformation functions over corresponding word embedding spaces. the proposed algorithm simultaneously optimizes transformation functions in both directions by using distributional matching and minimizing back-translation losses. this approach uses a neural network implementation to calculate the sinkhorn distance, a distributional similarity measure, and optimize objectives through backpropagation. for machine translation chen et al. 2022 demonstrate the importance of both multilingual pretraining and fine-tuning for effective crosslingual transfer in zero-shot translation using a neural machine translation (nmt) model. the paper presents sixt+, a many-to-english nmt model that supports 100 source languages but is trained on a parallel dataset in only six languages. sixt+ initializes the decoder embedding and full encoder with xlm-r large @xcite and trains encoder and decoder layers using a twostage training strategy. ## data we use two books, suomi eilen ja nyt (finland yesterday and now) by @xcite and павлик морозов (pavlik morozov) by @xcite both of which are available in finnish, erzya, moksha, komi-zyrian and udmurt. the sentences of the books have been aligned across all the languages at the research unit for volgaic languages in university of turku. the size of the corpus for each language can be seen in furthermore, we lemmatize all the texts using the fsts provided in uralicnlp (hämäläinen, 2019). the corpus is lemmatized because we intend to translate and align a lemmatized word embeddings model. this also makes the overall approach more robust given that covering the entire morphology of a language would require us to have much larger corpora. ## word embeddings word embeddings capture the semantic and syntactic links between words by constructing vector representations of words. these vectors can be utilized to measure the semantic similarity between words, find analogous concepts, cluster words @xcite and more. in this work, we use english and finnish as the big languages that facilitate aligning and classifying words and sentences for the endangered languages. english has an overnumerous amount of linguistic resources, whether as raw text or labeled data, while the endangered resources that we are working with have translation dictionaries for finnish. for this reason, we use finnish as the intermediate language that bridges these endangered languages with english resources. the english model that we utilize is trained on the english wikipedia dump of february 2017 and gigaword 5th edition foot_1 @xcite . for finnish, we used recent word embeddings trained by language bank of finland (2022). these embeddings have been trained on several finnish newspapers. both of these models have been trained on lemmatized text. the english word vectors have a dimension size of 300, while the finnish word vectors have a dimension size of 100. in order to make the dimension sizes of the two sets of embeddings compatible, dimensionality reduction is applied to the english embeddings using principal component analysis (pca) @xcite . this process reduces the dimensionality of the english embeddings to 100, allowing them to be compared and analyzed alongside the finnish embeddings. ## sentence embeddings word embeddings represent the meaning of a single word, whereas sentence embeddings represent the meaning of an entire sentence or docu- ## sentiment analysis we create a sentiment classifier that takes in the sentence embeddings and predicts a sentiment polarity label. for training the sentiment analysis model, we use the stanford sentiment treebank @xcite , amazon reviews dataset @xcite and yelp dataset foot_4 . these datasets are available in english and we use their sentiment annotations (positive-negative) to train our model. the sentiment classifier is constructed as a three-layer fully-connected network, wherein the hidden layers are comprised of 300 neurons each. in order to mitigate overfitting, a dropout operation @xcite is performed prior to the final classification layer. the model consists of 121,202 trainable parameters in total, and is trained over the course of three epochs. ## results in this section, we show the results of the sentiment classification model on the in-domain, english-language train splits of the sentiment corpora we used to train the model. furthermore, we show the results of the sentiment classification model when applied on our own annotated data for the 4 endangered uralic languages in question and finnish. these results can be seen in all in all, our model performs relatively well. the accuracy for finnish is almost as high as it is for english despite not having any finnish sentiment annotated training data. this means that our approach can achieve rather good results when there is a lot of translation data available between the two languages. the results drop for the endangered languages, but we do find the 69% accuracy for erzya to be quite formidable, however, the result for komi-zyrian of 56% leaves some room for improvement. ## conclusions in this paper, we outlined a method for translating word embeddings from a majority language, finnish, to four minority languages -erzya, moksha, udmurt, and komi-zyrian. the word embeddings were aligned and a new neural network model was introduced. this model was trained using english data to carry out sentiment analysis and was then applied to data in the endangered languages using the aligned word embeddings. we built an aligned sentiment analysis corpus for the four endangered languages and finnish and used it to test our model. the results were promising and our study demonstrated that even the latest neural models can be utilized with endangered languages if a dictionary between the endangered language and a larger language is available.