Leveraging Multilingual Resources for Language Invariant Sentiment Analysis (2020)

Abstract: Sentiment analysis is a widely researched NLP problem with state-of-the-art solutions capable of attaining human-like accuracies for various languages. However, these methods rely heavily on large amounts of labeled data or sentiment weighted language-specific lexical resources that are unavailable for low-resource languages. Our work attempts to tackle this data scarcity issue by introducing a neural architecture for language invariant sentiment analysis capable of leveraging various monolingual datasets for training without any kind of cross-lingual supervision. The proposed architecture attempts to learn language agnostic sentiment features via adversarial training on multiple resource-rich languages which can then be leveraged for inferring sentiment information at a sentence level on a low resource language. Our model outperforms the current state-of-the-art methods on the Multilingual Amazon Review Text Classification dataset [REF] and achieves significant performance gains over prior work on the low resource Sentiraama corpus [REF]. A detailed analysis of our research highlights the ability of our architecture to perform significantly well in the presence of minimal amounts of training data for low resource languages.

## introduction sentiment analysis refers to a series of methods, techniques, and tools aimed at extracting the intended sentiment from a written opinion. traditional sentiment analysis techniques have relied on using supervised term weighting methods including terms' distribution of classes, word-level polarity scoring and using svms @xcite and naive bayes classifiers @xcite for pattern extraction using hand-crafted features. the advent of deep learning techniques for sentiment analysis has now enabled the extraction of high quality sentiment data from written texts. one majorly overlooked factor in the performance of these neoteric approaches is their dependency on large annotated datasets compiled from multiple data sources related to or sourced from newspapers, tweets, photos and product reviews. @xcite @xcite @xcite . given global nature of the current information sharing infrastructure, most data generated belongs to one of the three languages : english, mandarin or spanish. this abundance of raw data aids and motivates the creation of annotated resources in these languages. conversely, the paucity of annotated data in most languages makes it a challenging task to develop deep learning based solutions for them. hence there is a pressing need to pay special attention to developing solutions capable of sentiment analysis in a low resource setting. some of the initial methods that attempt to tackle this problem of data scarcity using transfer learning (training a neural model on one language and applying the trained model on another language via weight sharing) do not perform well due to the limited overlap between the vocabularies of the different languages and difference in their syntactic structure @xcite . cross-lingual sentiment classification (clsc) methods try to alleviate this problem by leveraging labeled data from one language to improve the performance on another language @xcite . however, these methods typically rely on auxiliary cross-lingual resources such as a parallel corpora @xcite , bilingual lexicons @xcite or the use of machine translation systems @xcite @xcite . unfortunately, the curation of such cross-lingual resources is both a time and a labour intensive task. hence, there is a need for architectures that can perform well in the absence of such cross-lingual resources. in this paper, we address this problem by presenting a neural language invariant sentiment analyzer (lisa) architecture that is capable of training on multiple monolingual sentiment labelled datasets to learn language agnostic sentiment features that can be transferred to perform sentiment analysis in low-resource languages without leveraging any form of cross-lingual supervision. approach : we formulate this problem as a multi-lingual transfer learning (mltl) language adaptation task where we attempt to learn language agnostic sentiment features via adversarial training on labelled documents (s 1 , s 2 ...s n ) from multiple (source) languages to improve the performance on documents (t 1 , t 2 ...t m ) from a low resource (target) language. the key components of our approach include learning monolingual word embeddings from s 1 , s 2 ...s n , t 1 , t 2 ...t m and projecting them to a shared multilingual semantic space. we employ an lstm network to learn latent features (z) from this multilingual space which is then used by a sentiment classifier (s c ) to predict the sentiment polarity of a document @xmath0...s n , t 1 ...t m }. concurrently, a language classifier (c l ) is trained to predict the language of document d based on z. during the adversarial training we try to minimize the binary cross-entropy loss of c s , while at the same time we maximize the crossentropy loss of c l . this results in a setting where the lstm learns to produce latent features z that predicts the sentiment of document d correctly independent of the language of document d. we hy-pothesize that in this setting, the latent features (z) trained would contain sentiment features that are language agnostic. in summary, the main contributions of this paper are : the paper is structured as follows : section 2 highlights the related prior work in the field of clsc. section 3 introduces the datasets that are used in our experiments. section 4 presents the methodology used to align multiple monolingual semantic spaces to a common multilingual semantic space. section 5 describes in detail the various components of the lisa architecture. section 6 explains the adversarial training methodology employed. section 7 describes our experimental setup and provides a detailed comparison of our approach with prior work in both the low-resource and no-resource setting. section 8 addresses the advantages and shortcomings of the proposed approach and state our concluding remarks. clsc using cross-lingual resources : most popular methods in clsc makes use of crosslingual resources to bridge the language barrier and induce inter-language correspondence. @xcite used a bilingual dictionary to translate documents in the target language to the source language and trained a classifier in the source language for text classification. @xcite used a bilingual lexicon to translate subjective words and phrases in the source language into the target language. @xcite utilizes a bilingual dictionary to translate the classification model from a source language to a target language rather than the documents themselves. @xcite used wordnet senses as features for clsa in indian languages (hindi and marathi). the clmm model @xcite treated the source language and the target language words in an unlabeled bilingual parallel dataset as generated simultaneously by a set of mixture components. the cr-rl approach @xcite learned word embeddings by using a set of bilingual word pairs where one part of the word vector contains language specific features and the other part contains language independent features. cl-scl model @xcite leveraged structural correspondence learning with the help of a bilingual dictionary to learn a source-target feature space. @xcite used a parallel corpus between the source language and the target language to learn bilingual paragraph vectors (bi-pv). umm @xcite learned multilingual sentimentaware word representations based on unlabeled parallel data and used pivot languages to transfer sentiment information in the absence of parallel data . the cldfa approach (xu and yang, 1 https://translate.google.com/ 2017) adopted cross-lingual distillation and adversarial techniques on parallel corpora for clsc. our work draws inspiration from the adan-grl model @xcite which employed language adversarial training to learn language invariant features from bilingual word embeddings (bwe) which were created using a parallel corpus. in fact, our proposed model can be considered as a cross-lingually unsupervised variant of the adan-grl model as we do not rely on parallel corpora to learn word representations. furthermore, the adan-grl model is limited by the bwe to only incorporate two language pairs (source and target) during training, whereas our lisa system is capable of leveraging multiple source languages and the target language for adversarial training. clsc without cross-lingual supervision neoteric advances by @xcite alleviates the need for cross-lingual resources by introducing a shared-private mixture-of-experts model (moe) that learns both language specific features and language invariant features without cross-lingual supervision. our work, although related to moe in objective with respect to the lack of cross-lingual supervision, differs in the methodology. direct comparison of our architecture against moe ( ## dataset description we conduct our experiments on two publicly available sentiment classification datasets : the multilingual amazon review text classification dataset @xcite consists of sentiment labelled data in multiple languages. the vast amount of prior work on this dataset helps us to directly compare our results with the pre-existing state-of-the-art clsc methods. the sentiraama corpus (gangula and mamidi, 2018) is a real-world low resource sentiment corpus in telugu (an agglutinating indian language). we use this dataset to test the robustness of our system and evaluate our results in a truly low resource setting. in the following subsections we describe both the corpora in detail. ## multilingual word representation for our experiments, we train fasttext embeddings @xcite to project each word to a monolingual semantic space for each language in the datasets described in section 3. we then employ the unsupervised muse approach @xcite to align the monolingual spaces of each language in an adversarial manner to a common multilingual semantic space. while training muse we use english as the target semantic space and align all the other monolingual semantic spaces to this space. let @xmath1, x 2 , . . . x a } and @xmath2, y 2 , . . . y b } be the source and target fasttext word embeddings respectively. let w be a linear mapping from x to y. a discriminator is trained to discriminate between elements randomly sampled from w x and y while w (which acts as the generator) is jointly trained to fool the discriminator. the discriminator loss function l d (θ d |w ) is formulated as: the mapping objective function used to train w is given by: where θ d denotes the discriminator parameters and p θ d (@xmath3. next, a synthetic parallel vocabulary consisting of the most frequent words and their mutual nearest neighbors are extracted from the resulting shared embedding space w to fine-tune the mapping using the closed-form procrustes solution @xcite given by: where x and y are two aligned matrices containing the embeddings of the words in the trained space w , d represents the dimension of the embeddings, o d (r) is the space of d x d matrices of real numbers with the orthogonality constraint and svd(y x t ) represents the singular value decomposition of y x t . ## lisa architecture the input to the lisa model is a review r i that is made up of a sequence of words w 1 , w 2 , . . . w k . each review r i is associated with a language label l @xmath4, l 2 , . . . l p } is the set of all language labels used in training. additionally, each review r i is also associated with a sentiment label t @xmath5, negative} which denotes the sentiment polarity of the review. we project each word w i to the multilingual semantic space (from section 4) to obtain a sequence of n-dimensional word embeddings e 1 , e 2 , . . . e k where e @xmath6. the following subsections describe in detail the individual components of the lisa architecture. figure 1 shows the overall architecture of the proposed model. ## adversarial training inspired by recent works @xcite @xcite , we train the lisa model using adversarial training on a set of labeled reviews @xmath7, r 2 , . . . r n }. the aim of the lisa model is to predict the sentiment label t i for a given review r i independent of the language label l i . we formulate the learning objective in a way that minimizes the sentiment classification loss from c s and maximizes the language classification loss from c l . as a result, the lisa model tries to jointly optimize the below functions: where f denotes the loss function used. this results in a setting where the c l tries to predict l i based on a given h(r i ) and the encoder h tries to 'fool' the c l by learning to create h(r i ) that is minimally influenced by the language label l i while at the same time, is maximally influenced by the c s to predict he sentiment label t i correctly. the m-list model @xcite presents a similar setting for the task of open domain event detection that was trained using a gradient reversal layer grl λ @xcite between h and c l . by using grl λ , the optimization functions (equations 1 and 2) can be simplified as : ## experiments and results in this section we present an extensive set of experiments conducted on the multilingual amazon review text classification dataset and the telugu sentiraama sentiment classification corpus. we evaluate our approach in the two settings described below : low-resource setting : we evaluate the performance of the lisa architecture in the low-resource setting (termed lisa-lr) by training it on the train sets from multiple source languages and the limited train set in the target language and then testing on the test set of the target language. no-resource setting : in the no-resource setting, we assume that the training data is not available for the target language. we train the lisa model (termed lisa-nr) on the train sets of the source languages and evaluate the model on the target language test set. lisa -no language discriminator : to show the effectiveness of the language discriminator (c l ), we conduct ablation experiments in the lowresource setting where we remove c l from the lisa architecture. in this variant of the lisa model (termed lisa-nold), the sentiment analyzer only depends on the muse embeddings to learn h(r i ) to learn sentiment features. our experiments show that lisa-lr performs significantly better in most cases than lisa-nold. for the multilingual amazon review text classification dataset in the low-resource setting, we train lisa-lr on the train sets of all the four languages. we then test it on the test set of the target language. in the no-resource setting, we train lisa-nr on the train sets of three languages and test it on the test set of the fourth language. we do this for each domain in the corpus independently. we compare our results against prior state-of-theart methods that uses machine translation systems (cl-mt and bidrl), methods that leverage cross-lingual supervision (umm, bi-pv, cr-rl and cl-scl) and the cross-lingually unsupervised man-moe method of @xcite . the results are presented in for the sentiraama corpus in the low-resource setting, we train lisa-lr by leveraging the train sets of all the languages in the multilingual amazon dataset along with the sentiraama train set. we then test the system on the sentiraama test set. in the no-resource setting, lisa-nr only utilizes the train set of all the languages in the multilingual amazon dataset and test the system on the sentiraama test set. we do this for the books and movies domain separately. we evaluate the results of lisa-lr, lisa-nr and lisa-nold against the bernoulli naive bayes @xcite and svm @xcite baselines that use tf-idf features which were set by gangula and mamidi (2018). the experimental results are given in ## analysis and conclusion analysis : the results on the multilingual amazon review text classification dataset proves our hypothesis that our model learns language invariant features that can be generalized across languages. the empirical results in