Cross-Lingual Unsupervised Sentiment Classification with Multi-View Transfer Learning (2020)

Abstract: Recent neural network models have achieved impressive performance on sentiment classification in English as well as other languages. Their success heavily depends on the availability of a large amount of labeled data or parallel corpus. In this paper, we investigate an extreme scenario of cross-lingual sentiment classification, in which the low-resource language does not have any labels or parallel corpus. We propose an unsupervised cross-lingual sentiment classification model named multi-view encoder-classifier (MVEC) that leverages an unsupervised machine translation (UMT) system and a language discriminator. Unlike previous language model (LM) based fine-tuning approaches that adjust parameters solely based on the classification error on training data, we employ the encoder-decoder framework of a UMT as a regularization component on the shared network parameters. In particular, the cross-lingual encoder of our model learns a shared representation, which is effective for both reconstructing input sentences of two languages and generating more representative views from the input for classification. Extensive experiments on five language pairs verify that our model significantly outperforms other models for 8/11 sentiment classification tasks.

## introduction recent neural network models have achieved remarkable performance on sentiment classification in english and other languages @xcite @xcite . however, their success heavily depends on the availability of a large amount of labeled data or parallel corpus. in reality, some low-resource languages or applications have limited labeled data or even without any labels or parallel corpus, which may hinder us from training a robust and accurate sentiment classifier. to build sentiment classification models for low-resource languages, recent researchers developed cross-lingual text classification (cltc) models @xcite , which transfers knowledge from a resource-rich (source) language to a low-resource (target) language. the core of those models is to learn a shared language-invariant feature space that is indicative of classification for both languages. therefore a model trained from the source language can be applied to the target language. based on how the shared feature space is learned, there are three categories, namely word-level alignments @xcite , sentence-level alignments @xcite and document level alignments @xcite . those models can well capture the semantic similarity between two languages. they, however, require parallel resources such as a bilingual dictionary, parallel sentences, and parallel wikipedia articles. such a limitation may prevent these models from being applicable in languages without any parallel resources. recently, there have been several attempts at developing "zero-resource" models @xcite @xcite . most notably, @xcite proposed a cross-lingual & cross-domain (clcd) model that builds on pivot based learning and bilingual word embedding. although clcd does not directly need labeled data or parallel corpus, it requires bilingual word embeddings (bwes) @xcite that requires thousands of translated words as a supervised signal. @xcite developed an adversarial deep averaging network to learn latent sentence representations for classification, but it had an implicit dependency on bwes @xcite that requires pretraining on a large bilingual parallel corpus. @xcite extended the cross-lingual model in @xcite to multiple source languages by using the unsupervised bwes @xcite and adding individual feature extractor for each source language, which eliminated the dependency on a parallel corpus. nevertheless, their model is very sensitive to the quality of bwes and performs poorly on distant language pairs such as english-japanese, as illustrated in their experimental study. in parallel, cross-lingual language models (lms) trained from raw wikipedia texts, such as multilingual bert foot_0 @xcite and xlm @xcite , have been prevalent in solving zero-shot classification problems @xcite . those models use the bert-style transformer @xcite architecture simultaneously trained from multiple languages to construct a sentence encoder, and fine-tune the encoder and a classifier on labeled training data from the source language. then the fine-tuned model is applied to the target language. the whole process does not require any labeled data or parallel corpus. however, under the "zero parallel resource" setting, the encoder trained from self-supervised masked language modelling within each language may not well capture the semantic similarity among languages, which could harm the generalization performance of fine-tuned models. in this paper, we propose a sentiment classification model called multi-view encoder-classifier (mvec) in an unsupervised setting, in which we only have monolingual corpora from two languages and labels in the source language. different from previous language model (lm) based fine-tuning approaches @xcite that adjust parameters solely based on the classification error of training data, we utilize the encoder-decoder network from unsupervised machine translation (umt) @xcite to regularize and refine the shared latent space. in particular, the transformer-based encoder regularized by a language discriminator learns shared but more refined language-invariant representations, which are effective for both reconstructing sentences from two languages by the decoder and generating multi-view feature representations for classification from input documents. in our model, we construct two views from the en-coder: (i) the encoded sentences in the source language; (ii) the encoded translations of the source sentences in the target language. our proposed mvec is partially initialized by pretrained lms @xcite but further fine-tuned to align sentences from two languages better, accurately predict labeled data in the source language and encourage consensus between the predictions from the two views. the full model is trained in an end-to-end manner to update parameters for the encoder-decoder, the language discriminator, and the classifier at each iteration. our contributions in this paper are as follows: ## methodology in this section, we will introduce our model's general workflow, including the details of each component and our training algorithm. ## conclusion in this paper, we propose a cross-lingual multiview encoder-classifier (mvec) that requires neither labeled data in the target language nor crosslingual resources with the source language. built upon pretrained language models, our method utilizes the encoder-decoder component with a language discriminator from an unsupervised machine translation system to learn a languageinvariant feature space. our approach departs from previous models that could only make use of the shared language-invariant features or depend on parallel resources. by constructing the finetuned latent feature space and two views of input from the encoder-decoder of umt, our model significantly outperforms previous methods for 8/11 zero-shot sentiment classification tasks.