Curriculum Knowledge Distillation for Emoji-supervised Cross-lingual Sentiment Analysis (2022)

Abstract: Existing sentiment analysis models have achieved great advances with the help of sufficient sentiment annotations. Unfortunately, many languages do not have sufficient sentiment corpus. To this end, recent studies have proposed cross-lingual sentiment analysis to transfer sentiment analysis models from resource-rich languages to low-resource languages. However, these studies either rely on external cross-lingual supervision (e.g., parallel corpora and translation model), or are limited by the cross-lingual gaps. In this work, based on the intuitive assumption that the relationships between emojis and sentiments are consistent across different languages, we investigate transferring sentiment knowledge across languages with the help of emojis. To this end, we propose a novel cross-lingual sentiment analysis approach dubbed Curriculum Knowledge Distiller (CKD). The core idea of CKD is to use emojis to bridge the source and target languages. Note that, compared with texts, emojis are more transferable, but cannot reveal the precise sentiment. Thus, we distill multiple Intermediate Sentiment Classifiers (ISC) on source language corpus with emojis to get ISCs with different attention weights of texts. To transfer them into the target language, we distill ISCs into the Target Language Sentiment Classifier (TSC) following the curriculum learning mechanism. In this way, TSC can learn delicate sentiment knowledge, meanwhile, avoid being affected by cross-lingual gaps. Experimental results on five cross-lingual benchmarks clearly verify the effectiveness of our approach.

## introduction nowadays, sentiment analysis approaches perform very well by fine-tuning large-scale pre-trained language models @xcite @xcite . however, their success heavily corresponding author: f. lv. relies on manual sentiment annotations. therefore, they fail to recognize the sentiment polarities in low-resource languages that do not have sentiment supervision. still, recent studies have attempted to identify sentiment in unlabeled languages by transferring sentiment knowledge from resource-rich languages, (e.g., english), to low-resource languages (e.g., japanese and arabic) and propose the cross-lingual sentiment analysis (clsa) task @xcite . generally, these clsa approaches can be divided into two patterns, parallel corpus based approaches, and unsupervised approaches. through cross-lingual supervision provided by parallel corpus, the former kind of approaches can align the semantic gaps between source language and target language by auto-encoder @xcite , neural translation model @xcite and bilingual word embedding @xcite . however, the availability of parallel corpus limits the usability of these works. to avoid relying on external cross-lingual supervision, recent studies have paid attention to a more challenging setting dubbed unsupervised cross-lingual sentiment analysis (uclsa) @xcite @xcite . under this setting, sentiment classifiers need to adapt to new languages without the help of parallel corpus. cross-lingual language models such as m-bert @xcite , xlm-r @xcite and labse @xcite have been prevalent in uclsa. these models are pretrained by multilingual corpus and then fine-tuned by source language sentiment supervision. in addition, unsupervised machine translation @xcite and multilingual language model @xcite have been proposed to align source and target languages in uclsa. although the above unsupervised approaches have calibrated semantic representation over source and target languages, their performances are still in step 2, {isc 1 , isc 2 , ..., isc n } are distilled from the former model of each of them to obtain models with different transferability and sentiment veracity on w s . isc 1 is distilled from s and other isc j is distilled from isc j-1 . note that the attention weight of texts is decreased progressively with the model id j increased. thus, the model with smaller attention weights on texts has more transferability but less precise sentiment knowledge. in step 3, we distill {isc 1 , isc 2 , ..., isc n } into t sc following the curriculum learning mechanism on w t . the losses of those more transferable iscs begin with larger weights but then smaller. on the contrary, the weights of those less transferable iscs begin with larger but then smaller. the most intuitive way to learn esclsa is manually labeling polarities of emojis, and using the emoji label as the pseudo-label of target language sentences. however, emojis fail to represent neutral sentiment or distinguish delicate polarities, e.g., very positive and positive. meanwhile, the pseudolabel based method ignores the sentiment information in texts. thus, it can not be well adapted to practical application. to handle these challenges, an end-to-end framework of esclsa needs to be established. elsa @xcite is the only previous approach for esclsa. it uses emoji prediction task to learn sentence representations. after that, elsa uses these representations to classify sentiment polarities. the emoji-oriented representations are transferable. however, they ignore the information carried in text, which limits the performance of elsa. in this paper, we propose a novel esclsa approach dubbed curriculum knowledge distiller (ckd). the overall architecture of ckd is shown in figure 1 . texts can reveal the precise sentiment but are hard to transfer due to the cross-lingual gaps. on the other hand, emojis are more transferable, but the sentiment information in texts will be underestimated if we focus too much on the emojis @xcite . to address this limitation in the previous work @xcite , we propose to distill multiple intermediate sentiment classifiers (iscs) from source language sentiment classifier (ssc) on source language tweets with emojis. by adjusting the attention weights of texts in iscs, iscs has different transferability and sentiment veracity. the models which are trained with low attention weights of texts (i.e., emoji-dominant iscs), are more transferable, while the models with high attention weights of texts (i.e., text-dominant iscs) pay more attention to extract sentiment information from texts. based on these iscs, we distill the target language sentiment classifier (tsc) following the curriculum learning mechanism @xcite . the tsc is trained by distillation losses computed from different intermediate models. specifically, in the beginning of distilling tsc, the weights of emoji-dominant iscs are large, as they are easier to transfer. during the distillation process, we progressively increase the weights of text-dominant iscs to learn more precise sentiment knowledge from texts. compared with the previous works, ckd can integrate more transferable information from emojis with more precise sentiment information in texts. thus, ckd achieves the balance between transferbility and sentiment veracity. we verify our approach on five cross-language sentiment analysis benchmarks. the experimental results clearly support the effectiveness of our approach. to sum up, the contributions of this work are three-fold: ## related work this section briefly reviews works related to ours, including cross-lingual sentiment analysis, knowledge distillation, and curriculum learning. ## methodology in this section, we introduce the methodology of our proposed ckd. the key idea of our approach is using emojis that express the consistent sentiment in source and target languages to distill sentiment knowledge across languages. therefore, we use source language supervision to train ssc, firstly. then, as emojis can not explain such precise sentiment as texts but are easier to transfer, we propose to distill multiple iscs with various attention weights of texts. the sentiment knowledge learned algorithm 1 ckd training pipeline ssc m , isc m j , t sc m ← ssc, the j-th isc, and tsc models at iteration m; ssc m-1 , isc m-1 j , t sc m-1 ← ssc, the j-th isc, and tsc models at iteration m -1; x s ← source language labeled samples; w s ← source language unlabeled tweets with emojis; w t ← target language unlabeled tweets with emojis; by iscs with higher text attention weights is more precise but less transferable. finally, we distill iscs into tsc following the curriculum learning mechanism. specifically, for the iscs that are easy to transfer, we assign high weights of their distillation losses first, then decrease the weights during training. on the contrary, for the iscs that are hard to transfer, we reverse the direction of weight change. in this way, tsc can learn precise sentiment knowledge from those hard-to-transfer iscs, meanwhile avoiding the language discrepancy problem with the help of those easy-to-transfer iscs. in the following subsections, we will introduce the implementation details of our proposed approach. the overall architecture of ckd is shown in figure 1 . ## experiments in this section, we conduct extensive experiments to verify the effectiveness of our proposed approach. following fei and li (2020), eleven crosslingual tasks based on five language pairs are used in the experiments. all those pairs take english to be the source language, and the target languages are german, french, japanese, chinese and arabic, respectively. ## conclusion emojis are widely used in social networks of various languages. based on the intuitive assumption that the relationships between emojis and sentiments are cross-lingual consistent, we use the unsupervised multilingual corpus with emojis to transfer sentiment knowledge across languages. to achieve this goal, we propose ckd, a novel curriculum sentiment knowledge distillation framework. using the source language supervision trained model, we distill a series iscs. these iscs have different transferability and sentiment veracity. finally, we distill rscs into tsc under the curriculum learning mechanism. that is, in the beginning of training, the weights of emoji-dominant iscs are large, as they are easier to transfer. then we increase the weights of text-dominant iscs to learn more precise sentiment knowledge. in this way, ckd learns precise sentiment knowledge, meanwhile avoiding the language discrepancy problem. extensive experiments on five languages involving 11 tasks verify the effectiveness of our approach.