Multilingual and Multilabel Emotion Recognition using Virtual Adversarial Training (2021)

Abstract: Virtual Adversarial Training (VAT) has been effective in learning robust models under supervised and semi-supervised settings for both computer vision and NLP tasks. However, the efficacy of VAT for multilingual and multilabel emotion recognition has not been explored before. In this work, we explore VAT for multilabel emotion recognition with a focus on leveraging unlabelled data from different languages to improve the model performance. We perform extensive semi-supervised experiments on SemEval2018 multilabel and multilingual emotion recognition dataset and show performance gains of 6.2% (Arabic), 3.8% (Spanish) and 1.8% (English) over supervised learning with same amount of labelled data (10% of training data). We also improve the existing state-of-the-art by 7%, 4.5% and 1% (Jaccard Index) for Spanish, Arabic and English respectively and perform probing experiments for understanding the impact of different layers of the contextual models.

## introduction emotion recognition is an active and crucial area of research, especially for social media platforms. understanding the emotional state of the users from textual data forms an important problem as it helps in discovering signs of fear, anxiety, bullying, hatred etc. and maintaining the emotional health of the people and platform. with the advent of deep neural networks and contextual models, text understanding has advanced dramatically by leveraging huge amount of unlabelled data freely available on web. however, even with these advancements, annotating emotion categories is expensive and time consuming as emotion categories are highly correlated and subjective in nature and can co-occur in the same text. psychological studies suggest that emotions like "anger" and "sadness" are corelated and co-occur more frequently than "anger" and "happiness" @xcite . in a multilingual setup, the annotation becomes even more challenging as annotator team are expected to be familiar with different languages and culture for understanding the emotions accurately. imbalance in availability of the data across languages further creates problems, especially in case of resource impoverished languages. in this work, we investigate the following key points; a) can unlabelled data from other languages improve recognition performance of target language and help in reducing requirement of labelled data? b) efficacy of vat for multilingual and multilabel setup. to address the aforementioned questions, we focus our experiments towards semi-supervised learning in a multilingual and multilabel emotion classification framework. we formulate semi-supervised virtual adversarial training (vat) @xcite for multilabel emotion classification using contextual models and perform extensive experiments to demonstrate that unlabelled data from other languages l @xmath0, l 2 , . . . , l n } improves the classification on the target language l tgt . we obtain competitive performance by reducing the amount of annotated data demonstrating crosslanguage learning. to effectively leverage the multilingual content, we use multilingual contextual models for representing the text across languages. we also evaluate monolingual contextual models to understand the performance differences between multilingual and monolingual models and explore the advantages of domain-adaptive and task-adaptive pretraining of models for our task and observe substantial gains. we perform extensive experiments on the semeval2018 (affect in tweets: task e-c foot_0 ) dataset @xcite which contains tweets from twitter annotated with 11 emotion categories across three languages -english, spanish and arabic and demonstrate the effectiveness of semi-supervised learning across languages. to the best of our knowledge, our study is the first one to explore semi-supervised adversarial learning across different languages for multilabel classification. in summary, the main contributions of our work are the following: ## related work semi-supervised learning is an important paradigm for tackling the scarcity of labelled data as it marries the advantages of supervised and unsupervised learning by leveraging the information hidden in large amount of unlabelled data along with small amount of labelled data @xcite , @xcite . early approaches used self-training for leveraging the model's own predictions on unlabelled data to obtain additional information during training @xcite . @xcite proposed cross-view training (cvt) for various tasks like chunking, dependency parsing, machine translation and reported state-of-theart results. cvt forces the model to make consistent predictions when using the full input or partial input. ladder networks @xcite , mean teacher networks @xcite are another way for semi-supervised learning where temporal and model-weights are ensembled. another popular direction towards semisupervised learning is adversarial training where the data point is perturbed with random or carefully tuned perturbations to create an adversarial sample. the model is then encouraged to maintain consistent predictions for the original sample and the adversarial sample. adversarial training was initially explored for developing secure and robust models @xcite , @xcite , @xcite to prevent attacks. miyato et al. (2016), cheng et al. (2019), zhu et al. (2019) showed that adversarial training can improve both robustness and generalization for classification tasks, machine translation and glue benchmark respectively. miyato et al. (2016 emotion recognition is an important problem and has received lot of attention from the community @xcite , @xcite . the taxonomies of emotions suggested by plutchik wheel of emotions @xcite and @xcite have been used by the majority of the previous work in emotion recognition. emotion recognition can be formulated as a multiclass problem (scherer and wallbott, 1994), (mohammad, 2012) or a multilabel problem @xcite , @xcite . in the multiclass formulation, the objective is to identify the presence of one of the emotion from the taxonomy whereas in a multilabel setting, more than one emotion can be present in the text instance. binary relevance approach @xcite is another way to break multilabel problem into multiple binary classification problems. however, this approach does not model the co-relation between emotions. seq2seq approaches @xcite , @xcite solve this problem by modelling the relationship between emotions by inferring emotion in an incremental manner. an interesting direction for handling data scarcity in emotion recognition is to use distant supervision by exploiting emojis @xcite , hashtags @xcite or pretraining emotion specific embeddings and language models @xcite , @xcite . with the emergence of contextual models like bert @xcite , roberta @xcite etc., the field of nlp and text classification has been revolutionized as these models are able to learn efficient representations from a huge corpus of unlabelled data across different languages and domains @xcite , @xcite . social media content contains linguistic errors, idiosyncratic styles, spelling mistakes, grammatical inconsistency, slangs, hashtags, emoticons etc. @xcite , @xcite due to which off-the-shelf contextual models may not be optimum. we use languageadaptive, domain-adaptive and task-adaptive pretraining which has shown performance gains @xcite , @xcite , @xcite , @xcite , @xcite for different tasks and domains. ## methodology we consider the task of multilabel emotion classification, where given a text @xmath1, w 2 , . . . , w l }, we predict the presence of y emotion categories denoted by {1, 2, . . . , y}. t represents the corpus of all the sentences across the different languages and w i represent the tokens in the sentence. we leverage contextual models as feature extractors φ : t @xmath2, where x @xmath3. ## semi-supervised experiments we select a percentage (10%, 25%, 50%, 100%) of the data from the target language as labelled data and use the remaining data from same language along with data of other languages as the unlabelled data. in similar observations on english can be made from ## domain and task adaptive pretraining in this section, we perform supervised learning experiments with frozen and finetuned representations by using the labelled data of each language for evaluating the performance of domainadaptive, task-adaptive, monolingual and multilingual contextual models. in we further evaluate domain-adaptive (xlm-tw) and task-adaptive (xlm-tw-s) versions of the xlm-r multilingual model and observe substantial improvements. xlm-tw-s improves the jaccard index (ji) by 5.5%, 6.5% and 8.4% for arabic, english and spanish respectively, highlighting the advantages of task-specific pretraining for contextual models. xlm-tw also improves upon xlm-r for all the languages reiterating the importance of pretraining the contextual models with domain specific data. we study the impact of finetuning the monolingual and best performing multilingual model on our task to compare the capabilities of multilingual models with monolingual after finetuning on the task. we notice that finetuning bridges the gap to some extent but still the domain adaptive multilingual xlm-tw works better than the finetuned model ji mif1 maf1 xlm-tw-s 52.0 64.4 47.9 xlm-tw 49.3 62.2 47.1 xlm-r 45.2 58.4 42.9 mbert 37.5 51.2 36.2 arabert 46.4 59.7 43.7 overall, our results improve upon the existing approaches on jaccard index(ji) by 7% for spanish, 4.5% for arabic and around 1% for english and setup a new state-of-the-art for all the three languages highlighting the efficacy of semi-supervised learning and domain-adaptive multilingual models. ## probing experiments we perform experiments to evaluate the contribution of different layers of the xlm-tw-s model. we extract representation of the tokens of a sentence from a particular layer of the contextual model and take an average across tokens for obtaining the representation of the sentence. we train a classifier over these sentence representations and report the results. from figure 2 , we note that higher layers provide better performance for all the three languages showing that the higher-order contextual information is useful for understanding the emotions in the text. refer appendix a for detailed results. similar to @xcite , we also compute the improvement due to incrementally adding more layers to the previous layers and calculate the expected layer: ## training details we finetune the contextual models following huggingface foot_1 with a batch size of 8, learning rate of 2e-5 and weight decay of 0.01 using adamw optimizer for 30 epochs. the classifier is a two layered neural network with 768 hidden dimensions and 11 output dimensions with 0.1 dropout. for mlvat experiments, the number of examples sampled from the unlabelled set for each batch are 24, and α are set to 0.5 and 1 using cross validation. we apply sigmoid over the logits and train using binary cross entropy loss. we use validation set for finding optimal hyperparameters and evaluate on the test set using combination of training and validation set for training. ## conclusion in this work, we explored semi-supervised learning using virtual adversarial training (vat) for multilabel emotion classification in a multilingual setup and showed performance improvement by leveraging unlabelled data from different languages. we used mean square error (mse) as the divergence measure for leveraging vat for multilabel emotion classification. we also evaluated the performance of monolingual, multilingual and domain-adaptive and task-adaptive multilingual contextual models across three languages -english, spanish and arabic for multilabel and multilingual emotion recognition and obtained state-of-the-art results. we also performed probing experiments for understanding the impact of different layers of contextual models.