e MLM : A New Pre-training Objective for Emotion Related Tasks (2021)

Abstract: BERT has been shown to be extremely effective on a wide variety of natural language processing tasks, including sentiment analysis and emotion detection. However, the proposed pretraining objectives of BERT do not induce any sentiment or emotion-specific biases into the model. In this paper, we present Emotion Masked Language Modelling, a variation of Masked Language Modelling aimed at improving the BERT language representation model for emotion detection and sentiment analysis tasks. Using the same pre-training corpora as the original model, Wikipedia and BookCorpus, our BERT variation manages to improve the downstream performance on 4 tasks from emotion detection and sentiment analysis by an average of 1.2% F-1. Moreover, our approach shows an increased performance in our task-specific robustness tests.

## introduction language models have been studied extensively in the nlp community @xcite @xcite @xcite , with approaches attaining state-of-the-art results on multiple token-level or sentence-level tasks. bert @xcite is a pre-trained language model, which proposed a new pre-training objective inspired by the cloze task @xcite , which enables the training of a deep bi-directional transformer network. this objective, called masked language modeling (mlm) is used on large amounts of unlabeled data from wikipedia and bookcorpus to produce powerful universal language representations. however, the pre-training does not take into account the downstream task on which the model will be applied. in this paper, we posit that we can leverage the characteristics of a downstream task to design better task-tailored pre-training objectives. concretely, we induce information from emotion or sentiment lexicons into our bert pre-training objective to improve the performance on tasks from sentiment analysis and emotion detection. there are numerous studies that focus on emotion detection @xcite @xcite @xcite @xcite @xcite and sentiment analysis @xcite @xcite @xcite @xcite . various lexicons have been used to improve model performance on these tasks. for instance, @xcite used occurrences of emotion words to identify various emotion types in news headlines. moreover, emotion lexicons have been used to produce important features which can be used inside a machine learning algorithm to improve the performance on emotion detection tasks @xcite @xcite . in this paper, however, instead of leveraging these lexicons to design features, in contrast, we use them to obtain language representations that are more suitable for emotion and sentiment tasks. to this end, we introduce emotion masked language modeling (emlm), a new pre-training bert @xcite objective aimed at improving the bert performance on tasks related to sentiment analysis and emotion detection. inspired by the well-known masked language modeling objective, emlm adds only a few simple, yet powerful changes. instead of uniformly masking the tokens in the input sequence, emlm leverages lexicon information, and assigns higher masking probabilities to words that are more likely to be important in the sentiment or emotion contexts. to enable a fair comparison with the vanilla bert model, we train the emlm bert model in the same fashion as the vanilla bert, pre-training on wikipedia and bookcorpus @xcite . to our knowledge, we are the first to study different masking probabilities for the bert pre-training procedure guided by sentiment and emotion lexicons. similar to our work, some studies also focused on incorporating sentiment information into pre-trained language models. for example, yin et al. ( 2020 ) built an attention network on top of bert to predict sentiment labels of phrase nodes obtained through a constituency parse tree. on the other hand, tian et al. (2020) designed various pretraining objectives, such as masking and predicting all words from a pre-defined small set of seeds, and predicting an aspect-sentiment pair or the polarity of words. in contrast, we leverage information from available sentiment and emotion lexicons. we show the feasibility of our approach by testing emlm on two sentiment analysis benchmark datasets and two emotion detection datasets. these datasets span diverse domains, such as movie reviews, online health communities, and reddit discussions, enabling a comprehensive analysis of emlm. our contributions are as follows: 1) we introduce a new pre-training objective for bert (leveraging available lexicons), aimed at producing better task-guided universal representations for downstream tasks from sentiment analysis and emotion detection. we offer the pre-trained model as an easy way to leverage our approach on downstream applications. 2) we show the efficacy of our approach by testing our method on four benchmark datasets for emotion and sentiment and obtain an average improvement in f1 score of 1.2%. 3) we verify the robustness of our model in the face of input perturbations, which occur frequently in informal contexts (e.g., due to mispellings). ## proposed approach background bidirectional encoder from transformers for language understanding (bert) @xcite is a pre-trained language model trained on large amounts of unlabeled data using two objectives: 1) masked language modeling (mlm) randomly masks 15% of tokens in a sequence, followed by a supervised prediction of the masked tokens; 2) next sentence prediction (nsp) predicts in a binary fashion if two sentences follow each other. by using these two tasks on large-scale data repositories such as bookcorpus (800m words) @xcite and wikipedia (2, 500m words), bert produces powerful universal language representations, applicable on a wide range of tasks, such as sentiment analysis, question answering, and commonsense reasoning. however, to be used in various downstream tasks, bert has to undergo a task-specific finetuning step @xcite , where the contextualized embedding is adapted to the needed task. we posit that we can improve the downstream performance by focusing on the target task in the pre-training phase as well. specifically, we focus on sentiment analysis and emotion detection, and show that task-guided unsupervised pre-training helps the performance considerably. masking emotion words now we introduce emotion masked language modeling (emlm), a variation of mlm targeted at inducing emotion or sentiment-specific biases in the bert pre-training phase. specifically, unlike bert, which uses a uniform probability (15%) to mask the tokens in an input sentence, we assign higher probabilities to tokens which are emotionally rich words from an available lexicon l. we denote this probability by k, which is a hyperparameter in our emlm method. our masking process can be summarized as follows: given an input sentence s: 1) we extract the words that belong to the lexicon l, and we denote them by e; 2) we set the masking probability of these words as p (w e ) = k ∀ w @xmath0; 3) to ensure we mask 15% of the words in total, we lower the masking probability of the non-emotionally-rich words using the following formula: ## experiments and results in this section, we first describe our experimental setup ( §3.1), then present our datasets and lexicons ( §3.2), and then discuss the results that contrast emlm with the original bert mlm ( §3.3). ## robustness test it has been shown that neural models are often sensitive to various input perturbations @xcite . in this section, we aim to investigate the robustness of our proposed approach in the face of input noise. we focus on the following two questions: 1) does emlm improve the robustness of the model? 2) what type of input noise is successful in misleading our model? we study these questions on the sst-5 sentiment analysis task using the framework introduced by @xcite . we explore three ways to generate input perturbations and verify their "success." we say a perturbation is "successful" on a model m for an example e if 1) the model m classifies e correctly and 2) the model m misclassifies the example e when noise is applied to it. naturally, the lower the perturbation success rate, the more robust a model is. the perturbations that we considered are as follows: ## conclusion in this paper, we introduced a new bert pretraining objective suited for sentiment analysis and emotion detection tasks. we showed that the approach is feasible; it needs no additional pretraining compared to the vanilla bert, and improves the performance by 1.2% f1 on average on various tasks. our analysis also suggests that emlm is more robust in the face of input perturbations. as future work, we note that our approach is general enough, so we plan to leverage different lexicons outside the sentiment analysis and emotion detection domains to investigate if the model generalizes well on other domains (e.g., financial). we also plan to study if our method is effective for non-english languages. finally, we note that there exist lexicons that assign to words not only their emotion, but also their emotion intensity @xcite . therefore, we plan to investigate if associating the masking probability with the emotion intensity (i.e., assign a higher probability to a more intensive word) would further help improve the performance.