Continuing Pre-trained Model with Multiple Training Strategies for Emotional Classification (2022)

Abstract: Emotion is the essential attribute of human beings. Perceiving and understanding emotions in a human-like manner is the most central part of developing emotional intelligence. This paper describes the contribution of the LingJing team’s method to the Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis (WASSA) 2022 shared task on Emotion Classification. The participants are required to predict seven emotions from empathic responses to news or stories that caused harm to individuals, groups, or others. This paper describes the continual pre-training method for the masked language model (MLM) to enhance the DeBERTa pre-trained language model. Several training strategies are designed to further improve the final downstream performance including the data augmentation with the supervised transfer, child-tuning training, and the late fusion method. Extensive experiments on the emotional classification dataset show that the proposed method outperforms other state-of-the-art methods, demonstrating our method’s effectiveness. Moreover, our submission ranked Top-1 with all metrics in the evaluation phase for the Emotion Classification task.

## introduction emotion is an important component of human daily communication. however, with the growing interest in human-computer interfaces, machines still lag in possessing and perceiving emotions. understanding human emotional states in dialogue is crucial for building natural human-machine interaction, which aims to generate appropriate responses. emotion classification (emo) in the text is concentrated on projecting words, sentences, and documents to a set of emotions according to psychological models proposed by @xcite , which is an interdisciplinary field of study that span psychology and computer science. this task has evolved * these authors contribute equally to this work. from a purely research-oriented topic to play a role in various applications, including mental health assessment, intelligent agents, social media mining @xcite . therefore, emotion classification has become a hot topic in the field of natural language processing (nlp), and lots of research efforts have been devoted to its development. with the rapid development of artificial intelligence technology, especially deep learning, researchers have made substantial progress on emo tasks over the past few decades. before the era of deep learning, traditional emo methods not only ignore the order of occurrence of words in written text, but are also limited by fixed input sizes. however, obtaining contextual relations between words from the sequence texts plays a crucial role in understanding the complete meaning of sentences. with the popularity of data-driven techniques, deep learning based methods improve the shortcomings of traditional methods and achieve superior emo performance @xcite @xcite . more recently, the transformer self-attention architecture based @xcite pre-trained models have been successfully applied for learning language representations by exploiting large amounts of unlabeled data. these models mainly include bert @xcite , openai gpt @xcite , roberta @xcite . these architectures show superior performance when fine-tuning different downstream tasks, including machine translation @xcite ), text classification @xcite , emotion classification @xcite and question answering @xcite . recent works have shown that transformer-based pre-trained methods can achieve state-of-the-art performance in emo tasks @xcite . motivated by this, we adopt the deberta model @xcite with continual pre-training method for the masked language model (mlm) @xcite in this track 2 to improve final downstream performance. more feasible training strategies are designed to improve the final results further. in this paper, we describe our work for track 2 of the wassa shared task 2022, addressing the issue of emotion classification. ## method in this section, we will elaborate on the main methods for track 2 of the wassa 2022 shared task. more details about the training strategies are detailed at the end of this section. ## experimental setting this section will subsequently present emotion dataset, our experimental models, experimental settings, control of variables experiment. ## results and discussions the experiment results of the various methods on the evaluation dataset are displayed in the results of top-5 teams participating in the emo track for the post-evaluation are shown in as for the ablation study part, we implement different ablation settings to show the effectiveness of the proposed method. as shown in ## conclusion this paper illustrates our contributions to the wassa shared work on emotion classification. we use the deberta pre-trained language model enhanced by the continual pre-training method (mlm) and some training strategies to improve the emo performance. during the evaluation phase, our submission achieves top-1 on all metrics for the emotion classification task. in the future, we will explore more efficient pre-training methods to