YNU - HPCC at WASSA -2023 Shared Task 1: Large-scale Language Model with L o RA Fine-Tuning for Empathy Detection and Emotion Classification (2023)

Abstract: This paper describes the system for the YNU-HPCC team in WASSA-2023 Shared Task 1: Empathy Detection and Emotion Classification. This task needs to predict the empathy, emotion, and personality of the empathic reactions. This system is mainly based on the Decoding-enhanced BERT with disentangled attention (DeBERTa) model with parameter-efficient fine-tuning (PEFT) and the Robustly Optimized BERT Pretraining Approach (RoBERTa). Low-Rank Adaptation (LoRA) fine-tuning in PEFT is used to reduce the training parameters of large language models. Moreover, back translation is introduced to augment the training dataset. This system achieved relatively good results on the competition’s official leaderboard. The code of this system is available here.

## introduction the purpose of wassa-2023 shared task 1 @xcite is to use empathic reaction data to predict hidden sentiment and personality. this task consisted of five tracks: although the prediction goals are different, all five tracks can be considered as either a sentiment classification @xcite or regression task @xcite . one of the biggest challenges in this task lies in how to learn representation for the given text. the early exploration was based on text similarity @xcite or text alignment @xcite . with the development of neural networks, convolutional neural networks (cnn) @xcite and recurrent neural networks (rnn) @xcite and their variants are adopted to learn text representations. both cnn and rnn are shallow models, which only incorporate previous knowledge in the first layer of the model. the models are also based on word embeddings that are useful in only capturing the semantic meaning of words without understanding higher-level concepts like anaphora, long-term dependencies, and many more. beyond word embeddings, recent studies proposed embedding from language models (elmo), which can learn word embeddings by incorporating both word-level characteristics as well as contextual semantics @xcite . this also led to the emergence of pre-trained models (plm) using transformers as basic units. the plms, such as bert @xcite , roberta @xcite , albert @xcite , and deberta @xcite , are first fed a large amount of unannotated data, allowing the model to learn the usage of various words and how the language is written in general. then, they can be finetuned to be transferred to a natural language processing (nlp) task where it is fed another smaller taskspecific dataset. as the scale of plms increases, the model performance in downstream tasks becomes better and better. nevertheless, the fine-tuning procedure brings about increased requirements for model training costs. for example, the large sequence-to-sequence model gpt-3 has 175b parameters @xcite . to reduce training costs, recent studies suggest using parameterefficient fine-tuning (peft) @xcite to enable the efficient adaption of plms to downstream applications without fine-tuning all the parameters of the plms. to this end, this paper proposes to use deberta fine-tuned with low-rank adaptation (lora) @xcite in peft and roberta for all tracks in this competition. both the deberta and roberta were initialized from a well-trained checkpoint, e.g., deberta-v2-xxlarge with 1.5b parameters and roberta-base with 125m parameters. for finetuning, lora only fine-tuned a small number of (extra) model parameters, thereby greatly decreasing the computational and storage costs. for classification tasks, a softmax head with the crossentropy loss was applied, while a linear decoder head with the mean squared error was adopted for regression tasks. the experimental results on the development dataset show that the xxl version of deberta with lora and back translation achieves the best performance in tracks 1, 3, and 5. although the number of trainable parameters decreases, the model achieves performance comparable to that of full fine-tuning. additionally, roberta with back translation achieved the best performance in tracks 2 and 4. the difference in the performance of the two models on different tracks may be due to the impact of the size of the training dataset. the rest of this paper is organized as follows. section 2 describes the system model and method. section 3 discusses the specific experimental results. conclusions are finally drawn in section 4. ## system description the architecture of the proposed model is shown in figure 1 . the given text of conversations or essays is input into the tokenizer and then segmented into the corresponding token id. subsequently, de-berta or roberta's encoder is used to extract the features of the text in a vector format. meanwhile, lora is used to reduce fine-tuning parameters without degrading performance too much. finally, the encoded hidden representation is used for both sentiment classification and regression. ## experimental results this section evaluates the performance of the proposed system for both sentiment classification and regression tasks. ## conclusion this paper proposed a system submitted in shared task 1 of wassa-2023, which uses roberta and xxl version of deberta as the pre-trained models and fine-tuning the deberta model using lora. the experimental results indicate that this system has achieved good performance. in addition, this system has a lot of space for improvement compared to the top-ranked systems. future works will attempt to try other text augmentation and generation methods to achieve better results.