Efficient Cross-Task Prompt Tuning for Few-Shot Conversational Emotion Recognition (2023)

Abstract: Emotion Recognition in Conversation (ERC) has been widely studied due to its importance in developing emotion-aware empathetic machines. The rise of pre-trained language models (PLMs) has further pushed the limit of ERC performance. However, most recent works on ERC using PLMs are heavily data-driven, and requires fine-tuning the entire PLMs. To improve both sample and computational efficiency, we propose a derivative-free optimization method called Cross-Task Prompt Tuning (CTPT) for few-shot conversational emotion recognition. Unlike existing methods that learn independent knowledge from individual tasks, CTPT leverages sharable cross-task knowledge by exploiting external knowledge from other source tasks to improve learning performance under the few-shot setting. Moreover, CTPT only needs to optimize a vector under the low intrinsic dimensionality without gradient, which is highly parameter-efficient compared with existing approaches. Experiments on five different contextual conversation datasets demonstrate that our CTPT method has superior results on both few-shot scenarios and zero-shot transfers.

## introduction emotion recognition in conversation (erc) detects emotion categories (e.g., netural, happiness, sadness) of each utterance in a given textual conversation. as pre-trained language models (plms) @xcite have brought a huge breakthrough to natural language processing (nlp), plms are also increasingly employed by erc models as encoders to improve recognition performance @xcite @xcite . however, the performance gain from plms is often achieved at the exorbitant cost of expensive training and fine-tuning processes. plm-based erc models tend to suffer from poor sample efficiency and computational efficiency as they often involve a large number of training examples and millions of trainable parameters, which potentially prevents current plm-based models from achieving their best performance in low-resource scenarios. few-shot learning techniques @xcite hold the promise to improve both sample and computation efficiency for deploying plms in new scenarios where data can be limited. recently, prompt tuning @xcite , which trains a set of discrete or continuous prompt embeddings conditioned on a frozen plm, has shown promising results in few-shot learning settings @xcite @xcite . the prompt can be regarded as a way to retrieve the knowledge already memorized in the plm. the effectiveness of prompts lies in their capability to adapt to new tasks while preserving the knowledge embedded in plms, without causing overfitting issues that can arise from full-model fine-tuning @xcite . however, most recent works on erc are largescale data-driven that focus on the full dataset setting @xcite . @xcite firstly explore the few-shot erc task, but their setting is not strictly few-shot, which may lead to a variety of examples for each label. for example, their training set contains more than k examples for each label under the k-shot setting. to this end, we strictly define the erc task under the few-shot setting and propose a cross-task prompt tuning (ctpt) solution. existing prompt tuning methods independently learn task-specific knowledge from each task, yet such knowledge is often very limited in the few-shot setting. our proposed ctpt leverages cross-task knowledge by exploiting external knowledge from other source tasks to improve learning performance under the few-shot setting. the cross-task knowledge from other source tasks can be divided into two parts: external task-specific knowledge and emotional knowledge. for external task-specific knowledge, we utilize a multi-head attention module @xcite to learn knowledge from source tasks. for emotional knowledge, we combine the same emotion within different textual categories from different tasks and then reformulate the verbalizer that decodes the output to the label distribution. one limitation of prompt tuning is that it involves backpropagating the loss through all the transformer layers of a plm for every batch even though we freeze the plm, which can lead to computational inefficiency. to further improve the computational efficiency of plm-based erc models, we optimize a vector with intrinsic dimensionality @xcite instead of the whole continuous prompt, which reduces the number of parameters from hundreds of thousands to about 1,000. following @xcite , we use a covariance matrix adaptation evolution strategy (cma-es) @xcite to optimize the parameters, which is derivative-free. with the derivative-free optimization, we separate our approach from the plm and do not require backpropagation for parameter learning. compared with single-task prompt tuning, our proposed ctpt method can utilize external knowledge from other tasks to boost the performance of the target task. experiments under the fewshot scenarios and the zero-shot transfer show that ctpt can obtain a better result. in addition to this, our proposed ctpt is derivative-free, which does not need backpropagation. compared with derivative-based backpropagation, the experiment result shows that ctpt can obtain comparable results without derivative information. the main contributions of this paper are summarized as follows: ## conclusion in this paper, we strictly define the task of the few-shot setting for erc and propose a cross-task prompt tuning (ctpt) method to tackle this problem utilizing the cross-task knowledge. ctpt learns from external task-specific knowledge from other tasks and emotional knowledge from commonsense and then summarizes the learned crosstask knowledge to improve the validation performance. we use a derivative-free optimization method to optimize the parameters without gradient information, which skips the backpropagation stage. experiments on erc benchmarks show that ctpt can outperform baseline models in the fewshot setting and obtain a surprising result in the zero-shot transfer. in summary, ctpt is trainingefficient that includes: (1) sample-efficiency: it is trained by few-shot training examples, and (2) computational-efficiency: it tunes only about 1,000 parameters with derivative-free algorithms that skip the backpropagation.