UECA -Prompt: Universal Prompt for Emotion Cause Analysis (2022)

Abstract: Emotion cause analysis (ECA) aims to extract emotion clauses and find the corresponding cause of the emotion. Existing methods adopt fine-tuning paradigm to solve certain types of ECA tasks. These task-specific methods have a deficiency of universality. And the relations among multiple objectives in one task are not explicitly modeled. Moreover, the relative position information introduced in most existing methods may make the model suffer from dataset bias. To address the first two problems, this paper proposes a universal prompt tuning method to solve different ECA tasks in the unified framework. As for the third problem, this paper designs a directional constraint module and a sequential learning module to ease the bias. Considering the commonalities among different tasks, this paper proposes a cross-task training method to further explore the capability of the model. The experimental results show that our method achieves competitive performance on the ECA datasets.

## introduction recently, emotion cause analysis (eca) has obtained increasing attention. as a classic task of eca, emotion cause extraction (ece) was first proposed and defined as a clause-level classification problem @xcite . ece aims to explore the potential causes behind a certain emotional expression in a clause. however, the applications of ece are limited in real-world scenarios because the emotion must be annotated before cause extraction. therefore, emotion-cause pair extraction (ecpe) was proposed to identify all emotions and their corresponding causes from an unannotated text at the same time @xcite 2021b; @xcite to expand the applications of eca, this paper focuses on the most complex and challenging three clause-level sub-tasks: ece, ecpe, and ccrc. in literature, eca tasks have been widely studied. on the ece task, @xcite released a benchmark chinese emotion cause dataset and proposed a multi-kernel based method. several statistical learning @xcite and deep learning methods @xcite have been applied to ece, which show competitive performance on emotion cause prediction. on the ecpe task, @xcite proposed a two-step method to extract emotions and corresponding causes at the same time. this method first individually extracts the emotion set and cause set. then, it gets the emotion-cause pairs by applying a cartesian product to these two sets and trains a filter to remove the invalid pairs. however, the error may propagate from the first procedure to the second because of the inherent drawback of the pipelined framework. to address this issue, several works adopted end-to-end architecture. part of these works @xcite focused on the multi-task learning @xcite of the ecpe task with the joint learning framework @xcite . some other works represented the relation between emotion and cause with graph construction @xcite . in addition, transition-based parsing @xcite and unified sequence labeling @xcite are also employed. on the ccrc task, the relationship between emotion and cause clauses in different contexts is studied @xcite . the existing methods could be divided into multi-stage framework @xcite and end-to-end framework @xcite . both frameworks adopt the fine-tuning paradigm. as shown in fig. 1 , these fine-tuning methods firstly obtain clause-level features from the pre-trained embedding or pre-trained language models (plms). next, a contextual encoder is designed to yield contextual representations of clauses. then, the contextual representation is merged with clause-level position information. finally, the output is used for task prediction. in general, the existing methods have three obvious shortcomings: firstly, the contextual encoder has the deficiency of universality because they are designed for specific task objectives. secondly, the existing methods implicitly learn the relationships among multiple task objectives, rather than explicitly model their relations. thirdly, position information would make the model sensitive to data distribution and lack robustness. prompt @xcite @xcite is a new paradigm that can be traced back to @xcite . this paradigm could overcome the difficulty of uniformly solving various tasks by transforming specific fine-tuning tasks into the form of pre-training tasks. the universality of prompt has been widely validated in various tasks of natural language processing @xcite @xcite , especially in few-shot @xcite or zero-shot @xcite scenarios. multiple predictions should be performed for one sample with several clauses in eca. and eca is composed based on multiple task objectives (e.g. emotion extraction, cause extraction, emotioncause pair extraction). appropriate prompt construction could complete multiple predictions simultaneously. constructing special prompts could explicitly model the relations among multiple objectives in one task. in general, prompt could make up for the shortcomings of existing methods. inspired by this, this paper proposes a universal prompt-based method for eca tasks (ueca-prompt). ueca-prompt first modifies each task objective into a sub-prompt after decomposing eca tasks into multiple task objectives. then, it explicitly models the relations among different task objectives by combining different sub-prompts into a composite prompt. as most of the works @xcite , ueca-prompt is manually constructed to solve different eca tasks in a unified framework. as far as we know, ueca-prompt is the first attempt at multitask multiple predictions with prompt. some previous works @xcite artificially introduced position information into the model. such characteristic would make the model extremely sensitive to the distribution of positions of cause clauses relative to their corresponding emotion clauses in the dataset. specifically, existing methods with position information may not generalize well to de-bias dataset @xcite . because the cause clause of most samples in the de-bias dataset is not in proximity to the emotion clause. this paper designs a directional constraint module and a sequential learning module to better identify the emotion-cause pairs. these two modules could ease the bias by discarding position information between clauses. there are commonalities among different tasks. for example, ccrc focuses more on context information, and context information is also crucial for ece and ecpe. this paper proposes a crosstask training method. the model would be able to learn commonalities among tasks with the crosstask training method. we evaluate our method on three benchmark chinese emotion cause datasets. the experimental results show that our method can obtain better results than state-of-the-art methods on three eca tasks solely based on the bert. our contributions are summarized as follows: yesterday morning a policeman visited the old man with the lost money and told him that the thief was caught the old man was very happy and deposited the money in the bank [mask]e [mask]e [mask]e [mask]e [mask]e emotion emotion emotion emotion emotion clause clause clause clause clause [mask]ca [mask]ca [mask]ca [mask]ca [mask]ca cause cause cause cause cause clause clause clause clause clause [mask]s [mask]s [mask]s [mask]s [mask]s corresponds corresponds corresponds corresponds corresponds to to to to to [mask]d [mask]d [mask]d [mask]d [mask]d sample prompt c i : i-th clause of the document 𝐷 prompt function 𝑓 # (%) for sequential learning indicator function 𝑓 ' (%) for emotion indicator function 𝑓 () (%) for cause prompt function 𝑓 * (%) for directional constraint mlm head label words c 5 c 4 c 3 c 2 c 1 #5 #4 #3 #2 #1 others mlm head mlm head mlm head 𝑓 # (%) c 5 c 4 c 3 c 2 c 1 label words is isn't 𝑓'(%) c 5 c 4 c 3 c 2 c 1 label words is isn't 𝑓 () (%) label words c 5 c 4 c 3 c 2 c 1 #4 none others 𝑓 * (%) document d c 1 yesterday morning, c 2 a policeman visited the old man with the lost money, c 3 and told him that the thief was caught. c 4 the old man was very happy, c 5 and deposited the money in the bank. emotion clause: c 4 cause clause: c 2 , c 3 pairs: [(c 4 , c 2 ), (c 4 , c 3 )] ## method ueca-prompt is a bert-based method constructed as the form of the mlm task. this paper would discuss ueca-prompt from two perspectives, sub-prompt for task decomposition and composite prompt for multiple predictions. ## cross-task training method empirically, there are commonalities among different eca tasks. these commonalities would further improve the the model performance. this paper proposes a cross-task training method to make the model better adapt knowledge from one domain to another. the following section will introduce this training method through an example. the initial pre-trained model is defined as a. the first step, the model a is trained in the ece task to obtain the trained model b. second step, the trained model b is used to train for the ecpe task to obtain the trained model c. finally, model c is used to measure the performance of the ueca-prompt on ecpe. any two tasks of eca could perform the training steps mentioned above. the commonality among tasks could be learned by the model in the process of cross-task training. ## experiments this paper conducts experiments on the ece, ecpe, and ccrc tasks to evaluate our approach. we release our code at https://github.com/yajus/ueca-prompt . ## case study to further understand the operation principle of ueca-prompt, fig. 4 visualizes the attention for different tokens in the constructed prompt text of a chinese text. the subgraph (a), (b), and (c) in fig. 4 represent the attention weights in ecpe, ece, and ccrc, respectively. the query token is marked with the red box. it can be observed that the attention weights are mainly concentrated in the answer slot [m] and the context in emotion and cause clauses. this indicates that different sub-prompt modules in ueca-prompt could capture the key information of emotion and cause clauses as well as cooperate. ## conclusions this paper proposes a universal prompt method for emotion cause analysis tasks. ueca-prompt could uniformly model different eca tasks by decomposing eca tasks into multiple objectives and converting these objectives into sub-prompts. meanwhile, the proposed directional constraint module and sequential learning module could effectively ease the bias caused by position information. moreover, the cross-task training method further improves the performance of ueca-prompt. the ability of ueca-prompt to learn commonalities and contextual knowledge from different tasks is verified. the experimental results on three eca tasks demonstrate the effectiveness of the proposed method. this work chooses the general model, bert, for a fair comparison. the sentiment-related plm, such as skep @xcite , may further improve the performance of ueca-prompt, which will be explored in our future work.