An End-to-End Network for Emotion-Cause Pair Extraction (2021)

Abstract: The task of Emotion-Cause Pair Extraction (ECPE) aims to extract all potential clause-pairs of emotions and their corresponding causes in a document. Unlike the more well-studied task of Emotion Cause Extraction (ECE), ECPE does not require the emotion clauses to be provided as annotations. Previous works on ECPE have either followed a multi-stage approach where emotion extraction, cause extraction, and pairing are done independently or use complex architectures to resolve its limitations. In this paper, we propose an end-to-end model for the ECPE task. Due to the unavailability of an English language ECPE corpus, we adapt the NTCIR-13 ECE corpus and establish a baseline for the ECPE task on this dataset. On this dataset, the proposed method produces significant performance improvements (∼ 6.5% increase in F1 score) over the multi-stage approach and achieves comparable performance to the state-of-the-art methods.

## introduction there have been several works on emotions prediction from the text @xcite as well as generating emotion oriented texts @xcite @xcite . however, recently the focus has also shifted to finding out the underlying cause(s) that lead to the emotion expressed in the text. in this respect, @xcite proposed the emotion cause extraction (ece), a task aimed at detecting the cause behind a given emotion annotation. the task is defined as a clause level classification problem. the text is divided at the clause level and the task is to detect the clause containing the cause, given the clause containing the emotion. however, the applicability of models solving the ece problem is limited by the fact that emotion annotations are required at test time. more recently, @xcite introduced the emotion-cause pair extraction (ecpe) task i.e. extracting all possible emotion-cause clause pairs in a document with no emotion annotations. thus, ecpe opens up avenues for applications of real-time sentiment-cause analysis in tweets and product reviews. ecpe builds on the existing and well studied ece task. figure 1 shows an example with ground truth annotations. @xcite use a two-stage architecture to extract potential emotion-cause clauses. in stage 1, the model extracts a set of emotion clauses and a set of cause clauses (not mutually exclusive) from the document. in stage 2, it performs emotioncause pairing and filtering, i.e. eliminating pairs that the model predicts as an invalid emotion-cause pair. however, this fails to fully capture the mutual dependence between emotion and cause clauses since clause extraction happens in isolation from the pairing step. thus, the model is never optimized using the overall task as the objective. also, certain emotion clauses are likely not to be detected without the corresponding cause clauses as the context for that emotion. recent methods such as @xcite and @xcite use complex encoder and classifier architectures to resolve these limitations of the multi-stage method. in this paper, we propose an end-to-end model to explicitly demonstrate the effectiveness of joint training on the ecpe task. the proposed model attempts to take into account the mutual interdependence between emotion and cause clauses. based on the benchmark english-language corpus used in the ece task of the ntcir-13 workshop @xcite , we evaluate our approach on this dataset after adapting it for the ecpe task. we demonstrate that the proposed approach works sig-clause 1: adele arrived at her apartment late in the afternoon after a long day of work. clause 2: she was still furious with her husband for not remembering her 40th birthday. clause 3: as soon as she unlocked the door, she gasped with surprise; clause 4: mikhael and harriet had organized a huge party for her. figure 1 : an example document. the above example contains two emotion-cause pairs. clause 2 is an emotion clause (furious) and is also the corresponding cause clause (for not remembering her 40th birthday). clause 3 is an emotion clause (surprise) and clause 4 is its corresponding cause clause (organized a huge party for her). ## related work the problem of emotion cause extraction (ece) has been studied extensively over the past decade. ece was initially proposed as a word-level sequence detection problem in @xcite . attempts to solve this task focused on either classical machine learning techniques @xcite , or on rule-based methods @xcite . subsequently, the problem was reframed as a clause-level classification problem @xcite and the chinese-language dataset introduced by @xcite has since become the benchmark dataset for ece and the task has been an active area of research @xcite @xcite @xcite . however, the main limitation of ece remains that it requires emotion annotations even during test time, which severely limits the applicability of ece models. to address this, @xcite introduced a new task called emotion-cause pair extraction (ecpe), which extracts both emotion and its cause without requiring the emotion annotation. they demonstrated the results of their two-stage architecture on the benchmark chinese language ece corpus @xcite . following their work, several works have been proposed to address the limitations of the two-stage architecture ( ding et al. (2020a), ding et al. (2020b), fan et al. (2020), yuan et al. (2020), cheng et al. (2020), chen et al. (2020) ) . in order to explore the corpus further and to encourage future work from a broader commu-1 https://github.com/aaditya-singh/ e2e-ecpe nity, we use an english language ece corpus. (see section 5). ## task definition formally, a document consists of text that is segmented into an ordered set of clauses @xmath0, c 2 , ..., c d ] and the ecpe task aims to extract a set of emotion-cause pairs @xmath1..., (c i , c j ), ...} (c i , c @xmath2, where c i is an emotion clause and c j is the corresponding cause clause. in the ece task, we are additionally given the annotations of emotion clauses and the goal is to detect (one or more) clauses containing the cause for each emotion clause. ## approach we propose an end-to-end emotion cause pairs extraction model (figure 2 ), henceforth referred to as e2e-pext e (refer to section 6 for the naming convention). the model takes an entire document as its input and computes, for each ordered pair of clauses (c i , c j ), the probability of being a potential emotion-cause pair. to facilitate the learning of suitable clause representations required for this primary task, we train the model on two other auxiliary tasks: emotion detection and cause detection. we propose a hierarchical architecture. word level representations are used to obtain clausal representations (vi bilstm) and clause level representations are further contextualized using another bilstm network. the resulting contextualized clause representations are then used for the classification task. let w j i represent the vector representation of the j th word in the i th clause. each clause c i in the document d is passed through a word-level encoder (bilstm + attention @xcite ) to obtain the clause representation s i . the clause embeddings are then fed into two separate clause-level encoders (emotion-encoder and cause-encoder) each of which corresponds, respectively, to the two auxiliary tasks. the purpose of the clause-level encoders is to help learn for the clause c i , its contextual representations r e i and r c i are then used to predict whether the clause is an emotion-clause and a cause-clause respectively, i.e., as observed by @xcite , we also noticed that performance on the two auxiliary tasks could be improved if done in an interactive manner rather than independently. hence, the cause-encoder also makes use of the corresponding emotion-detection prediction y e i , when generating r c i (figure 2 ). for the primary task, every ordered pair (c i , c j ) is represented by concatenating r e i , r c j and pe ij , wherein pe ij is the positional embedding vector representing the relative positioning between the two clauses i, j in the document @xcite . the primary task is solved by passing this pairrepresentation through a fully-connected neural network to get the pair-predictions. to train the end-to-end model, loss function is set as the weighted sum of loss on the primary task as well as the two auxiliary tasks i.e. ## corpus/data description we adapt an existing emotion-cause extraction (ece) @xcite corpus for evaluating our proposed models (as well as the architectures proposed in previous work). the corpus was introduced in the ntcir-13 workshop @xcite for the ece challenge. the corpus consists of 2843 documents taken from several english novels. each document is annotated with the following information: i) emotioncause pairs present in the document, that is, the set of emotion clauses and their corresponding cause clauses; ii) emotion category of each clause; and iii) the keyword within the clause denoting the labeled emotion. we do not use the emotion category or the keyword during training of either ece or ecpe tasks only the emotion-cause pairs are used. at test time, none of the annotations are used for the ecpe task. for ece task, emotion annotation is provided at test time and the model predicts the corresponding cause clauses. 80%-10%-10% splits are used for training, validation and testing. 10 such randomly generated splits are used to get statistically significant results, and the average results are reported. for the purpose of evaluation on our dataset, we reproduced the two-stage model: ecpe 2-stage @xcite . we also adapted two models that achieve state of the art performance on the ecpe task: ecpe-2d(bert) @xcite and ecpe-mll(isml-6) @xcite and compared them against our model: e2e-pext e . the model is trained for 15 epochs using adam optimizer @xcite . the learning rate and batch size were set to 0.005 and 32 respectively. model weights and biases were initialized by sampling from a uniform distribution u(-0.10, 0.10). glove word embeddings @xcite of 200 dimension are used. for regularization, we set the dropout rate to 0.8 for word embeddings and l2 weight decay of 1e-5 over softmax parameters. we set λ c : λ e : λ @xmath3: 1 : 2.5. the values chosen through grid search on the hyperparameter space reflect the higher importance of the primary pair detection task compared to the auxiliary tasks. to obtain better positional embeddings which encode the relative positioning between clauses, we trained randomly initialized embeddings after setting the clipping distance @xcite to 10 all clauses that have a distance of 10 or more between them have the same positional embedding. we use the same evaluation metrics (precision, recall and f1-score), as used in the past work on ece and ecpe tasks. following @xcite , the metric definitions are defined as: the results are summarized in ## conclusion in this paper, we demonstrated that a simple endto-end model can achieve competitive performance on the ecpe task by leveraging the inherent correlation between emotions and their causes and optimizing directly on the overall objective. we also showed that a variant of our model which further uses emotion annotations, outperforms the previously best performing model on the ece task, thereby showing its applicability to variety of related tasks involving emotion analysis. in future, we plan on developing a larger benchmark englishlanguage dataset for the ecpe task. we plan to explore other model-architectures which we expect will help us learn richer representations for causality detection in clause-pairs.