Token Sequence Labeling vs. Clause Classification for E nglish Emotion Stimulus Detection (2020)

Abstract: Emotion stimulus detection is the task of finding the cause of an emotion in a textual description, similar to target or aspect detection for sentiment analysis. Previous work approached this in three ways, namely (1) as text classification into an inventory of predefined possible stimuli (“Is the stimulus category A or B?”), (2) as sequence labeling of tokens (“Which tokens describe the stimulus?”), and (3) as clause classification (“Does this clause contain the emotion stimulus?”). So far, setting (3) has been evaluated broadly on Mandarin and (2) on English, but no comparison has been performed. Therefore, we analyze whether clause classification or token sequence labeling is better suited for emotion stimulus detection in English. We propose an integrated framework which enables us to evaluate the two different approaches comparably, implement models inspired by state-of-the-art approaches in Mandarin, and test them on four English data sets from different domains. Our results show that token sequence labeling is superior on three out of four datasets, in both clause-based and token sequence-based evaluation. The only case in which clause classification performs better is one data set with a high density of clause annotations. Our error analysis further confirms quantitatively and qualitatively that clauses are not the appropriate stimulus unit in English.

## introduction research in emotion analysis from text focuses on classification, i.e., mapping sentences or documents to emotion categories based on psychological theories (e.g., @xcite , @xcite ). while this task answers the question which emotion is expressed in a text, it does not detect the textual unit, which reveals why the emotion has been developed. for instance, in the example "paul is angry because he lost his wallet." it remains hidden that lost his wallet is the reason for experiencing the emotion of anger. this stimulus, e.g., an event description, a person, a state of affairs, or an object enables deeper insight, similar to targeted or aspectbased sentiment analysis @xcite @xcite @xcite . this situation is dissatisfying for (at least) two reasons. first, detecting the emotions expressed in social media and their stimuli might play a role in understanding why different social groups change their attitude towards specific events and could help recognize specific issues in society. second, understanding the relationship between stimuli and emotions is also compelling from a psychological point of view, given that emotions are commonly considered responses to relevant situations @xcite . models which tackle the task of detecting the stimulus in a text have seen three different problem formulations in the past: (1) classification into a predefined inventory of possible stimuli @xcite , similarly to previous work in sentiment analysis @xcite , (2) classification of precalculated or annotated clauses as containing a stimulus or not @xcite , and (3) detecting the tokens that describe the stimulus, e.g., with iob labels @xcite . we follow the two settings in which the stimuli are not predefined categories (2+3, cf. figure 1 ). these two settings have their advantages and disadvantages. the clause classification setting is more coarse-grained and, therefore, more likely to perform well than the token sequence labeling setting, but it might miss the exact starting and endpoints of a stimulus span and needs clause annotations or a syntactic parse with the risk of error t stim. map det. (t, l) i (c, y) j (t, l) i (c, y) j comparable output clause detection cc sl @xmath0: framework for emotion stimulus detection. tokens t are split into clauses for clause class. mapping ensures that both methods result in clause classifications (t, l) i and token sequences with labels (c, y) j . these two different formulations are naturally evaluated in two different ways and have not been compared before, to the best of our knowledge. therefore, it remains unclear which task formulation is more appropriate for english. further, the most recent approaches have been evaluated only on mandarin chinese, with the only exception being the emotioncauseanalysis dataset being considered by @xcite , but not in comparison to token sequence labeling. no other english emotion stimulus data sets have been tackled with clause classification methods. we hypothesize that clauses are not appropriate units for english, as @xcite already noted that: "such granularity [is] too large to be considered an emotion stimulus in english". a similar argument has been brought up during the development of semantic role labeling methods: @xcite stated that "argument[s] may span over different parts of a sentence". our contributions are as follows: (1) we develop an integrated framework that represents different formulations for the emotion stimulus detection task and evaluate these on four available english datasets; (2) as part of this framework, we propose a clause detector for english which is required to perform stimulus detection via clause classification in a real-world setting; (3) show that token sequence labeling is indeed the preferred approach for stimulus detection in most available english datasets; (4) show in an error analysis that this is mostly because clauses are not the appropriate unit for stimuli in english. finally, (5), we make our implementation and annotations for both clauses and tokens available at http://www.ims.uni-stuttgart.de/ data/emotion-stimulus-detection . the remainder of the paper is organized as follows. we first introduce our integrated framework of stimulus detection which enables us to evaluate clause classification and token sequence labeling in a comparable manner (section 2). we then turn to the experiments (section 3) in which we analyze results on four different english data sets. section 4 discusses typical errors in detail, which leads to a better understanding of how stimuli are formulated in english. we conclude in section 6. ## an integrated framework for stimulus detection the two approaches for open-domain stimulus detection, namely, clause classification and token sequence labeling, have not been compared on english. we propose an integrated framework (figure 2 ) which takes tokens t as input, splits this sequence into clauses and classifies them (clause detection can be bypassed if manual annotations of clauses are available). the token sequence labeling does not rely on clause annotations. the output, either clauses c with classifications y (@xmath1, no} n ) or tokens t with labels l are then mapped to each other to enable a comparative evaluation. we explain these steps in the following subsections. ## experiments and results we now put the models to use to understand the differences between sequence labeling and clause classification for english emotion stimulus detection and the suitability of clauses as the unit of analysis. ## error analysis in the following, we analyze the error types made by the different models on all data sets and investigate in which ways sl improves over the icc and jcc models. we hypothesize that the higher flexibility of token-based sequence labeling leads to different types of errors than the clause-based classification models. for quantitative analysis, we define different error types, illustrated in err example model data set steve talked to me a lot about being abandoned and the pain that caused. jcc eca no what i told about the way they treated you and me made him angry. sl eca fuck mitt romney and fuck barack obama ... god got me !!!!! icc et maurice mitchell wants you to do more than vote. icc gne and he started to despair that his exploration was going to be entirely unsuccessful ... icc eca deeply ashamed of my wayward notions , i tried my best to contradict myself. icc es anyone else find it weird i get excited about stuff like the rnc tonight ?! # polisciprobs sl et doesn't he do it well said the girl following with admiring eyes, every movement of him. jcc eca if he feared that some terrible secret might evaporate from them , it was a mania with him. sl eca i was furious because the mac xl wasn't real said hoffman . sl eca with such obvious delight in food, it 's hard to see how blanc remains so slim. sl es triad thugs use clubs to punish hong kong ' s protesters . jcc gne i'm glad to see you so happy lupin icc es figure 6: examples for error types for different models and data sets. extracted clauses are separate by . the relaxed evaluation settings. do icc and jcc particularly miss starting or end points of the stimulus annotation? we see in the results and error types differ across data sets (see particularly , , ). this points out what we have seen in the evaluation already: the structure of a stimulus depends on the domain and annotation. the least challenging data set is emotionstimulus with the lowest numbers of errors across all models. this result is caused by most sentences having similar syntactic trees, all stimuli are explicit and mostly introduced in a similar way. for qualitative analyses, figure 6 shows one example of each type of error described above. in the first example, the jcc model does not learn to include the second part of the coordination -"and the pain". in the second example, similarly, the sl model misses the right part of the coordination. for most cases of independent clauses that we inspect, we see a common pattern for both types of models, which is that the prediction stops while encountering coordinating conjunctions. in the sixth example, the prediction span includes the emotion cue. this issue could be solved by doing sequence labeling instead or by informing the model of the presence of other semantic roles. these examples raise the following question: would improved clause segmentation lead to improvements for the clause-classification models across all data sets? ## related work the task of detecting the stimulus of an expressed emotion in text received relatively little attention. next to the corpora we mentioned so far, the reman corpus @xcite consists of english excerpts from literature, sampled from project gutenberg. the authors consider triples of sentences as a trade-of between longer passages and sentences. further, @xcite annotated english sentences on the token level. besides english and mandarin, @xcite developed a method for the identification of italian sentences that contain an emotion cause phrase. @xcite annotate japanese sentences on newspaper articles, web news articles, and q&a sites. all recently published state-of-the-art methods for the task of emotion stimulus detection via clause classification are evaluated on the mandarin data by @xcite . they include multi-kernel learning @xcite and long short-term memory networks (lstm) @xcite . @xcite propose a convolutional multipleslot deep memory network (convms-memnet), and @xcite a co-attention neural network model, which encodes the clauses with a coattention based bi-directional long short-term memory into high-level input representations, which are further passed into a convolutional layer. @xcite proposed an architecture with components for "position augmented embedding" and "dynamic global label" which takes the relative position of the stimuli to the emotion keywords and use the predictions of previous clauses as features for predicting subsequent clauses. @xcite integrate the relative position of stimuli and evaluate a transformer-based model that classifies all clauses jointly within a text. similarly, @xcite proposes a word-phrase-clause hierarchical network. the transformer-based model achieves state of the art, however, it is shown that the rnn based encoders are very close in performance @xcite . therefore, we use a comparable model that is grounded on the same concept of a hierarchical setup with lstms as encoders. further, there is a strand of research which jointly predicts the clause that contains the emotion stimulus together with its emotion cue @xcite . however, the comparability of methods across data sets has been limited in previous work, as ## conclusion we contributed to emotion stimulus detection in two ways. firstly, we evaluated emotion stimulus detection across several english annotated data sets. secondly, we analyzed if the current standard formulation for stimulus detection on mandarin chinese is also a good choice for english. we find that the domain and annotation of the data sets have a large impact on the performance. the worst performance of the token sequence labeling approach is obtained on the crowdsourced data set electoraltweets. the well-formed sentences of emotionstimulus pose fewer difficulties to our models than tweets and headlines. we see that the sequence labeling approaches are more appropriate for the phenomenon of stimulus mentions in english. this shows in the evaluation of the comparably coarse-grained clause level and is also backed by our error analysis. for future work, we propose closer investigation of whether other smaller constituents might represent the stimulus better for english and a check of whether the strong results for the sequence labeling hold for other languages. notably, the clause classification setup has its benefits, and this might lead to a promising setting as joint modeling or as a filtering step to finding parts of the text which might contain a stimulus mention. another step is to investigate if the emotion stimulus and the emotion category classification benefit from joint modeling in english as it has been shown for mandarin @xcite .