Affective Event Classification with Discourse-enhanced Self-training (2020)

Abstract: Prior research has recognized the need to associate affective polarities with events and has produced several techniques and lexical resources for identifying affective events. Our research introduces new classification models to assign affective polarity to event phrases. First, we present a BERT-based model for affective event classification and show that the classifier achieves substantially better performance than a large affective event knowledge base. Second, we present a discourse-enhanced self-training method that iteratively improves the classifier with unlabeled data. The key idea is to exploit event phrases that occur with a coreferent sentiment expression. The discourse-enhanced self-training algorithm iteratively labels new event phrases based on both the classifier’s predictions and the polarities of the event’s coreferent sentiment expressions. Our results show that discourse-enhanced self-training further improves both recall and precision for affective event classification.

## introduction in recent years, researchers have been tackling the problem of identifying affective events, which are events that have a positive or negative effect on people who experience the event. for example, events that are typically positive include being hired for a new job, breaking a sports record, or buying a home. conversely, events that are typically negative include being fired from a job, breaking an arm, or having your house burn down. people's world knowledge about events and how they impact people is sufficient for humans to infer the affective state of someone who experiences such an event, even if that person does not explicitly express an emotion. consequently, we will refer to these events as having positive or negative polarity with respect to an implicit affective state. research has shown that recognizing affective events is important for a variety of natural language processing tasks, including narrative text comprehension and summarization @xcite , dialogue systems @xcite , response generation @xcite , and sarcasm detection @xcite . much of the prior work on recognizing affective events has focused on producing lexical resources of verbs or event phrases with corresponding affective polarity values @xcite @xcite @xcite . these resources reflect substantial progress toward recognizing affective events in text, but their coverage is limited by their fixed content. we hypothesized that deep learning architectures that encode rich meaning representations could lead to a more effective approach for identifying affective events. specifically, neural classification models have the capacity to generalize across lexically and syntactically different phrases that are semantically similar, and similar events are usually associated with the same affective polarity. to explore this approach, we created a bert-based model for affective event classification and show that it recognizes affective events more effectively than a large affective event knowledge base. our research also introduces a discourseenhanced self-training method that further improves affective event classification with unlabeled data. self-training is a well-known method for using a classifier's own predictions on unlabeled instances to generate more training data. however, self-training has limitations. using the most confident labels may not improve recall much because the new training instances are familiar, while using less confident labels often decreases precision because the training data becomes more noisy. to overcome these issues, we designed a discourseenhanced self-training method that combines the classifier's predictions with information from local discourse contexts to robustly assign labels to new training instances. the key to this approach is to exploit unlabeled event phrases that occur near coreferent sentiment expressions. specifically, we extract event phrases that are followed by a sentiment expression in a syntactic structure that suggests it likely refers to the event. for example, consider the statement "i got engaged today. it is exciting.". "it" refers to the act of getting engaged, so the positive sentiment applies to that event. our algorithm then predicts the affective polarity for unlabeled events using both the classifier's prediction for the event phrase as well as the associated sentiment expressions. we show that our discourse-enhanced self-training method improves both recall and precision for affective event classification. ## related work several lines of research have focused on the problem of recognizing events that have implicit affective states. research on narrative understanding used bootstrapped learning to identify patient polarity verbs, which impart affective polarity to their patients @xcite . @xcite extracted "emotion-provoking events" using the seed pattern "i @xmath0, pattern expansion, and clustering. @xcite learned lexico-syntactic patterns associated with first-person affect to improve affective sentence classification alongside supervised learners. @xcite extracted "major life events" from twitter by clustering tweets that occurred with speech act words, such as "congratulations" or "condolences". but their work did not assign affective polarity to events, and focused only on major life events that prompt expressive speech acts. our work has a broader scope, aiming to recognize everyday events as well (e.g., being hungry is negative, and seeing a rainbow is positive). work in opinion analysis created a +/-effectwordnet @xcite to recognize the effects of events on entities, although the effects are not necessarily "affective" because the entities need not be animate (e.g., baking a cake has a positive effect on the cake because it is created). subsequent work developed implicature rules to use +/-effects for opinion analysis @xcite . there has also been work on recognizing the connotation of words and senses @xcite and connotation frames @xcite , which infer connotative polarities for a verb's arguments from the writer's and entity's perspective. these efforts associated polarity with individual verbs, not event phrases. @xcite used discourse relations to propagate affective polarity from seeds using a japanese web corpus. they extracted events that co-occur with seeds in a large corpus, then used discourse relations as constraints in the learning process. another line of related work is emotion cause extraction, which links emotion expressions to the events that caused the emotion @xcite @xcite @xcite . this research uses datasets created from chinese news and microblogs that contain an explicitly mentioned emotion. this work assigns polarity to events in the context of a specific text passage. in contrast, our work aims to identify the prior affective polarity of an event, irrespective of context. consequently, our classifier can be used to predict the affective polarity of events in contexts that do not contain any explicit emotion or sentiment indicators. our research is most closely related to the work by @xcite , which identifies stereotypically affective events and their prior polarity, irrespective of context. the affective event knowledge base (aekb) produced by @xcite ) contains over half a million event phrases coupled with polarity labels. these events were extracted from nearly 1.4 million personal blog posts in the icwsm 2009 and 2011 spinn3r datasets foot_0 . the polarity labels were generated automatically using a weakly supervised method. their approach optimizes for semantic consistency over a graph of event nodes that are linked by edges capturing three types of semantic relations. our discourse-enhanced self-training algorithm adds a new twist to traditional self-training methods @xcite @xcite . the approach is also reminiscent of co-training @xcite , which trains two classifiers based on independent views of the data. however in co-training, each classifier must be able to make reliable predictions on its own. we do not expect the coreferent sentiment expressions used by our approach to be sufficient by themselves because they are quite noisy (e.g., due to imperfect coreference, imperfect sentiment ## harvesting events with coreferent sentiment expressions the key idea behind our approach is to create a self-training method that uses not only the classifier's own predictions but also a secondary source of information derived from local discourse contexts. intuitively, the secondary signal confirms the classifier's prediction when they agree, or creates doubt about the classifier's prediction when they disagree. by taking both signals into account, we can assign high-quality labels to a diverse set of new examples in each cycle, which creates a robust self-training process. from this point on, we turn our attention to twitter because it is a vast resource that we can query to acquire a large set of event phrases in specific contexts, and where people share their everyday experiences. we acquire our unlabeled data by searching for event phrases on twitter that occur with coreferent sentiment expressions. we use a heuristic to identify sentiment expressions that are likely to refer to an event in the preceding sentence. specifically, we look for sentiment expressions that begin a sentence and match one of the following forms: (a) {this/that/it/i}, {be/feel/seem}, {adj+} (b) {this/that/it}, {be/feel/seem}, {adj* n+} where the head adjective (adj) or head noun (n) is a sentiment term with positive or negative polarity. the sentiment expression cannot be followed by any events in its sentence and must follow a sentence that contains at least one event. given these restrictions, the pronouns "this', "that", and "it" are likely referring to an event in the previous sentence, although this is not guaranteed. similarly, the pronoun "i" is referring to the speaker who is likely expressing their sentiment toward something that was just mentioned, which is often (though not always) the prior event. we will call the phrases that match these patterns coreferent sentiment expressions because they express a sentiment that refers back to something mentioned earlier. we found that the syntactic constructions above typically convey a sentiment about an event in the tweet1: i rode a horse today! that was fun. i, ride, horse, -tweet2: someone was abducted on the street right next to mine. it's terrifying. tweet3: disrupting my daily routine and alienating many people. i am angry ! -, disrupt, my daily routine, --, alienate, people, -, - ## discourse-enhanced self-training we designed an enhanced self-training algorithm that learns from unlabeled data by iteratively labeling new instances using both the affective event classifier's prediction as well as polarities associated with the event's discourse contexts. we will refer to this method as discourse-enhanced selftraining. the intuition is that (1) new instances are labeled only if both sources of information agree, which yields high-quality labels, and (2) a more diverse set of instances will be labeled than if only the classifier's most confident predictions were used. figure 1 illustrates how an unlabeled event is scored during discourse-enhanced self-training. each event is paired with the set of coreferent sentiment expressions that occurred with it in our twitter dataset. the affective event classifier is applied to the event and generates a probability distribution over the three polarity values. in parallel, an external sentiment classifier produces a probability distribution over the polarity classes for each of the coreferent sentiment expressions. the probability distributions are then averaged to produce an average probability distribution for the set of sentiment expressions as a whole. finally, a joint scoring function takes the two probability distributions and produces a joint probability distribution for the event. the polarity with the highest probability is used as the event's label. algorithm 1 outlines our discourse-enhanced self-training procedure in detail. the process begins with a gold labeled set of events e l , a set of unlabeled events e u where each event e i in e u is paired with a set of coreferent sentiment expressions cse i , an external sentiment classifier, and two confidence thresholds θ jnt and θ neu . each iteration starts by training the event classifier on e l . the event classifier is then applied to every unlabeled event e i in e u to produce an event score vector s e i . next, the sentiment classifier is applied to every coreferent sentiment expression cse in cse i to produce a polarity distribution. then the polarity distributions of all cse in cse i are averaged to produce an average polarity distribution scse i for the whole set cse i . the joint scoring function then produces a joint score vector s jnt i for the event e i by the equation below: we generate a set of new labeled events e jnt by assigning labels to unlabeled events that have a polarity @xmath1. all other events remain unlabeled. however, we found that this process labels relatively few events as neutral. since neutral events can also co-occur with positive and negative sentiment expressions, they may have relatively low neutral scores. to better maintain the distribution of events over all three polarities, we also add a new set of events e neu , which the event classifier predicts as neutral with @xmath2. discourse-enhanced self-training needs an external sentiment classifier, so we fined-tuned a bertbased model with the gold standard twitter dataset from semeval-2017 @xcite following the experiment setups in section 3.2 and section 3.3. in our experiments, we set θ neu to 0.9 and θ jnt to 0.95 based on the model's performance over the validation set. ## experimental results we performed 10-fold cross validation over the gold twitter dataset, where each of the 10 runs used 80% of the data (8 folds) for training, 10% of the data (1 fold) for validation/tuning, and 10% of the data (1 fold) for testing. we compare discourseenhanced self-training (dest) with strictly supervised learning and traditional self-training. during each iteration of the traditional self-training model, the affective event classifier aff-bert is applied to each unlabeled event. events with polarity @xmath3.9 are selected as new labeled data. we chose 0.9 as the threshold based on the model's performance on the validation set. for the dest model, to ensure a rich set of discourse contexts, we only used unlabeled events that (a) had at least 10 distinct coreferent sentiment expressions and (b) did not include "this", "that" or "it" as a subject or object of the event phrase because an event is often vague without knowing what the pronoun refers to. this resulted in 8,532 events in the unlabeled event set. ## analysis to better understand the behavior of the resulting classifier, we did a manual analysis of events whose polarity was impacted by the coreferent sentiment expressions. the top portion of the bottom portion of we also found some events that were correctly labeled as positive by the affective event classifier but labeled as negative by the sentiment classifier with high confidence, and consequently the event classifier's correct predictions were overridden. most of these cases were expressions of love or empathy in response to negative events, such as god, help, us, -, my heart, go, -, to family , you, have, my sympathy, -. this is an interesting phenomenon that may require better discourse modeling, including the recognition of expressive speech acts. ## conclusion in this work, we proposed a bert-based supervised classifier for affective event recognition and showed that it substantially outperforms a large affective event knowledge base. we also designed a novel discourse-enhanced self-training algorithm to leverage unlabeled data iteratively. by combining both the affective event classifier's prediction and the polarities of coreferent sentiment expressions, our algorithm substantially improved upon the supervised learning results. the resulting classification model is substantially more effective for affective event recognition than previous methods. we also believe that the general idea behind our discourse-enhanced self-training approach could be useful for many other types of problems where additional information can be extracted from larger contexts to serve as a secondary signal to help confirm or disconfirm a classifier's predictions.