Automatic extraction of personal events from dialogue (2020)

Abstract: In this paper we introduce the problem of extracting events from dialogue. Previous work on event extraction focused on newswire, however we are interested in extracting events from spoken dialogue. To ground this study, we annotated dialogue transcripts from fourteen episodes of the podcast This American Life. This corpus contains 1,038 utterances, made up of 16,962 tokens, of which 3,664 represent events. The agreement for this corpus has a Cohen’s Kappa of 0.83. We have open-sourced this corpus for the NLP community. With this corpus in hand, we trained support vector machines (SVM) to correctly classify these phenomena with 0.68 F1, when using episode-fold cross-validation. This is nearly 100% higher F1 than the baseline classifier. The SVM models achieved performance of over 0.75 F1 on some testing folds. We report the results for SVM classifiers trained with four different types of features (verb classes, part of speech tags, named entities, and semantic role labels), and different machine learning protocols (under-sampling and trigram context). This work is grounded in narratology and computational models of narrative. It is useful for extracting events, plot, and story content from spoken dialogue.

## motivation people communicate using stories. a simple definition of story is a series of events arranged over time. a typical story has at least one plot and at least one character. when people speak to one another, we tell stories and reference events using unique discourse. the purpose of this research is to gain better understanding of the events people reference when they speak, effectively enabling further knowledge of how people tell stories and communicate. there has been no work, to our knowledge, about event extraction from transcripts of spoken language. the most popular corpora annotated for events all come from the domain of newswire @xcite . our work begins to fill that gap. we have open sourced the gold-standard annotated corpus of events from dialogue. 1 for brevity, we will hearby refer to this corpus as the personal events in dialogue corpus (pedc). we detailed the feature extraction pipelines, and the support vector machine (svm) learning protocols for the automatic extraction of events from dialogue. using this information, as well as the corpus we have released, anyone interested in extracting events from dialogue can proceed where we have left off. one may ask: why is it important to annotate a corpus of dialogue for events? it is necessary because dialogue is distinct from other types of discourse. we claim that spoken dialogue, as a type of discourse, is especially different than newswire. we justify this claim by evaluating the distribution of narrative point of view (pov) and diegesis in the pedc and a common newswire corpus. pov distinguishes whether a narrator tells a story in a personal or impersonal manner, and diegesis is whether the narrator is involved in the events of the story they tell. we use pov and diegesis to make our comparisions because they give information about the narrator, and their relationship to the story they tell. we back our claim (that dialogue is different than newswire) by comparing the distributions of narrative point of view (pov) and diegesis of the narrators in pedc with the reuters-21578 newswire corpus. 2 @xcite found that narrators in newswire texts from the reuters-21,578 corpus use the first-person pov less than 1% of the time, and are homodiegetic less than 1% of the time. however, in the 14 episodes (1,028 utterances) of this american life, we found that 56% narrators were first-person, and 32% narrators were homodiegetic. we found these distributions in pedc by using the automatic pov and diegesis extractors from @xcite , which were open sourced. 3 comparing the distributions of pov and diegesis for the pedc to that of newswire demonstrates how different spoken dialogue is. this shows why building an annotated corpus specifically for event extraction of dialogue was necessary. it is substantial that so many of the utterances in the pedc are first-person narrators and homodiegetic. this means that people are speaking about their lives. they are retelling stories. they are speaking in more personal ways than narrators do in newswire. this is where the personal in the personal events in dialogue corpus comes from. additionally, using the modifier personal aligns this work with @xcite who extracted personal stories from blog posts. we want our work to help researchers studying computation models of narrative. ## personal events in dialogue annotation study when beginning to think about extracting events from dialogue, we realized there is no corpus of transcribed dialogue annotated for events. there are many corpora of other text types with event annotations. timebank contains newswire text @xcite . meantime is made up of wikinews articles @xcite . additionally there are many event annotation schema; one of the more prominent ones is timeml @xcite . we decided to develop our own annotation scheme due to the complexity of timeml; it's an extremly fine-grained annotation scheme, with specific tags for different types of events, temporal expressions and links. we decided it would be too difficult to use timeml while maintaining a high inter-annotator agreement and finishing the annotation study in a short amount of time (three months), and within a modest budget. given that our goal was to understand spoken conversational dialogue, we decided to create a corpus from transcribed audio. this matches the nature of the data we intend to use for our event exctractor: audio recordings of dialogue that have been transcribed as a text file. we weighed a number of different sources for the text transcripts, but we ultimately decided to use transcripts from the podcast this american life foot_2 . we chose this podcast because: 1) the transcripts are freely available online. 2) a significant portion of these podcasts are made up of conversations, as opposed to narration. additionally, this american life formats their transcripts so that the conversations are indented as block quotations. this made it easy to separate conversations from typical podcast narration. 3) the subject matter of this american life are typically stories from people's lives. we wanted our corpus to be made up of unscripted conversations; contemporary everyday conversations, so that the extractors we train from this data are better suited to understanding people talking about their lives. ## results we report our results in we will compare the results in ## discussion our best performing event extractor uses pos and verb class features, and the ml protocols used were undersampling and trigrams, however, the performance is not significantly better than the extractors that only use either one of the two protocols. our best peforming event extractor with no extra ml protocols was the extractor with pos, ner, and verb class features. the performance of the extractor that had all four features had the same performance as the former, so we can say that the addition of srl features adds no extra information to the classification process. it is interesting to see the affect of undersampling on performance. it boosted the event f1 for most feature sets. not only did it boost the f1, but it flipped the values of precision and recall with respect to the original experiment. without undersampling, the precision is always higher than the recall. once undersampling is toggled, the recall becomes larger than the precision. also, the undersampled recall is typically higher than the non-undersampled precision. this flippage is important to note for situations when the event extractor is actually used in real-world systems. if the situation requires a minimal number of false positives, than precision should be maximized, therefore no undersampling should be used when training the model. however, if minimal false negatives is a bigger priority, then recall should be maximized, hence undersampling should be used in training. whether undersampling is used, or not, depends on the actual context the event extractor is being deployed. in general, undersampling helped boost performance of event classification in most experiments. trigrams gave an even smaller boost to event classification in most experiments. experiments that had both undersampling and trigrams had the largest boost when compared to the experiment with no extra ml protocols. there were two feature sets that trigram context had a significant affect, both pos + srl and pos + verb + srl. these are the only experiments where the trigram context protocol led to the greatest performance for the feature set, and by a significant margin. overall, trigrams had a much smaller affect on overall performance. we hypothesize that there are better ways to implement this form of context. either a classifier that's better suited for sequential data should be used, or a different form of encoding the context feature should be explored. another note about a negative result: the impact of the srl features was much less influential than we hypothesized. going forward, we think that the actual semantic frames instantiated should be used as features, as well as different frame elements, and not just occurence of predicates and arguments. ## contributions in this paper we presented two sets of contributions: first, we have open sourced the first corpus of dialogue annotated for events. 10 this corpus can be used by researchers interested in the automatic understanding of dialogue, specifically dialogue that is rich with the personal stories of people. second, we share the design and evaluate the performance of 57 unique event classifiers for dialogue. these results can be used by researchers to decide which features and machine learning protocols should be implemented for their own event extractors. our best performing extractor has a 0.68 f1, which is over 100% higher than baseline. we hope that this work can be used by the community to better understand how people reference events from stories in dialogue.