Towards Layered Events and Schema Representations in Long Documents (2021)

Abstract: In this thesis proposal, we explore the application of event extraction to literary texts. Considering the lengths of literary documents modeling events in different granularities may be more adequate to extract meaningful information, as individual elements contribute little to the overall semantics. We adapt the concept of schemas as sequences of events all describing a single process, connected through shared participants extending it to for multiple schemas in a document. Segmentation of event sequences into schemas is approached by modeling event sequences, on such task as the narrative cloze task, the prediction of missing events in sequences. We propose building on sequences of event embeddings to form schema embeddings, thereby summarizing sections of documents using a single representation. This approach will allow for the comparisons of different sections of documents and entire literary works. Literature is a challenging domain based on its variety of genres, yet the representation of literary content has received relatively little attention.

## introduction events generally describe any change of state @xcite and are often used in information extraction scenarios @xcite . the modeling of sequences of events has the potential of aiding literary scientists in understanding narrative patterns and devices. determining which events in a narrative are crucial is challenging and relates to a variety of related tasks, such as summarization, comparison, or even story generation. understanding the contexts of an event requires modeling its arguments and semantics. a simple representation can be the subject and object relating to a given verb, in conjunction with the verb's lemma @xcite . if one only wants to include events involving a single character in a story, it is necessary to consider only those predicates with arguments coreferring to the character. the narrative coherence assumption says that "verbs sharing coreferring arguments are semantically connected by virtue of narrative discourse structure" @xcite . verbs connected in this way are, under the assumption, considered to be part of the same so-called narrative chain @xcite . previous work has focused on finding chains as representations of narratives in short documents, combining individual narrative chains, each focused on one character, into a schema involving multiple chains and thereby multiple characters @xcite . while the overall narrative in a long document could be regarded as a large schema, a variety of sub-schemas exists describing each scene using individual events. as a result, a typical document in our domain contains multiple schemas. figure 1 illustrates a potential separation of an event sequence into schemas. for each event e c n in any given text we know, based on coreference resolution, which entities c are involved with it (i.e.: occur as its arguments). intuitively a separation boundary is preferably found between nonconnected events. the verbs "leaving" and "arriving", for example, are strongly connected events; we expect them to often appear in sequence. after modeling the likelihood of different events occurring in sequence, we can calculate the model's perplexity with regard to a specific event and use this information for the separation of chains. even in our simple example (fig. 1 ) it is not clear where exactly to place separations, e 7 could, for example, form a social gathering schema with e 5 and e 6 instead of a separate transportation schema. e {b} 0 bob enters the store. e {b} 1 he picks up some milk. e {b} 2 after he pays for it he e {b} 3 leaves in his car. e {b} 4 bob arrives home. e {a,b} 5 alice visits him. e {a,b} 6 they watch a movie. e {a} 7 alice drives home. figure 1: one possible separation of the events into four schemas splits the event up into a shopping, a transportation, a social gathering, and another transportation schema. ## research questions generally, the proposed thesis seeks to model broader narratives by building up from single events. we aim to build two-layered models, building from events to schemas by segmenting chains of events into semantically related subchains. those sub-chains sharing coreferring arguments form what we call a schema. over a simple sequence model of events, this has the potential benefits of allowing for human analysis and simplifying comparisons between multiple texts. rq 1: how can events be represented? we approach the detection of events by processing verb occurrences. we aim to make use of dense vector representations of frames instead of using discrete frames. this is motivated by coverage concerns as well as the intuitive insight that frames have vary-ing semantic distances between each other, which we hope can be represented by vector space distances. the approach will be evaluated on existing semantic frame resources as well as regarding their contribution towards schemas. rq 2: how can schemas be represented? through the use of sequence models, we will attempt to find semantically related sequences of events. this may mean finding common-sense event sequences. for example "take cart" -"take fruit" -"queue up" -"pay" clearly is a sequence of events typical for grocery shopping, even though no individual event is uniquely indicative of grocery shopping. in this way, we may find semantic structures in texts that only emerge from the combination of several events. we will experiment with different approaches to transforming sequences of events into schema representations. a simple approach may be averaging of event representations; more advanced approaches involving neural sequence models are also to be explored. rq 3: which role does coreference play in schema representations? coreference allows us to resolve the arguments of frames to their entities. predicates that share corefering arguments may, depending on the segmentation, be part of the same schema. entities will be chosen based on their prevalence, only entities with multiple occurrences are of interest. as a result, all predicates not involved with entities of interest are discarded immediately; this is an implied filtering step removing many predicates that do not constitute events. descriptions of scenery for example would usually be discarded in such a scenario. the evaluation of coreference resolution can be performed on existing datasets. literary datasets generally only annotate characters, rather than all entities. it is conceivable that representation learning on events in text order may, in our case, be an appropriate replacement for coreference resolution. in this case, the presence of multiple events in proximity would be modeled rather than an explicit interaction. initial filtering of non-event predicates is, in this case, required to not include predicates irrelevant to the story at large. rq 4: how can event and schema representations be adapted to literary works. we hypothesize to encounter the following challenges in our approach to literary works: document length, vocabulary mismatch with pre-trained models, and a diversity of domains (i.e.: different literary genres). to address these, we will explore the role of segmentation for processing documents in sections, the viability of incremental processing, and the role of pre-training and unsupervised fine-tuning. aside from intrinsic evaluations of schemas based on their similarity and predicates based on them constituting events, we plan to derive summaries from the schema structure and compare them to human-generated summaries in literary lexicons (e.g. arnold, 2009). ## methodology from the research questions, two immediate directions emerge: event extraction, including coreference, and event representations. later in the research process, we plan to build two-layer models transforming sequences of events into schemas. ## conclusion we proposed segmenting chains of events to form multiple schemas in long documents, mentioning different approaches to the representation of events and to their segmentation. further, we discussed the options for representing schemas to allow for their analysis and thereby the comparison of different documents. an open question for us is if the two-layer approach to schemas and events is sufficient, if needed a hierarchical approach involving levels of schemas will be considered. as part of the event extraction process in this thesis, work on both semantic frame induction and coreference resolution for german language content will be advanced. the representation of events using continuous frame embeddings is a new approach in the domain of information extraction. specifics of sequence modeling and feature learning on events are vague, iterations on the proposed concepts are planned. the open question of how exactly schemas boundaries are to be defined still needs to be explored. we intend to help enable the computational analysis of literary texts. schema representations may be used for finding previously hard to find similarities in different documents, whereas event features can be used to identify events that are important to the narrative. statistical and machine-learningbased approaches to event modeling will advance the understanding of events in a domain that yet received relatively little attention.