Script Parsing with Hierarchical Sequence Modelling (2021)

Abstract: Scripts capture commonsense knowledge about everyday activities and their participants. Script knowledge proved useful in a number of NLP tasks, such as referent prediction, discourse classification, and story generation. A crucial step for the exploitation of script knowledge is script parsing, the task of tagging a text with the events and participants from a certain activity. This task is challenging: it requires information both about the ways events and participants are usually uttered in surface language as well as the order in which they occur in the world. We show how to do accurate script parsing with a hierarchical sequence model and transfer learning. Our model improves the state of the art of event parsing by over 16 points F-score and, for the first time, accurately tags script participants.

## introduction script knowledge is a category of common sense knowledge that describes how people conduct everyday activities sequentially @xcite . script knowledge of a specific scenario, e.g. grocery shopping, includes the events that comprise the scenario, the participants involved, and the relations between them. script knowledge is useful for various downstream nlp applications, such as referent prediction @xcite , discourse sense classification @xcite , story generation @xcite . script parsing identifies pre-defined sets of script events and participants from surface text (see figure 1 ). for a specific scenario, script parsing essentially boils down to determining what each verb and each np (which we term candidate) refers to in the context of that scenario. script parsing is an under-investigated, complex task. it is highly contextualized and corresponds to script parsing identifies events and participants from surface text. we contribute the following to the area of script parsing: methodology-wise, (1) we propose a hierarchical sequence model, which learns patterns in different granularity levels with different sequence models. (2) we investigate data augmentation approaches to this task. in terms of results, (3) we achieve accurate participant parsing results for the first time and (4) improve the state of the art in event parsing by over 16 points f1-score. ## related work theoretical considerations of scripts in ai (as described by @xcite were analyzed in a wide coverage empirical study by @xcite , who crowdsourced event descriptions of several everyday activities (scenarios). they applied unsupervised methods to compute a graph representation of the script's temporal structure. as a direct extension, @xcite and @xcite collected the inscript and descript corpora of event descriptions and manually annotated the scenario-specific types of events and participants to accommodate aligning surface text with data-driven script knowledge. the goal is to identify spans of the text that refer to the events and participants that are typically involved in a script. for the case of the script about fixing a bike, the typical events include riding a bike, noticing a flat tire, getting tools, repairing the tire and testing it. these events and participants are pre-defined for each activity and the task is to label the tokens with these abstract classes, whereby the surface forms vary. the model by @xcite is the state of the art for event parsing over inscript which was formulated as a sequence tagging task. the authors used a linear crf to identify the script events. its features include syntax, framenet @xcite features, pre-trained word embeddings and a number of script-related features encoding script-specific aspects like event order. our work shares many similarities with @xcite . to do question answering in the biological domain, they first build a graph representation of events, participants and their relations given a text about a biological process. token spans that denote an occurrence of an event are considered event triggers. however, unlike our approach, these events are not based on a pre-defined set. they separately train a model for identifying event triggers and a model for finding plausible argument candi-dates. the features used rely on syntax, semantic roles and some external domain resources. in contrast, we propose a model that jointly learns how to identify as well as label events and participants. much of the previous work on inferring script knowledge from text is focused around completing an event chain by predicting the missing event @xcite @xcite , the missing text @xcite or both @xcite . these approaches consider surface forms of event verbs and syntactic relation types of their arguments (subj, obj), while our task operates on abstract event and participant types. ## error analysis we manually classified the validation set errors made by our best-performing model, hierarchical bt , case by case. a breakdown is presented in noisy corpus labels. the corpus annotations of these instances are possibly incorrect. this is quite common, given the complexity (thus the moderate inter-annotator agreement) of the task. for example, in a story about borrowing a book from a library, ... i had to get a library card ... is a clear match for the event obtain card, as is predicted by our model; but in the corpus it was annotated as 'irregular', a mistake probably due to the light verb 'get' seemingly irrelevant to the scenario at first glance by the original annotator. false positives of irregular. a large proportion of errors feature a wrongly predicted irregular. we identified two main sources of such errors: (1) small class sizes; (2) instances that are particularly difficult because pragmatic inference is needed to make the right decision. as an example, ... get materials for the assignment ... corresponds to the event class evoking library, i.e. it evokes the scenario of borrowing a book from a li-brary without explicitly referring to a scenario event. however, without taking the situational context of the scenario into consideration, it cannot be inferred that 'get materials for the assignment' actually means 'borrow a book from a library'. wrong category. a small number of events are tagged as participants and vice versa, e.g. some homonyms of verbs and nouns (board or love). ## conclusion we present the first model that achieves high performance on both event and participant parsing. the model adopts a hierarchical design to model both the sequence of tokens and the sequence of script events and participants. further exploitation of domain adaptation and data augmentation methods yields a substantial performance boost. this work has established methods to accurately parse both script events and participants, in a supervised learning framework. our next step is approaching this complex task with less supervision, to lift the requirement on finely-annotated data, thus enabling wide-coverage script parsing.