Zero-shot Script Parsing (2022)

Abstract: Script knowledge is useful to a variety of NLP tasks. However, existing resources only cover a small number of activities, limiting its practical usefulness. In this work, we propose a zero-shot learning approach to script parsing , the task of tagging texts with scenario-specific event and participant types, which enables us to acquire script knowledge without domain-specific annotations. We (1) learn representations of potential event and participant mentions by promoting cluster consistency according to the annotated data; (2) perform clustering on the event / participant candidates from unannotated texts that belongs to an unseen scenario. The model achieves 68.1/74.4 average F1 for event / participant parsing, respectively, outperforming a previous CRF model that, in contrast, has access to scenario-specific supervision. We also evaluate the model by testing on a different corpus, where it achieved 55.5/54.0 average F1 for event / participant parsing.

## introduction script knowledge is a type of commonsense knowledge that captures how people conduct everyday activities @xcite . it expresses that in a certain scenario, participants tend to act out events in a certain order; an example from the scenario fixing a flat tire is shown in fig. 1 . humans use script knowledge to fill in events that are not explicitly mentioned in a text, and script knowledge is useful for many downstream nlp applications, including referent prediction @xcite , discourse classification @xcite , and story generation @xcite . a key challenge in dealing with script knowledge is coverage: it is costly and time-consuming to spell out the prototypical events and participants of a scenario and how they can be expressed in language. existing resources are mostly crowdscript parsing identifies events and participants from texts. the picture is taken from @xcite . sourced @xcite ; they annotate stories from a limited number of scenarios with script events and participants (cf. figure 1 ). script parsers, which predict these event and participant labels given a text, can achieve high accuracies on scenarios that were seen in training @xcite . nonetheless, these parsers only operate on known scenarios: they could only predict event types on the same scenario as they were trained. the limited coverage restricts the practical usefulness of script parsers and thus the practical usefulness of script knowledge for downstream tasks in general. in this paper, we acquire script knowledge by tackling the task of zero-shot script parsing: we present the first system which accurately performs script parsing on scenarios that were not seen at training time. for instance, given training data that talks about taking a bath and going to a restaurant, the parser labels events and participants in the fixing a flat tire story of fig. 1 , allowing us to arrange script knowledge into a graphical form as fig. 2 . this offers a way of overcoming the coverage limitations of script knowledge, by generalizing from the training scenarios to arbitrary other ones. our method learns to extract script-specific representations from general-purpose pretrained word embeddings, and then uses agglomerative clustering at inference time to group together naturallanguage phrases that refer to the same event or participant of the unseen script. finally, we evaluate our model on mcscript, a different corpus than the training data. our model achieves a micro-f1 score on zeroshot event labeling of up to 68.1 and on participant labeling of up to 74.4, on par with the supervised model of @xcite that assumes training data for the same scenario. we find that our method yields script graphs with reasonable event clusters that are temporally ordered in a reasonable way; the majority of errors on event labeling are due to issues with the granularity of events. we also find in probing tasks that our model learns to amplify information about sentence ordering from the pretrained embeddings, while suppressing lowlevel information about morphology and syntax, which are less relevant for the script parsing task. in order to investigate its potential for practical use, we also evaluated the model on a different, unannotated corpus, where the model achieved 55.5/54.0 average f1 for event / participant parsing. ## related work scripts were introduced as an approach to capturing commonsense knowledge in ai by @xcite ; see also @xcite . much research in nlp has simplified the acquisition of script knowledge to identifying "event chains" in narrative text @xcite . event chains represent typical sequences of events, each represented by one predicate, and can be learned from large corpora. other work has followed in this tradition @xcite @xcite . in this paper, we instead build upon work by @xcite , who explicitly capture script knowledge about a given scenario in a temporal script graph (see fig. 2 ). a tsg specifies the abstract events and participants that make up a script with their temporal ordering; each of these events and participants can be expressed in language in many different ways. a tsg is also more expressive than mere event sequences, because it encodes different manners how a scenario play out in real life. regneri et al. learned script graphs by crowdsourcing. we instead rely on manually script-annotated corpora @xcite . trained with scenario-specific supervision, script parsing can be performed accurately. @xcite developed a linear crf model to perform script parsing as a sequence labelling task. @xcite developed a hierarchical model for supervised script parsing, making use of pre-trained contextualized word embeddings. the model learns patterns at the word level, as well as the narrative level with respective sequence models. these existing approaches are limited to scenarios for which training data is available, whereas our work focuses on unseen scenarios. zero-shot learning is a family of methods that establish a classifier for unseen classes, based on labelled data from seen classes. one common approach is to learn a latent representation space that all instances embed into, thus the knowledge of the source domain, encoded in the labelled training instances, could be transferred to the target domain. it tackles data scarcity in various situations, such as machine translation for low-resource languages (e.g. @xcite @xcite , generation @xcite ), text classification (see, e.g. @xcite and question answering (e.g. @xcite . ## data we work with inscript @xcite and mcscript @xcite . ## evaluation on inscript, we evaluate our method with 10-fold cross-validation where each fold is the data associated with a scenario. note that the texts in the validation and test data are always from scenarios that were unseen in training. on mcscript, the model is evaluated on the mcscript test set. ## conclusion we have presented the first approach to script parsing without scenario-specific knowledge. we do this by clustering specialized word representations which are trained by optimizing cluster consistency; the model is further improved by the use of coreference and event-participant dependency information. the model greatly outperforms a baseline with general-purpose word embeddings, and performs on par with an earlier supervised model. our model thus makes it possible, for the first time, to label large quantities of unannotated data with script information. in future work, we plan to experiment also with corpora that contain a larger variety of naturally occurring texts than mcscript, in which the sentences are relatively simple, pertinent to the scenario and are often in the correct temporal order.