Mining Logical Event Schemas From Pre-Trained Language Models (2022)

Abstract: We present NESL (the Neuro-Episodic Schema Learner), an event schema learning system that combines large language models, FrameNet parsing, a powerful logical representation of language, and a set of simple behavioral schemas meant to bootstrap the learning process. In lieu of a pre-made corpus of stories, our dataset is a continuous feed of “situation samples” from a pre-trained language model, which are then parsed into FrameNet frames, mapped into simple behavioral schemas, and combined and generalized into complex, hierarchical schemas for a variety of everyday scenarios. We show that careful sampling from the language model can help emphasize stereotypical properties of situations and de-emphasize irrelevant details, and that the resulting schemas specify situations more comprehensively than those learned by other systems.

## introduction work on the task of event schema acquisition is largely separable into two main camps: the symbolic camp, with its structurally rich but infamously brittle representations @xcite @xcite ; and the statistical camp, which utilizes complex models and vast amounts of data to produce large numbers of conceptually varied event schemas at the cost of representational richness and control over what schemas are learned @xcite @xcite . in an attempt to bridge these camps together, we introduce the neuro-episodic schema learner (nesl), a composite pipeline model bringing together (1) a large, pre-trained language model; (2) word vector embedding techniques; (3) a neural framenet parsing and information extraction system; (4) a formal logical semantic representation of english; and (5) a hierarchical event schema framework with extraordinary expressive power. ## learning pipeline nesl learns schemas using a multi-step pipeline, briefly described in order here, and further expounded upon in following subsections: ## future evaluation this project is a work in progress; although nesl can generate one general schema every 10 minutes, and has generated several hundred to date, qualitative evaluation of the generated schemas has not yet been performed. imminently, we intend to carry out two human-judged studies: one evaluating the quality of inferences generated by the learned schemas when given unseen stories, and another evaluating the quality of the schemas themselves. logical schemas enable consistent, structured, and interpretable inferences about novel text, by matching pieces of the text to pieces of the schema, replacing schema variables with entities from the story, and treating other formulas in the schema that use those newly-filled variables as inferences. it is crucial that we demonstrate the inferential capacity of these learned schemas, and as future work, we will be sourcing suitable inference datasets, designing a means of presenting inferences to human judges, and collecting quality evaluations. in addition to inferences, we would like to evaluate whether the schemas we obtain are both topically cohesive, i.e., focused descriptions of one kind of situation; and interesting, i.e., capable of generating useful and novel inferences about situations, rather than obvious or redundant ones. using our gpt-2 verbalization model and schema rendering software, described in section 3.5, to make our schemas and inferences readable to untrained human judges, we intend to immediately move forward with the design and execution of these quality evaluation studies to complete the work. ## conclusion we have described nesl, a hybrid neural and formal-logical schema learning system with which we aim to combine a richly structured schema representation, a human learning-inspired approach to schema acquisition, the linguistically flexible sentence understanding characteristic of neural nlp systems, and the large amount of knowledge contained in large language models. by bringing a large and varied number of components from across the literature to bear, we have shown that general, semantically rich, and seemingly sensical schemas may be extracted from large language models and represented interpretably. in the immediate future, we also plan to demonstrate that these schemas are useful for inference tasks. a.3 from the prompt "mailing a letter" lea ed sc e a (? ne schema. ) ** ?e roles (?a e . ) (?b d c e . ) (?c e e . ) (?d e e e. ) (?f ca i . ) (?g a . ) (?j a e . )