I n F illmore: Frame-Guided Language Generation with Bidirectional Context (2021)

Abstract: We propose a structured extension to bidirectional-context conditional language generation, or “infilling,” inspired by Frame Semantic theory. Guidance is provided through one of two approaches: (1) model fine-tuning, conditioning directly on observed symbolic frames, and (2) a novel extension to disjunctive lexically constrained decoding that leverages frame semantic lexical units. Automatic and human evaluations confirm that frame-guided generation allows for explicit manipulation of intended infill semantics, with minimal loss in distinguishability from human-generated text. Our methods flexibly apply to a variety of use scenarios, and we provide an interactive web demo.

## introduction a popular strategy for automatic story generation is to proceed in a coarse-to-fine manner: first by proposing a story plan, and then realizing it into natural language form using large pretrained neural language models @xcite . in this work, we study the use of framenet frames @xcite as representational units for such plan guidance. in frame semantics @xcite , words evoke structural situation types (frames) that describe the common schematic relationships between lexical items. we hypothesize that these structured types can be used to effectively induce the semantic content of text generated by increasingly powerful pretrained language models, yielding a flexible, controllable and domaingeneral model for surface realization of story plans with a variety of dimensions for user guidance. based on this supposition, we fine-tune a recent infilling model @xcite with a frameguided denoising objective. we contrast this approach with a novel method for frame-guided generation that modifies only the decoding step of a standard language model through lexical manipulation. the idea originates from the annotation scheme of framenet, where each semantic frame is annotated with a set of evocative lexical units (lus). we posit that it is possible to guide the model's generation with frames without modifying its training procedure by instead lexically constraining its generation output to contain frame-associated lus. therefore, we develop an extension to lexically-constrained decoding that leverages lus as ordered disjunctive constraint sets. given a possibly multi-frame sequence and a generative model, our method en-forces the generation of one of the associated lus for each frame in the sequence. this decoding method is implemented as a plug-and-play module that can be imposed on top of any standard generative language model. 2 we evaluate through a sentence-infilling task based on rocstories @xcite , assessing performance on two dimensions: 1) the quality of generation, as measured through perplexity and human evaluation; and 2) the fidelity, which scores whether generated text evokes the frames used as guidance. we demonstrate that our methods utilize guidance to generate frame-evoking surface realizations without meaningfully detracting from the contextual narrative coherence. we also demonstrate the practical applicability of frame-guided generation in a variety of example use cases. constrained generation separate lines of work employ lexical constraints to achieve the same goal of guided and diverse generation. as such, lexically constrained beam search methods such as grid beam search @xcite and dynamic beam allocation @xcite were proposed as the decoding methods for causal generation with disjunctive positive constraints @xcite , paraphrasing @xcite , machine translation @xcite , and abstractive summarization @xcite . @xcite generalize beam-search based methods with an algorithm that supports lexical constraints in the conjunctive normal form. parallel are the approaches that handle lexical constraints in an editing manner: starting with a sequence of keyword constraints and fleshing out a 2 fairseq-based implementation and data to be released. ## data framenet framenet is a lexical database of english based on fillmore's theory of frame semantics. it defines more than 1200 frames spanning various semantic domains, where each frame schematically describes a type of event, relation, or entity. a frame is defined with a set of corresponding frame elements (fes): the participants in the frame with relational roles, and a set of lexical units (lus): words that evoke the frame in text. for example, the apply_heat frame that describes the concept of cooking consists of core fes food, cook, container, heating_instrument, and temperature_setting, and has evocative lus that include fry, bake, boil, and broil. frame annotations provide a partial (albeit rich) picture of sentence meaning, i.e. information not governed by the syntax/semantics interface. we find that they serve as an effective, theory-grounded formalism for discrete semantic guidance of generation. conceptually, our choice to use framenet as guiding semantics builds upon trends in generative modeling of discourse @xcite that treat text documents as mixtures of hi- story charles went shopping. he bought fruit. then he left. ilm charles went shopping. [blank] then he left. [sep] he bought fruit. s-ffl [sep] [food] he bought fruit. a-ffl [sep] [commerce_buy] [food] he bought fruit. erarchical latent variables in accordance with classical theories of frame semantics (e.g. @xcite ; fillmore (1976)). as described by @xcite , framenet frame information can be used to learn a hierarchical latent representation of sentence-level semantics that produces discourse models that better fit to natural text data. our work then asks whether this information can be used to harness the increasingly powerful ability of recent neural language models for the purposes of controlled story generation. rocstories @xcite introduce the rocstories corpus, which comprises over 98k 5-sentence simple stories that can serve as a resource for commonsense narrative schema learning and story generation @xcite . we use this dataset to evaluate the performance of our methods (described in section 5). ## experiments we test the effectiveness of our models on a frameguided sentence infilling task derived from roc-stories. we use a state-of-the-art neural framenet parser @xcite to obtain the set of frames evoked by each sentence in the dataset. we then present models with a five-sentence roc story with one masked out. the model must infill the missing sentence given one or many frame id tokens parsed from the masked-out sentence. for evaluations requiring generated outputs (all but perplexity), we use beam search with beam size 20. we find that beam search achieves higher frame fidelity and coherence than the random sampling approach used by @xcite . we train our models (all gpt-2 'base') using the provided train split of rocstories. for s/m/a-ffl, each example contains one/multiple/all frame id tokens sampled randomly from the parser output. to test lcd, we re-train the original ilm using the identical rocstories training data to our ffl models but without frame tokens (training details described in a.2). unlike donahue et al. (2020), we do not include story titles. we also use this ilm as a baseline with no guidance. to investigate whether enforcing generated frame order impacts model performance, we evaluate both lcd-ordered and -unordered; we also evaluate ffl-ordered models fine-tuned to generate frames in the order in which they are provided. ## analysis fidelity from the results in there is a considerable gap between the performance of our models and perfect performance (1.0). this is because ffl operates only with soft "control code" constraints, and although lcd is strictly required to generate trigger lus for every frame, it does not produce sentences that always successfully evoke the frame. while some of this gap might be the result of imperfections of the parser, we find word sense ambiguity to be a contributing problem. many lus, such as work.v, see.v, or call.v have multiple senses each associated with a different frame. since neither lcd nor ffl imposes hard constraints on word sense, it is entirely figure 4 : from the lexical constraints on the lus of the frame request (as asked.v in the gold infill), the decoder selects call.v, but in the generated context call becomes a surfaces realization of frame contacting. as illustrated in figure 4 , lcd forces picking the lu call.v for the target frame request, but given the subsequent output call my friend to tell her i was hurt, the call.v unit takes on a sense that triggers the incorrect frame contacting. perplexity ## case study: interactive generation to demonstrate the practical applicability of our frame-guided infilling methods, we qualitatively explore them in a variety of human-in-the-loop use cases based on recent work in text generation. in the following cases, we use models for both frame id inference and text infilling conditioned on surrounding context. for frame inference, we use the forward frame token probability of an unorderedframe m-ffl model trained as in section 3, with the modification that training examples have between 0 and 4 surrounding sentences as context. this allows for more flexibilty than a model trained only on complete 5-sentence stories. we modify the training data by taking a random contiguous slice of each 5-sentence example. figure 6 shows examples of each scenario. for infilling, we use ffl for a, b and d and lcd for c. a. iterative story refinement for a maximally free-form and extensible use case, we devise a scenario in the spirit of @xcite in which a user interfaces with a model to collaboratively construct an open-domain story given any combination of text and/or frames. over the course of a human-system dialog, the user can iteratively either choose for the model to predict new frames at specified locations in the context or select from candidate infills conditioned on selected frames. as discussed in @xcite , this type of process allows for a symbiotic relationship in which the user can correct, suggest or revise content generated by the machine and vice versa. injecting frame guidance into this scenario enables for an extra degree of interactive flexibility in both suggestion and specification. ## conclusion we propose the application of frame semantics in the context of controlled text generation. we in-troduce two extensions of neural text generation that leverage framenet frames as guiding signals: our results show that our methods enable explicit manipulation of semantics at the frame level with competitive generation quality, and we exhibit a variety of use cases that enable new dimensions of user guidance on generation. to improve the lexical and semantic diversity in triggering frames, we construct disjunctive sets on a more fine-grained semantic level. we divide each set of lus into k subsets using hierarchical clustering over the glove embeddings of lus @xcite . in particular, we use the agglomerativeclustering class of scikitlearn foot_4 to perform hierarchical clustering over the glove embedding of lus to divide each set of frame-associate lus into subsets. in the experiments, we set number of clusters to 8. for multiframe constraints, we set number of clusters to 4 for the frame with the most number of lus and 2 for the frame with the second most of, we do not divide any lu sets for remaining frames (if any), this could ensure the total combination of multi-frame lu subsets equals 8. figure 7 shows the clustering results of three frames: collaboration, ingestion and departing, with number of clusters set to 4. to ensure that the decoder will be able to explore all possible combinations of lus, we build lists of tries for every combination of lu subsets. the constrained beam search is then run separately on each of them. to ensure that candidates from each lu subset are considered, final candidates are selected in a round-robin manner: the top-1 scored hypothesis is picked for each subset, followed by the top-2, and so on.