Toward Diverse Precondition Generation (2021)

Abstract: A typical goal for language understanding is to logically connect the events of a discourse, but often connective events are not described due to their commonsense nature. In order to address this deficit, we focus here on generating precondition events. Precondition generation can be framed as a sequence-to-sequence problem: given a target event, generate a possible precondition. However, in most real-world scenarios, an event can have several preconditions, which is not always suitable for standard seq2seq frameworks. We propose DiP, the Diverse Precondition generation system that can generate unique and diverse preconditions. DiP consists of three stages of the generative process – an event sampler, a candidate generator, and a post-processor. The event sampler provides control codes (precondition triggers) which the candidate generator uses to focus its generation. Post-processing further improves the results through re-ranking and filtering. Unlike other conditional generation systems, DiP automatically generates control codes without training on diverse examples. Analysis reveals that DiP improves the diversity of preconditions significantly compared to a beam search baseline. Also, manual evaluation shows that DiP generates more preconditions than a strong nucleus sampling baseline.

## introduction preconditions are an important part of language understanding with numerous applications, ranging from event understanding to story generation. they provide the semantic glue to understand (or generate) the chains of events common in narrative text. how can we build intelligent systems to fill in these chains, or to identify semantically related events in context? @xcite took a first step by introducing a precondition generation task, where given a target event mention the goal is to generate text that describes a precondition for the target. they released the 'peko' dataset for training, and showed that a gpt-2 model can be fine-tuned on target: [blank] to fill mr. lavelle 's seat, for a term that expires on dec. 31, 2008. the senate voted overwhelmingly on thursday the senate voted on wednesday the senate voted overwhelmingly on wednesday the senate voted overwhelmingly on tuesday mr. lavelle was appointed by gov. eliot spitzer generating non-repetitive diverse outputs is a challenge for any conditional language generation system. our analysis of the gpt-2 based model shows that this is also the case for preconditions. how can we induce a model to generate diverse outputs without losing quality? context sensitivity might help with quality, but it also hinders diversity. to address this we introduce a three-stage generative process, which we call dip. in the first stage, dip uses an event sampler whose only goal is to generate event trigger words as precondition candidates. in the second stage, dip forces the generative model to use the candidate triggers from the first stage to produce the full description of the precondition event. in the third stage, dip reranks and filters the generated descriptions using a precondition classifier (also trained from the same training data). 2 a brief example is shown here: target event: i apologized for the debacle of the day before, and [blank] to help me make it right stage 1 stage 2 stage 3 trying i am trying now delete use i use my time delete used i used my time #2 asked asked me #3 hired hired a new staff #1 experiments on the peko dataset show that dip produces more diverse and better quality preconditions compared to standard beam decoding, as well as an iterative filtering extension that applies a standard repetition penalty in a sampling strategy. analyses show that dip is able to better balance the need for diversity against quality. while the iterative repetition penalty method generates lexically diverse outputs, it often introduces irrelevant information rather than producing distinct types of preconditions. our human evaluation shows that dip on the other hand is able to produce text that is more likely to be preconditions. all code and data are available at https:// stonybrooknlp.github.io/dip/ . ## related work most work on logical preconditions has focused on identification/extraction from text. for example, @xcite identified preconditions using a svm-based score function with hand-crafted pmi and wordnet based features. @xcite extracted domain-specific precondition relations from instructions for the game of minecraft. this paper is instead focused on generating novel preconditions. to the best of our knowledge, only the prior peko work @xcite has attempted this. we are building on those initial ideas. there has been research for diverse generation using control codes or latent variables. some works use explicit cues to control text generation. @xcite used emotion embeddings to generate dialogue responses in a specific mood. keskar et al. (2019) trained a lm with human readable control codes, which describe domain, style, or topics. then the model learns to generate text conditioned on a given code. the model requires manually predefined control codes and a corresponding training corpus for each code. other diverse generation works learn latent representations or codes from input text, and then generate text conditioned on those codes. @xcite applied a sentence embedding to generate syntactically diverse translations. they find that syntax-based encoding with treelstm @xcite yields better diversity than a contextual encoding using bert @xcite or fasttext @xcite . @xcite used k categorical latent variables to control the generation context of dialogue responses and pick the highest probability response from the responses generated using the latent variables. cod3s @xcite is designed to generate diverse causal relations. it uses locality-sensitive hashing (lsh) @xcite on representations from sentence-bert (reimers and gurevych, 2019). conditioning on these 16-bit lsh signatures, it generates cause/effect sentences using a transformer architecture @xcite but with a limited vocabulary size of 10k. these previous approaches have some drawbacks -they either require explicit control codes and training examples, or they have low interpretability of their codes. our approach addresses these two limitations: control codes are learned from non-diverse input text and the codes are human-readable events. and these approaches are not directly comparable to our method without proper modification, which would not be fair comparisons. thus, we present our own baselines for evaluation, and these baselines serve as a proxy of ablation studies as well. ## diverse precondition generation this section describes our diverse precondition generation task and our methodology for solving it. our proposed approach does not require additional diverse training examples. ## evaluation our goal is to investigate the impact of our dip approach for generating diverse and high-quality preconditions. we closely follow @xcite for the experimental setup and the gpt-2 based generation system for our evaluation. ## conclusion target event: [blank] to maintain below-market rents. beam search rps dip in the last few years, real estate prices have risen and real estate prices have dropped wilhelmina's stock dropped 8 percent in early 2005 after a lull, but more actively revived the strong dollar, lending to an expected influx of funds the city has reached a deal with the tenants in the last few years, real estate prices have risen and real estate investment trusts have grown the city council's 10-member city planning committee voted 11 to 6 yesterday the city will use the money in the last few years, real estate prices have risen and real estate prices have dropped, but rents have continued property values are rising, rising as the vacancy rate is expected in the 1980s, the city moved its building to a new site in the east river in the last few years, real estate prices have risen and real estate investment trusts have grown in size in less than a year, such improvements have increased through acquisitions and capitalization at new york city police stations, legal firms and cruise ships, suggesting that housing can be bought the city council passed a bill on wednesday that would give the city the authority to build a new building at the site of the old erez subway station, and in the last few years, real estate prices have risen and real estate prices have dropped, and rents have risen in the last few years matthew hallico, president of the general electric company in manhattan, and robert chrisin, a sales vice partner at ira g. albrecht his comments about the incentive package raised many concerns about how it works, as well as what shareholders might do in the 1980s, the city began a program target event: by about 10 p.m., the proposals appeared dead for now [blank]. beam search rps dip mr. spitzer took office after the judge, col richard kultura of thailand, signaled the end of his sentence after the commission filed a proposal to provide @xmath0 2 billion tax break the city council passed them to the city council after the state senate voted yes on key issues after a suicide bomber killed a man in an internet chat room target event: [blank] to scout potential recruits. beam search rps dip the n.f.l. and the n.b.a. have taken steps the pending replacement of carl crawford has enticed some intelligence officials and top qaeda leaders the police took over the department 's operations , and they began the n.f.l. and the n.f.l. have taken steps employees are giving them the opportunity the department is sending a new system the n.c.a.a. has taken steps most somalis want a law that would enable them the new york state department of education began a program last year the n.f.l. and the n.b.a. have taken similar steps shortly after katrina , post servicemen were chasing selectors after the storm 's onset the department has sent a handful of officers to the police the n.f.l. and the n.f.l. have taken similar steps ever since college opened in 1983 , he shopped for school assignments the n.c.a.a. set up a task force