F acts2 S tory: Controlling Text Generation by Key Facts (2020)

Abstract: Recent advancements in self-attention neural network architectures have raised the bar for open-ended text generation. Yet, while current methods are capable of producing a coherent text which is several hundred words long, attaining control over the content that is being generated—as well as evaluating it—are still open questions. We propose a controlled generation task which is based on expanding a sequence of facts, expressed in natural language, into a longer narrative. We introduce human-based evaluation metrics for this task, as well as a method for deriving a large training dataset. We evaluate three methods on this task, based on fine-tuning pre-trained models. We show that while auto-regressive, unidirectional Language Models such as GPT2 produce better fluency, they struggle to adhere to the requested facts. We propose a plan-and-cloze model (using fine-tuned XLNet) which produces competitive fluency while adhering to the requested content.

## introduction story generation is a challenging task in natural language processing, which requires automated systems to produce creative, open-ended text that remains coherent, cohesive and preferably engaging: an ability humans are clearly capable of. earlier attempts at harnessing neural techniques for story generation struggled with staying coherent for long periods of time, and proposed solutions in the form of two-stage processes which first generate skeletal structures and then expand them into longer texts @xcite @xcite . since then, advancements in self-attention architectures and large-scale training resulted in series of pre-trained language models @xcite that demonstrate an ability to remain on topic, while generating cohesive passages which are several hundreds of words long. while the generated text is almost indistinguishable from text produced by humans, the ability to control the contents of the text remains very limited and is still an active research topic @xcite @xcite . attempts to control the content of the text focus on supplying a prefix @xcite , a headline @xcite or a writing prompt @xcite , mechanisms that can influence the overall theme of the story, but do not provide much control over its content. can we harness the power of large pre-trained models to generate coherent text, while allowing finer-grained controlled over the generated text? we suggest a task in which the content of the text is specified using a sequence off key facts expressed in natural language, which the story should follow, in order. an example of key facts 1 and a corresponding story appear in figure 1 . this formulation suggests natural evaluation criteria (section 4.1): the generated text should include all facts, while remaining coherent. we show how to derive a large training corpus of fact-sets and corresponding stories (section 5), and experiment with fine-tuning pre-trained language models on this objective (section 6, section 7). as left-to-right generation (gpt2, bart) tend in 1920s chicago, italian immigrant antonio "tony" camonte acts on the orders of italian mafioso john "johnny" lovo and kills "big" louis costillo, the leading crime boss of the city's south side. johnny then takes control of the south side with tony as his key lieutenant, selling large amounts of illegal beer to speakeasies and muscling in on bars run by rival outfits. however, johnny repeatedly warns tony not to mess with the irish gangs led by o'hara, who runs the north side. tony soon starts ignoring these orders, shooting up bars belonging to o'hara, and attracting the attention of the police and rival gangsters. johnny realizes that tony is out of control and has ambitions to take his position. meanwhile, tony pursues johnny's girlfriend poppy with increasing confidence. at first, she is dismissive of him but pays him more attention as his reputation rises. at one point, she visits his "gaudy" apartment where he shows her his view of an electric billboard advertising cook's tours, which features the slogan that has inspired him: "the world is yours." tony eventually decides to declare war and take over the north side. he sends the coin flipping guino rinaldo, one of his best men and also his close friend, to kill o'hara in a florist's shop that he uses as his base. this brings heavy retaliation from the north side gangs, now led by gaffney and armed with thompson submachine guns, a weapon that instantly captures tony's dark imagination. tony leads his own forces to destroy the north side gangs and take over their market, even to the point of impersonating police officers to gun down several rivals in a garage. tony also kills gaffney as he makes a strike at a bowling alley. johnny believes that his protege is trying to take over, and he arranges for tony to be assassinated while driving in his car. tony manages to escape this attack, and he and guino kill johnny, leaving tony as the undisputed boss of the city. tony's actions have provoked a public outcry, and the police are slowly closing in. then he sees his beloved sister francesca ("cesca") with guino, and kills his friend in a jealous rage, before the couple can inform him of their secret marriage. his sister runs out distraught and tells the police what he has done. the police move to arrest tony for guino's murder, and tony holes up in his house and prepares to shoot it out. cesca comes back, planning to kill him, but ends up helping him to fight the police. moments later, however, she is killed by a stray bullet. as the apartment fills with tear gas, tony leaves down the stairs, and the police confront him. tony pleads for his life, but then makes a break for it, only to be gunned down by the police. outside, the electric billboard blazes "the world is yours". figure 1 : training sample for the facts-to-text task. the facts were automatically extracted from the plot of "scarface" (1932) using the salie system @xcite . note that a fact does not correspond to a contiguous span in the story. to diverge from the input and focus on the generated content (a behavior consistent with the literature @xcite ), we propose a plan-and-cloze method, based on a fine-tuned in-place cloze model (xlnet), which performs considerably better. our contributions in this work are: our code and data are publicly available for future research. more recent works @xcite have experimented in how to utilize an already pre-trained unidirectional model like gpt2 @xcite . their approach argues that if this model can self-attend to the preceding content in a way that controls its next output, then it is possible to produce the representations the model attends to, without the need for explicit text. @xcite demonstrate the ability to encode different problem-domains to representations that can be appended to the context the model attends to, in order to produce control of the desired text. the large flexibility of this approach in regards to the supplied input makes it a viable candidate for our task while our experiments highlight some of its limitations. both @xcite and @xcite have taken an approach of shaping the unsupervised objective of language models to be a sequence-tosequence denoising objective, where the input is some corruption of the text and the expected output is the original text. these works which demonstrated high performance on several text generation benchmarks, are also suitable for our proposed task and so we experiment with bart @xcite in our desired setting. ## stories and facts in contrast to the earlier works in neural story generation referenced in section 2.1, our main focus in this work is the degree to which generated stories can adhere to the desired content, as we harness the power of massively pre-trained self-attention models, to mitigate the difficulty these works encountered with keeping long texts on track. additionally we differ from previous works that explore controlling massively trained transformers (section 2.2) in that we focus solely on fine-tuning, and represent the desired content in a more explicit, semantic meaningful manner, that is easy to use, modify and evaluate, and encompasses content that should appear throughout the text as opposed to various forms of seeding. we therefore seek a method for control-over-content, with a representation of the desired plotline that satisfies both the following requirements: event representations that have been used in previous works, are considerably structured, unnatural for non experts, verb-centric, and are very minimal in information, adding challenges to the transition to natural language that are unnecessary for our setting. on the other hand, natural sentences vary significantly in length, may fuse a large number of events, employ various discourse markers and include other complex phenomenas which we would like to minimize. we suggest the use of an intuitive, loosely defined notion of key facts, simple sentences that describe factual information in the world of the story. each fact should describe an event that occur in the narrative (dorothy's house is swept away by a tornado), state the characteristics of a person or place (the magical forest is haunted by gruesome trolls), specify an emotion experienced by a character (the queen is jealous of snow white) and so forth. consequently, this more concise nature of facts provides a larger degree of freedom in the mapping from facts to complete sentences in a surrounding context. key facts accordingly, are primary pieces of information that are central to the full desired plot (see figure 1 ). ## the task given an input of 5 ordered key facts, we wish to generate a story 100 to 1000 words long, that includes all the facts given as input in their order of appearance, while expanding upon them to produce a broader, yet coherent, narrative. ## deriving appropriate corpora traditional approaches to narratology often separate between the story as a sequence of events and techniques of telling the story @xcite . in this work we focus on descriptions of movie plots as they express in natural language clear series of events, providing stories with a large variance in subject matter, yet that conform to a similar narration approach, as opposed to the large variety found in literature works that employ more complex techniques like dialogues, first person story teller and so forth. we use a large corpus of movie plots taken from wikipedia @xcite as they also conform to similar lengths and format. ## baselines: sequence-to-sequence models the facts-to-story task requires conditioning on the input facts. this is naturally achieved in sequenceto-sequence models like bart, as well as prefix-conditioned models like gpt-2. our baselines are thus using these models while fine-tuning them on our training set. fine-tuned gpt2 + pseudo self attention while gpt2 @xcite has demonstrated impressive generation abilities, its unidirectional left-to-right paradigm, introduces several challenges. our preliminary experiments with concatenating the salient facts as a prefix for the generated text, has failed to converge to reasonable results on our training data. attributing this to vast difference in structure between the key facts and the consecutive prefixes the model has been exposed to during pre-training, we resort to the technique of pseudo self attention as proposed by @xcite . we follow their proposed architecture for story generation on our custom input, hence learning an encoder that can map the key facts input text, to representations that are concatenated to the gpt2 decoder hidden states, therefore enabling the decoder to attend to the input while learning to map it effectively. the input for this encoder is the 5 key facts in sentence form, concatenated consecutively, where in addition to the token embeddings and the position embeddings, we add fact embeddings, that distinguish between the 5 different facts and are learned as part of the fine-tuning process. these facts are added to every token within each fact similarly to the sentence embeddings of @xcite . fine-tuned bart bart @xcite is a sequence-to-sequence model similar in architecture to the original transformer @xcite , pre-trained on a de-noising objective where the input to the encoder is a corrupted version of the expected output. there are several corruption mechanisms that were applied to the input text during pre-training, where the two mechanisms most relevant to our setting are the deletion of random tokens as well as the replacement of random tokens and spans with a custom @xcite token. this de-noising paradigm fits rather naturally to our proposed task and so we supply as input the concatenated facts in their sentence form, separating between the different facts with the [mask] token. this setting requires the model to generalize phenomenas it was exposed to in pre-training to a larger number of missing tokens, while also appropriating the [mask] token as a signal to separate between the different facts. ## proposed approach: plan and cloze both of the above suggested approaches, essentially consist of a unidirectional, left-to-right, autoregressive language model, conditioning itself both on all its previous outputs as well as the supplied input. our experiments (section 8) suggest a limited ability of this approach to conform sufficiently to the conditions supplied in the input. therefore, we diverge from the seq-to-seq approach and re-frame the problem as literal text expansion. our approach consists of a trained planning step, followed by a "fill-in-the-blanks" cloze step. the planning step builds a cloze-structure template, placing all tokens supplied in the input at specific positions in the output. this determines both the incorporation of fact tokens into more complex sentences, as well as the spacing between facts. having a hard constraint on the existence of these tokens in their selected positions, the second step forms a sensible story by filling in the blanks in the contrived template. this target of filling in blanks with hard constraints on existing words in specific positions, correlates well to the de-noising objectives of masked language models @xcite or a permutated language model like xlnet @xcite that receive the exact structure of the target text. we select xlnet, as it is also auto-regressive on a permutated order of predictions, aligning to the common decoding strategy of text generation. planning stage. we wish to learn to construct a cloze structure given only the text of the key facts. since our training data is comprised of facts extracted from the original text, the words in the fact correlate to specific positions in the text. it is important to note that due to the clause rich nature of natural text, facts may appear in a non consecutive form, having other clauses or words appear in between the words that constitute a chosen fact. to support this we predict the spacing between the words of each fact along with the spacing between the different facts. to effectively learn this spacing we assume the following: under these assumptions, we define a learning task that focuses on a desired spacing for each word (or more precisely, for each bpe token). given one long sequence of all the facts concatenated, each token t i of this text is mapped to a number signifying the number of tokens in the original text between it and t i-1 (see figure 3 ). we capitalize on the pre-trained xlnet model's knowledge of words and context, to map a sequence of words to a sequence of positive numbers. the input to the model is the concatenated sequence of all facts with the positional embeddings sequential from 1 to |f actst okens|. to assist the model to differentiate between the facts we add 5 "fact embeddings" that are randomly initialized and are learned as part of this task, each appropriate fact embedding vector is added to all encoded tokens that are part of the corresponding fact similar to the sentence embeddings described in @xcite . we replace the last layer of the model with a new layer of embedding vector size × 1 and apply a relu activation function to obtain non-negative outputs. the corresponding label for each token is the actual spacing in our dataset between this token and its predecessor. we use a modified mse loss, that penalizes mistakes on small numbers more than mistakes on large numbers: this follows the observation that adjacent words (i.e. {should, not}) split far apart by mistakenly predicting a large number have a more severe impact than a mistake on words that were originally located far apart. in inference time we round each output down to the closest non-negative integer. ## conclusion we suggest a challenging task of controlling story generation content by key facts. we demonstrate an approach for deriving a large corpus for this task, suitable for supervised fine-tuning, and evaluate the ability of notable massively pre-trained models, fine-tuned on this corpus, to generate stories adhering to our suggested representations of content. we show a simple planning technique of copying and structuring, that enables reframing the objective as formulating a cloze task followed by filling in the blanks with a pre-trained, permutated order, autoregressive language model, thus achieving competitive results in regard to the coherence of the generated text and substantially superior performance in adhering to the key facts controlling mechanism.