COINS : Dynamically Generating CO ntextualized Inference Rules for Narrative Story Completion (2021)

Abstract: Despite recent successes of large pre-trained language models in solving reasoning tasks, their inference capabilities remain opaque. We posit that such models can be made more interpretable by explicitly generating interim inference rules, and using them to guide the generation of task-specific textual outputs. In this paper we present Coins, a recursive inference framework that i) iteratively reads context sentences, ii) dynamically generates contextualized inference rules, encodes them, and iii) uses them to guide task-specific output generation. We apply to a Narrative Story Completion task that asks a model to complete a story with missing sentences, to produce a coherent story with plausible logical connections, causal relationships, and temporal dependencies. By modularizing inference and sentence generation steps in a recurrent model, we aim to make reasoning steps and their effects on next sentence generation transparent. Our automatic and manual evaluations show that the model generates better story sentences than SOTA baselines, especially in terms of coherence. We further demonstrate improved performance over strong pre-trained LMs in generating commonsense inference rules. The recursive nature of holds the potential for controlled generation of longer sequences.

## introduction narrative story understanding, and similarly story generation, requires the ability to construe meaning that is not explicitly stated through commonsense reasoning over events in the story @xcite . previous work in modeling narrative stories has focused on learning scripts 1 @xcite and learning narrative schemas using corpus statis-1 scripts are structured knowledge about stereotypical event sequences together with their participants. beginning: s1: janie was excited to see her sister's play in theatre. s2: janie got a call from her boss about an emergency work. end: s5: janie watched a video of the play later. ## task definition we formulate the narrative story completion task (nsc) as follows: given an incomplete story (@xmath0, s 2 , s n ) as a sequence of tokens @xmath1, t 2 , ..., t sep , ..., t m } (with t sep a mask token delimiting s 2 and s n ), the goal is to generate the missing sentences (s 3 , ..., s n-1 ) as a sequence of tokens y s @xmath2, y s i 2 , ..., y s i v } (with @xmath3, ..., n-1 and v the maximum length of each sentence). in the setting of the nsc task, we expect the completed story to be coherent. that is, the generated sentences should exhibit reasonable logical connections, causal relationships, and temporal dependencies with each other and the given beginning and ending of the story. in this paper, we define a discourse to be coherent if successive sentences that are about the same entities, and the reported events involving them can be construed to reflect common knowledge about how events are typically connected in a temporal sequence or by causal relations. similar to @xcite , the criteria to conclude that discourse is coherent include require that there are reflections of causality in the text. our take on this task is to incrementally generate contextualized inference rules from the given context, and to make use of this knowledge to generate missing story sentences. relation type dimensions cause (dim 1-5) (1) event that directly causes or enables x; (2) emotion or basic human drive that motivates x; (3) location state that enables x; (4) a possession state that enables x; (5) other attribute that enables x. effect (dim 6-10) (6) an event that is directly caused or enabled by x; (7) an emotion that is caused by x; (8) a change of location that x results in; (9) a change of possession that x results in; (10) other change in attribute that x results in. ## discourse-aware inference rules this section details how we construct training data for the nsc task, by enriching stories with automatically predicted contextualized inferences. 5 we utilize the glucose @xcite dataset, which contains implicit commonsense knowledge in form of semi-structured general and specific inference rules 6 (cf. given the amount of commonsense knowledge needed for real-world tasks, a static knowledge resource is always incomplete. thus, we fine-tune a pre-trained gpt-2 model on the annotated part of glucose to dynamically generate inference rules for each sentence x i of each story s i from the underlying rocstories data. we fine-tune two separate language models csi gen and csi spec for general and specific rules, respectively ( the 10 dimensions d in glucose cover im5 for testing we rely on glucose's manually validated inference rules on a small subset of the rocstories corpus. ## coins: contextualized inference and narrative story completion model in this section we introduce a recursively operating reasoning and sentence generation model: coins. an overview is given in figure this formulation allows us to i) examine inference and generation capabilities separately from each other, ii) helps determine the impact of inferential knowledge on story generation, and iii) can give us insight into how knowledge can guide story generation in a recursive inference framework. ## experiments 6.1 dataset we apply coins to the nsc and the story ending generation tasks. 9 for data statistics see ## results our experimental results are summarised in tables 4 and 6 . nsc task. rules from glucose (for coins). 11 the improvement of coins over glucose-gpt-2 indicates that our model is well able to utilize and profit from the inference rules. in the oracle setting, sr performs much better than gr. this is expected, since oracle rules with access to the full context will deliver more contextually-relevant inferences, while gr rules may diverge more from the story context. however, in the realistic nsc task setting ( ## manual evaluation automatic metrics can give us some indication of nlg quality, however, these metrics do not necessarily reflect the coherence of generated story sentences. we thus conduct a human evaluation focusing on the grammaticality and coherence of the generated sentences in their story context. we 13 in future work, we will test the above hypothesis by experimenting with a bi-directional transformer generation model. the human evaluation results are presented in relevance of generated inferences rules. we further conduct human evaluation to validate the effectiveness and relevance of the generated inference rules. we randomly select 50 instances from the nsc dev set. we asked three annotators to evaluate the (gr) inference rules foot_8 . we define an inference rule to be relevant if (a) it captures implicit causes and effects of a selected sentence x given an incomplete story s , and (b) it is providing useful explanations for the incomplete story s . the result for this evaluation is shown in fig. 3 , for effect and cause relations. we find that in 36% and 34% of cases for effects and causes, respectively (computed on the basis of majority agreement), our algorithm was able to generate relevant inference rules. our annotations yielded fair inter-annotator agreement of fleiss' κ = 0.45. case study. we provide an example from nsc with different generation outputs ( hence, the rules provide both an intermediate representation and explanations for how knowledge can guide or influence story generation. we provide more qualitative examples in the appendix. ## conclusion we addressed a narrative story completion task that allows us to probe the coherence capabilities of a neural generation model. we proposed coins, a model that iteratively generates commonsense inference rules grounded in the context and generates story sentences, using the generated inferences as a guide. human and automatic eval- incomplete story: s1: ken was driving around in the snow. s2: he needed to get home from work. s5: his tires lost traction and he hit a tree. missing sentences: s3: he was driving slowly to avoid accidents. s4: unfortunately the roads were too slick and ken lost control. coins (igr) someonea is going somewhereb cause/enables someonea is at somewhereb, someonea is driving somethinga fast cause/enables somethinga hits somethingb (that is a tree), someonea possess(es) somethinga (that is a car ) @xmath4, he needed to get home enables he drove home, he was driving on ice causes/enables his tires lost traction, he was driving on ice causes/enables he lost control of his vehicle. coins(msgr) he was driving too fast. he lost control of his car . coins(mssr) he was driving on ice. he lost control of his vehicle . gpt-2 he stopped at a gas station. he filled his tank. gpt-2 glu-cose when he got to the house he realized he was stuck. ken had to pull over to get help. ke when he got home, he noticed his tires were flat. he decided to pull over. grf he pulled over to see what was wrong. he saw that his car was stuck in the snow. human he was going very fast. the street was slippery from the snow.