GLUCOSE : G enera L ized and CO ntextualized Story Explanations (2020)

Abstract: When humans read or listen, they make implicit commonsense inferences that frame their understanding of what happened and why. As a step toward AI systems that can build similar mental models, we introduce GLUCOSE, a large-scale dataset of implicit commonsense causal knowledge, encoded as causal mini-theories about the world, each grounded in a narrative context. To construct GLUCOSE, we drew on cognitive psychology to identify ten dimensions of causal explanation, focusing on events, states, motivations, and emotions. Each GLUCOSE entry includes a story-specific causal statement paired with an inference rule generalized from the statement. This paper details two concrete contributions. First, we present our platform for effectively crowdsourcing GLUCOSE data at scale, which uses semi-structured templates to elicit causal explanations. Using this platform, we collected a total of ~670K specific statements and general rules that capture implicit commonsense knowledge about everyday situations. Second, we show that existing knowledge resources and pretrained language models do not include or readily predict GLUCOSE’s rich inferential content. However, when state-of-the-art neural models are trained on this knowledge, they can start to make commonsense inferences on unseen stories that match humans’ mental models.

## introduction humans make countless implicit commonsense inferences about everyday situations. for example, consider the following short story from the roc-stories corpus @xcite : gage was riding his bike. a car turned in front of him. gage turned his bike sharply. he fell off of his bike. gage skinned his knee. when even young children read this story, they construct a coherent representation of what happened and why, combining information from the text with relevant background knowledge @xcite . for example, they can construct the causal chain that explains how the car's unexpected turn ultimately led to gage falling, describe how gage's emotion and location changed throughout the story, and even hypothesize that he likely shouted for help after falling. though humans build such mental models with ease @xcite , ai systems for tasks such as reading comprehension and dialogue remain far from exhibiting similar commonsense reasoning capabilities. two major bottlenecks have been acquiring commonsense knowledge and successfully incorporating it into state-of-the-art ai systems. to address the first bottleneck, we have built an effective platform to acquire causal commonsense knowledge at scale. to address the second, we show that pre-trained neural models can start to make similar inferences when trained on such rich curated data. we introduce the glucose 1 (generalized and contextualized story explanations) dataset. given a short story and a sentence x in the story, glu-cose captures ten dimensions of causal explanation related to x. these dimensions, inspired by human cognitive psychology, cover often-implicit causes and effects of x, including events, location, possession, and other attributes, the vast majority of which are not captured by existing resources 1 human brain functions such as thinking, memory, and learning are closely linked to the glucose levels and how efficiently the brain uses this fuel source @xcite . if there is not enough glucose in the brain, neurotransmitters are not produced and communication between neurons breaks down. we are calling this resource glucose, since we believe ai brains need this source of fuel to enable their basic thinking and fill in their reasoning gaps! dimension semi-structured specific statement and inference rule: antecedent connective consequent 1: event that directly causes or enables x a car turned in front of him causes/enables gage turned his bike subject verb preposition object subject verb object somethinga turns in front of somethingb (that is someonea's vehicle) causes/enables subject verb preposition object someonea turns somethingb away from somethinga subject verb object1 preposition object2 2: emotion or basic human drive that motivates x gage wants safety causes/enables gage turned his bike subject verb object subject verb object someonea wants safety causes/enables someonea moves away from somethinga (that is dangerous) subject verb object subject verb preposition object 3: location state that enables x gage was close to a car enables gage turned his bike away from the car subject verb preposition object subject verb object1 preposition object2 someonea is close to somethinga enables someonea moves away from somethinga subject verb preposition object subject verb preposition object 4: possession state that enables x gage possesses a bike enables gage turned his bike subject verb object subject verb object someonea possesses somethinga enables someonea moves somethinga subject verb object subject verb object 5: other attributes enabling x: n/a (the dimension is not applicable for this example) 6: event that x directly causes or enables gage turned his bike causes/enables he fell off his bike subject verb object subject verb object someonea turns somethingb (that is someonea's vehicle) causes/enables someonea falls off somethingb subject verb object subject verb object 7: an emotion that is caused by x: n/a 8: a change in location that x results in gage turned his bike away from the car results in gage was further from the car subject verb object1 preposition object2 subject verb object1 preposition object2 someonea moves away from somethinga results in someonea is further from somethinga subject verb preposition object subject verb preposition object 9: a change of possession that x results in: n/a 10: other changes in property that x results in: n/a and models. importantly, glucose encodes commonsense knowledge in the form of semistructured inference rules foot_0 (mini-theories about the world), each grounded in a specific story. as the examples in to facilitate acquisition at scale, we designed an effective multi-stage crowdsourcing platform and used it to acquire more than 670k glucose annotations in the context of children's stories. our analysis shows that these explanations extend substantially beyond the scope of the existing knowledge resources. given the breadth of commonsense knowledge needed for real-world inference tasks, no static knowledge source is expected to provide sufficient coverage. glucose's key contribution is enabling models to dynamically produce general inference rules to explain novel scenarios. to systematically evaluate such models, we present an evaluation task where given a story s, a sentence x, and dimension d, a model predicts relevant specific and general rules as captured in glucose. we evaluate on the task using a curated test set, based on novel stories not used for any training purposes. we show a strong correlation between human and automatic evaluation metrics, which makes systematic and reliable evaluation of models feasible. we show that pre-trained neural models perform poorly on the task; however, when finetuned on glucose data, they are able to generate commonsense explanations that rival humans'. this finding supports our hypothesis that a promising recipe for giving machines commonsense is to use quality-monitored crowdsourced commonsense knowledge for training neural models that have preexisting lexical and conceptual knowledge. ## related work recently, there has been a renewed interest in commonsense reasoning @xcite @xcite , further fostered by the increasing need for explainable ai systems @xcite . one well-known type of commonsense knowledge is script knowledge, defined by @xcite as structured knowledge about stereotypical event sequences and their participants. however, manual encoding of such knowledge is notoriously unscalable and brittle. a more recent line of work is unsupervised learning of "narrative schemas" @xcite @xcite , where common event sequences are automatically induced from large corpora. while promising, this approach has not produced high-quality knowledge usable for downstream tasks at scale @xcite . furthermore, since commonsense knowledge is often implicit, such corpus-based methods are unlikely to induce implicit commonsense inferences @xcite . in contrast, our data collection framework has enabled us to acquire high-quality and robust commonsense knowledge, including often unstated rules such as "someone a gives someone b something a results in someone b possesses something a " or "someone a is at somewhere a enables someone a puts something a at somewhere a ". the most fruitful efforts to date for acquiring commonsense knowledge have been crowdsourced knowledge resources. conceptnet @xcite , a partially-crowdsourced resource, is a relational knowledge graph that connects short naturallanguage phrases via semantic edges. most con-ceptnet knowledge is taxonomic, consisting of factoids like "apple is a fruit", however, it also includes some causal relations, e.g., "kill is motivated by revenge." despite its broad coverage, conceptnet has been found to be noisy @xcite . its knowledge also lacks context, hampering accurate application at inference time, e.g., "kill requires eat breakfast" is hard to make sense of without more context. a more directly relevant resource is atomic @xcite , which consists of 877k textual descriptions of if-then knowledge. each entry describes a likely cause/effect of one of 24k+ events. atomic entries are organized into nine categories such as xintent (personx's intention) and xeffect (effect on personx). for instance, "personx makes persony's coffee xeffect personx gets thanked". atomic is a great step forward in acquiring highquality inferential knowledge. however, it has two main shortcomings. first, atomic is noncontextual and conflates knowledge about an event that may have occurred under different scenarios, which hinders interpreting and applying the knowledge in context. for example, the event "personx arrives the next day" has xintents "to go on vacation" and "to attend a reunion," and xeffects "get time to relax" and "meet some friends." although each xintent should be associated with only one of the xeffects, such dependencies are not encoded in atomic. as a result, atomic cannot be used to determine which xeffect is more likely given an xintent. glucose addresses this by grounding each piece of inferential knowledge to a particular story context consistent across dimensions. second, events and relations in atomic are person centric; agentless events are not covered, and each relation is either about personx or per-sony. as a result, atomic cannot describe events involving common entity types such as places, things, or groups of people, nor can it encode causes and effects other than to personx and their peers. in glucose, sentence x can describe any event/state, and glucose general rules can refer to indexed variables such as "someone a " or "somewhere c ." beyond these major shortcomings, atomic also does not cover many commonsense knowledge types in glucose, including change of attributes such as location, which will be further discussed in section 4.3. ## the knowledge model of glucose glucose has a unique take on explaining story events. as illustrated in ## empirical evaluation task we set up a standalone evaluation task for evaluating models that predict glucose explanations: given a story s, a story sentence x, and a dimension d, provide an explanation in both specific and general forms. test set curation for a test set on commonsense reasoning to offer accurate and reliable evaluation, it should contain unambiguous examples with clear gold answers. this led to a curation process that identifies examples on which humans have high agreement, as follows: we sampled s, x pairs annotated by any three workers with the highest quality rating. a dimension d for s, x was allowed into the test set if 1) d was annotated by all three workers, and 2) the three specific statements had a round-robin average sentence-level bleu @xcite ) score foot_4 above 0.75. finally, two in-house annotators manually removed cases with typographical or core content errors, resulting in a test set of 500 story/sentence pairs, each with 1-5 dimensions answered. human and automatic evaluation human evaluation is crucial for any language generation task. we crowdsourced our human evaluation on mturk, using a dedicated ui, 12 asking three of our top-rated crowd workers from the main glucose crowdsourcing job to rate the predictions. we set up the following evaluation process to ensure calibrated judgments: the judge first reads a story with a highlighted sentence x, then reads a question about x corresponding to a glucose dimension. next, they are shown a shuffled list of candidate answers, each produced by a different system. finally, the judge rates each candidate answer on a four-point likert scale: "completely incorrect," "almost incorrect," "almost correct," and "completely correct." to compare system performance, the ratings are mapped to numerical scores of 0-3, which are then averaged. automatic evaluation for tasks involving language generation has been a major bottleneck for research @xcite ). bleu's ease of replicability has made it a popular automated metric, but its correlation with human judgement has proven weak on various tasks @xcite . for automatic evaluation, we use sacrebleu @xcite with equal weights up to 4-grams at corpuslevel on the three-reference test set. using pairwise correlation analysis, we found strong correlation between human and bleu scores on our test set, with correlation coefficients @xmath1.891, @xmath2.855, and kendall's τ = 0.705, all with p-@xmath3.001. the high correlation is due to various design choices, including 1) semi-structured inference rules in glucose are designed to be evaluable, where the structure constrains the variability of the rules, and 2) we minimized the noise in our human evaluation by designing a ui that could collect calibrated ratings from human judges educated about the task. the strong correlation suggests that bleu is a viable metric for reporting future results on the glucose test set. ## models we developed several models for tackling the prediction task described in section 5. the train and development sets for each model consisted of the initial 440k total annotations foot_6 (in the context of 3,360 stories) in the glucose dataset, minus the entries that share the context story with the test instances. due to their superior performance in sequence prediction, all our neural models use transformer blocks @xcite , which use multiheaded attention and fully connected layers to encode sequences. for decoding, all models use top-k random sampling @xcite . details on all the models we experimented with can be found in appendix c. ## results and discussion in fact, enc-dec performs competitively with humans in many dimensions. the strength of this model's performance in predicting both specific 15 a clear drawback of language models is that the model's representation of the ith item depends only on items preceding i, and not the full input context. we show that better predictions can be made given full visibility into the entire input sequence. model dim 3: a location state that enables x dim 6: an event that x causes/enables full-lm karen is at home enables karen made a pan of lasagna and brought it to the party karen made lasagna causes/enables karen ate lasagna someonea is in somewherea enables someonea makes somethinga (that is edible) someonea cooks somethinga (that is food) causes/enables some peoplea to be turned away because of somethinga (that is food) enc-dec karen is in the kitchen enables karen makes a pan of lasagna karen makes a pan of lasagna causes/enables karen eats it for a week someonea is in a kitchen enables someonea cooks somethinga someonea makes somethinga (that is food) causes/enables someonea eats somethinga human karen is in the kitchen enables karen made a pan of lasagna karen made a pan of lasagna causes/enables she brought it to a party someonea is in a kitchen enables some-onea prepares somethinga (that is a dish) someonea prepares somethinga (that is a dish) causes/enables someonea takes somethinga to somethingb (that is an event) and general rules is a testament to the high quality of the glucose training data. its worst performance is on general rules for dimensions 5 and 10, which have the lowest number of training points and are the most diverse in content. other models perform as expected. pt-lm's poor performance shows that finetuning on our dataset significantly improves the commonsense inference capabilities of lms. 1s-lm, which only predicts half of an inference rule, outperforms full-lm in predicting specific statements, but lacks the ability to generalize them. we also tested various other baselines, including an atomic-trained transformer model @xcite , retrieval of k-nearest-neighbors, and non-contextual variants of the presented models, all of which significantly underperformed the results in our results also show that our best models perform noticeably better on specific statements than on general rules. this is because generating a specific statement involves paraphrasing a story sentence and predicting an antecedent/consequent, while a general rule requires further generalizing the paraphrase and the antecedent/consequent appropriately such that the rule remains a generally valid statement about the world. although rule generalization can sometimes be as simple as replacing a named entity (e.g., gage) with a typed variable (someone a ), more often more complex transformations are needed, such as generalizing the action and producing type constraints on variables in the form of attribute phrases. for example, take into account the enc-dec results in overall, our evaluation results show that the state-of-the-art pre-trained models finetuned on the glucose dataset are well capable of dynamically producing glucose-like inference rules on the fly, which is the ultimate usecase of the glu-cose dataset. it is important to note that there is still a consistent performance gap between the best-performing model and human's on generating specific statements and general rules, which indicates that there is still a large headroom for improvement on designing better models for generalizable commonsense reasoning. note that in our current evaluation setup, we have made the simplifying assumption of evaluating each dimension for each sentence individually, without consideration for consistency across dimensions or across sentences. joint prediction of all the dimensions and sentences across the story is a considerably more challenging task that can potentially yield more accurate predictions for a downstream task. we encourage the future work to focus on building models that perform joint predictions, which can be readily evaluated using our test-set. it is important to note that static test sets are inherently narrow and prone to hidden curation biases @xcite . we believe that the ultimate evaluation for models that show glucose-like commonsense reasoning capabilities should be on naturally-occurring arbitrary stories and through our presented human evaluation process. as future work, we are planning to show the value of incorporating glucose-trained models in other downstream nlp tasks such as reading comprehension and dialog. ## conclusions we introduced glucose, a large-scale dataset of implicit commonsense knowledge, encoded as ex-planatory mini-theories grounded in a narrative context. the theories are categorized into ten causal dimensions, inspired by cognitive psychology. we presented our multi-stage pipeline for acquiring semi-structured causal explanations at scale from lay workers, resulting in ˜670k annotations in the context of everyday children's stories. we demonstrated the utility of glucose data in two ways. first, our analysis showed that glucose rules capture knowledge not available in existing resources or pre-trained models. second, in order to evaluate how well ai models can predict glu-cose knowledge on novel inputs, the ultimate value of such a dataset, we defined a standalone evaluation task for predicting specific and general inference rules given a story/sentence pair and a dimension. we curated a doubly-vetted test set, developed a platform to facilitate human judgment of system outputs, and validated bleu as a strong automated evaluation metric. we show that training on glucose data improves model performances significantly on unseen stories. our results validate our hypothesis that a promising approach for imbuing machines with commonsense is to use carefully-crafted data, as in glu-cose, to train neural architectures that have a wide range of lexical and conceptual knowledge encoded, as in models pretrained on large corpora. together with this paper, we release our dataset 16 and models foot_9 , which we hope will enable the ai research community to explore effective approaches to incorporate commonsense reasoning capabilities into various downstream tasks. main task the qualified workers were able to access large batches of data with no limit. the main task starts with a page like the one shown in the figure 5 . the user loops through each of the 10 dimensions of glucose data collection, in order, presented as questions. note that the user could answer the question by simply marking the dimension as not applicable and skipping it. if they choose to answer, as shown in figure 6 , they will be presented with the structured rule slots to input their answers. the full main glucose ui, along with all the detailed instructions that were visible to the workers, is accessible here https://bit. ly/2r8xctt . expert review for work contributed through the main ui, data quality was controlled through daily monitoring of a percentage of incoming submissions and statistics on average dimensions filled out. for managing this process, we built a specialized ui for reviewing the incoming structured data. the percentage of answers reviewed by an in-house expert were used to update worker ratings. workers enter the task with a score of "1", then advance to "2" as they become more proficient, getting a bonus increase. the top rating is "3". select workers with a "3" rating were also moved into "top rated" batches that paid more per hit and included higher bonuses and incentives. if work quality dropped, workers' ratings were adjusted accordingly. if their work was at a risk of degrading the quality of the dataset, they were disqualified from the task. 19