Fabula Entropy Indexing: Objective Measures of Story Coherence (2021)

Abstract: Automated story generation remains a difficult area of research because it lacks strong objective measures. Generated stories may be linguistically sound, but in many cases suffer poor narrative coherence required for a compelling, logically-sound story. To address this, we present Fabula Entropy Indexing (FEI), an evaluation method to assess story coherence by measuring the degree to which human participants agree with each other when answering true/false questions about stories. We devise two theoretically grounded measures of reader question-answering entropy, the entropy of world coherence (EWC), and the entropy of transitional coherence (ETC), focusing on global and local coherence, respectively. We evaluate these metrics by testing them on human-written stories and comparing against the same stories that have been corrupted to introduce incoherencies. We show that in these controlled studies, our entropy indices provide a reliable objective measure of story coherence.

## introduction automated story generation is one of the grand challenges of generative artificial intelligence. ai storytelling is a crucial component of the human experience. humans have always used storytelling to entertain, share experiences, educate, and to facilitate social bonding. for an intelligent system to be unable to generate a coherent story limits its ability to interact with humans in naturalistic ways. there have been a number of techniques explored for story generation; these include symbolic planning, case-based reasoning, neural language models and others. despite extensive research, automated story generation remains a difficult task. one of the reasons why automated story generation is such a difficult area of research is due to weak objective validation measures. traditional automated measures of natural language quality-perplexity and n-gram based methods such as bleu @xcite -are insufficient in creative generation domains such as story generation. these metrics assume that generated language can only be good if is resembles testing data or a given target story. this precludes the possibility that stories may be good yet be completely novel. indeed, the goal of story generation is usually the construction of novel stories. in the absence of automated evaluation metrics, the alternative is to use human participant studies. human participants, typically recruited via crowdsourcing platforms (e.g mechanical turk or prolific), are asked to read the stories generated by various systems and provide subjective rating or rankings. questionnaires may ask participants to rate or rank the overall quality of stories, but may also ask specific questions about features of stories such as fluency or coherence. coherence is particularly difficult feature of stories to measure because the term "coherence" can mean different things to different participants. in this paper, we introduce a technique for objective human participant evaluation, called fabula entropy indexing @xcite . fei provides a structure for metrics that more objectively measure story coherence based on human question-answering. a fabula is a narratological term referring to the reader's inferred story world that a story takes place in, whether it be similar to the real world or a fantasy or science fiction world. the reader may of course be surprised by certain events but other events may seem implausible or contradictory, thus disrupting coherence. as they read, humans form cognitive structures to make sense of a story, which in turn can be used to answer simple true/false questions about the story. as such, an incoherent story results in readers making random guesses about the answers to these questions. fei metrics thus measure the entropy of the answers-how much the answers disagree with each other-which directly correlates with the coherence of the story. we introduce two such fei metrics: entropy of transitional coherence (etc) and entropy of world coherence (ewc), measuring (respectively) sequential coherence between events in a story, and the internal coherence of the story world: the facts about characters, objects, and locations that distinguish a story. the correlation between human question-answering and these metrics are grounded in narratological 1 theories. to validate the measure, we test our metrics on human-written stories as well as corrupted versions of those stories. for the corrupted stories, we artificially reduce the coherence by altering elements of the story. we show that fei metrics evaluate non-corrupted human-written stories as having low entropy and corrupted stories as having higher entropy. ## preliminaries in this section we review narratological definitions that will be relevant to understanding how to measure the fabula entropy indices. definition 3.1. a narrative is the recounting of a sequence of events that have a continuant subject and constitute a whole @xcite . an event describes some change in the state of the world. a "continuant subject" means there is some relationship between the events-it is about something and not a random list of unrelated events. all stories are narratives, but also include some additional criteria that are universally agreed upon. structural narratologists suggest there are different layers at which narratives can be analyzed: fabula and syuzhet @xcite definition 3.2. the fabula of a narrative is an enumeration of all the events that take place the story world. definition 3.3. the syuzhet of a narrative is a subset of the fabula that is presented via narration to the audience. the events in the fabula are temporally sequenced in the order that they occur, which may be different than the order in which they are told. most notably, the events and facts in the fabula might not all exist in the final telling of the narrative; some events and facts might need to be inferred from what is actually told. it is not required that the syuzhet to be told in chronological order, allowing for achronological tellings such as flash forward, flashback, ellipses (gaps in time), etc. they key is that readers interact more closely with syuzhet and must infer the fabula through the text of the syuzhet. because a fabula inferred, it may be occuring in one of many possible worlds in a modal logic sense @xcite . definition 3.4. a story world is a set of possible worlds that are consistent with the facts and events presented to the reader in the syuzhet. as events and facts are presented throughout the narrative, the probability cloud over story worlds collapses and a reader's beliefs become more certain. events in the fabula and story world have different degrees of importance: definition 3.5. a kernel is a narrative event such that after its completion, the beliefs a reader holds as they pertain to the story have drastically changed. definition 3.6. a satellite is a narrative event that supports a kernel. they are the minor plot points that lead up to major plot points. they do not result in massive shift in beliefs. satellites imply the existence of kernels, e.g. small plot points will explain and lead up to a large plot point, but kernels do not imply the existence of satellites-kernels do not require satellites to exist. a set of satellites, @xmath0, . . . , s n }, is said to be relevant to a kernel k if, after the kernel's competition, the reader believes that the set of questions posed by k are relevant to their understanding of the story world given prior s. an implication of kernels and satellites is that one can track a reader's understanding of a story over time by asking the reader questions relevant to the story before and after each major plot point. as kernels change the reader's beliefs about the story world and the fabula, then their answers to questions change as well. ## fabula entropy indexing fabula entropy indexing (fei) measures story coherence based on human question-answering. humans build cognitive structures to make sense of a story, which in turn can be used to answer simple true/false questions about the story. a coherent narrative results in readers having well-formed cognitive models of the fabula and story world @xcite . because the cognitive models formed during reading are predictable across readers one can infer that coherent stories result in readers being more likely to answer questions about a story similarly @xcite . incoherent stories thus result in readers making random guesses about the answers to questions. fei looks at the entropy of the answershow much readers disagree with each other-as a signal of coherence of the story. we decompose fei into two separate metrics. entropy of transitional coherence (etc) measures the necessity of transitional ordering: in time t, event or fact x is necessary to maintain a story's coherence. in other words, was this fact probable before t? this establishes whether a reader could reasonably anticipate the occurring between two events. entropy of world coherence (ewc) on the other hand is not time dependent. ewc measures the probability of an event or fact y occurring at any time in a story world. the core idea of fabula entropy indexing is that readers can be asked true/false questions and that the agreement in readers' answers indicates coherence. however, questions must take the form of implications q : @xmath1. definition 4.1. for a question about a story, q, of the form "if a then b" with possible values for @xmath2, f } and possible values for @xmath3, f }. identifying a with the set of possible answers to it, we say that the relevance of b to a given some prior γ is note that the relevance of b to a depends on the ground truth. consider the case where a is "is harry potter the prophesied heir of slytherin?" and b is "can harry potter speak parseltongue because he is a descendent of slytherin?". if harry is a blood descendant of slytherin and that is why he can speak parseltongue, then b is highly relevant to a. however, the actual truth of the matter is that harry's abilities are completely independent of his heritage. therefore b does not have relevance to a even though it could have had relevance to a had the ground truth been different. ## experiments to validate fabula entropy indexing in general, and etc and ewc in particular, we need to verify that the methodology in section 4.3 produces low entropy values for coherent stories and high entropy values for incoherent stories. because automated story generation is still an open research question, we validate etc and ewc on humanwritten stories that are known to be coherent. we assume that human-written stories are coherent. to compare entropy indices against incoherent stories, we devise a technique for corrupting human written stories in particular ways that are likely to result in incoherent stories. exemplar corruptions include negating adjectives, swapping events from different stories or randomly changing key descriptors of characters. ## discussion from the results, we can make some observations. the first is that the corrupted stories are not a traditional experimental baseline. the corruptions were designed to show that intentionally introduced incoherencies do in fact result in an increase in entropy. second, the corruptions are designed to introduce the smallest possible amount of incoherence to stories as possible. therefore, we would not expect a large increase in entropy due to a single corrup-tion per story. the fact that entropy increases with the introduction of minimalist corruptions indicates that fabula entropy indexing is sensitive to such small changes. we would anticipate an automated story generator that routinely makes transitional or world coherence errors to result in much more significant differences in entropy values. the entropies for corrupted stories have more dense distributions. not only was there more disagreement about the answers to questions, but the disagreement was consistent across all stories. this is to be expected because the corruptions are synthetically designed to damage story coherence. the entropy distributions for real stories was spread over a wider range of entropy values per story. etc might not be as strong a metric as ewc. the average etc of uncorrupted stories is higher than the ewc of uncorrupted stories. this may be due to ( a the quality of the entropy indices are highly dependent on the extent to which the true/false questions target points in the story where potential incoherence can arise. it may theoretically be possible for some automated story generators to automatically generate good sets of questions, however this is currently an open research problem. the authors of this paper could have generated a better set of true/false questions targeting etc and ewc than those unaffiliated with the research. however, doing so introduces the possibility of experimenter bias, which needs to be avoided by those who use this evaluation technique. fei has a couple of limitations. first, to measure etc one must be able to identify kernels and make questions about elements before and after the kernels. second, to measure ewc, the stories must be highly descriptive in nature and that there are plot points that are dependent on adjectives; many story generators do not produce descriptive texts. fei was validated on short stories, of 10 sentences or less. while there is no theoretical reason it will not work on longer stories, it will require substantially more questions to be produced and answered by human participant studies. we have used the fabula entropy indexing method described in this paper to evaluate an automated story generation system in (under review, 2021). the redacted system was designed explicitly to increase coherence of automatically generated stories over a large pretrained transformer language model baseline. the combined etc and ewc for the experimental system were lower than the language model baseline. moreover, we also compared the entropy indices of human-written baseline stories, showing that human stories result in lower entropy values than ai generated stories, which is to be expected at this time. this constitutes the first successful use of fei for its intended purpose of evaluating automated story generation systems. as part of the above real-world test case of fei, we also performed a subjective human-participant study, showing that the entropy indices are low when humans report perceived coherence. we did not perform a subjective human participant study for this paper since we were working on stories that came from sources with reliable coherence. ## conclusions automated story generation research requires strong, reliable evaluation metrics, which have largely been absent, hampering research progress. we present the fabula entropy indexing technique for objectively evaluating the coherence of stories. we demonstrate the effectiveness of this technique by showing how two fei metrics, entropy world coherence and entropy transitional coherence, can be used to clearly discriminate between stories with and without coherence corruption. in contrast to subjective human participant studies, where it is challenging to get participants to answer questions about coherence, fei provides a numerical rating of the coherence of stories that is grounded in theory.