  Automated story generation remains a difficult area of research because it
lacks strong objective measures. Generated stories may be linguistically sound,
but in many cases suffer poor narrative coherence required for a compelling,
logically-sound story. To address this, we present Fabula Entropy Indexing
(FEI), an evaluation method to assess story coherence by measuring the degree
to which human participants agree with each other when answering true/false
questions about stories. We devise two theoretically grounded measures of
reader question-answering entropy, the entropy of world coherence (EWC), and
the entropy of transitional coherence (ETC), focusing on global and local
coherence, respectively. We evaluate these metrics by testing them on
human-written stories and comparing against the same stories that have been
corrupted to introduce incoherencies. We show that in these controlled studies,
our entropy indices provide a reliable objective measure of story coherence.
