AIS tory S imilarity: Quantifying Story Similarity Using Narrative for Search, IP Infringement, and Guided Creativity (2024)

Abstract: Stories are central for interpreting experiences, communicating, and influencing each other via films, medical, media, and other narratives. Quantifying the similarity between stories has numerous applications including detecting IP infringement, detecting hallucinations, search/recommendation engines, and guiding human-AI collaborations. Despite this, traditional NLP text similarity metrics are limited to short text distance metrics like n-gram overlaps and embeddings. Larger texts require preprocessing with significant information loss through paraphrasing or multi-step decomposition. This paper introduces AIStorySimilarity, a novel benchmark to measure the semantic distance between long-text stories based on core structural elements drawn from narrative theory and script writing. Based on four narrative elements (characters, plot, setting, and themes) as well 31 sub-features within these, we use a SOTA LLM (gpt-3.5-turbo) to extract and evaluate the semantic similarity of a diverse set of major Hollywood movies. In addition, we compare human evaluation with story similarity scores computed three ways: extracting elements from film scripts before evaluation (Elements), directly evaluating entire scripts (Scripts), and extracting narrative elements from the parametric memory of SOTA LLMs without any provided scripts (GenAI). To the best of our knowledge, AIStorySimilarity is the first benchmark to measure long-text story similarity using a comprehensive approach to narrative theory. Code and data are available at https://github.com/jon-chun/AIStorySimiliarity.

## introduction stories and narrative are universally used by humans to communicate, interpret, store, and react to the world around them @xcite . when organized within a narrative framework, information can be more readily understood, stored, and recalled @xcite . beyond traditional fiction, researchers are now applying narrative theory to enhance medicine @xcite , law @xcite , business @xcite , and national identity rhetoric @xcite . narratives show immense potential for emotional persuasion @xcite and, when used in combination with emotionally intelligent ai @xcite , are classified as high risk by the eu ai act (eu (parliament) -and jaume duch guillot, 2023). a number of traditional nlp subtasks relate to stories and narratives, both for analysis and generation. analysis is typically restricted to short-text lengths from approximately one sentence to several paragraphs at most (e.g. moverscore, bertscore, qaeval). nlp tasks include identifying sentiment, topics, characters, dialog, and events. long texts can be analyzed with a sequential sliding-window of short-text substrings. this enables the extraction of distributed narrative elements from long texts including character social networks @xcite , event timelines @xcite or plot related information like diachronicemotional arcs (chun 2018) and narrative crux points @xcite . most traditional nlp techniques like sentiment classification, ner, and pos limit story analysis to relatively short texts. however, the introduction of the transformer architecture @xcite and rapid progress in llm performance since the launch of chatgpt (openai, 2022) has revolutionized nlp. while smaller traditional models like bert and bart can still be competitive for structured narrow tasks like ner (paper with code, 2024a) and pos (papers with @xcite , llms generally dominate the nlp leaderboards @xcite . more importantly, trained on trillions of tokens of language, llms have acquired a fluency, coherence, common-sense reasoning, expressiveness, and creativity with natural language that enable new, more complex and open-ended nlp tasks like human-level story generation @xcite and analysis @xcite . however, there are serious limitations to trying to understand long-text stories by using short-text nlp techniques over a sequence of sentences or paragraphs. authors, readers, and ip lawyers generally evaluate stories at higher levels of abstractions that escape short-text decomposition techniques or suffer information loss in the process. powerful narrative elements like character arcs, themes and complex plot devices are often latent, implied, and disseminated throughout the text and require a global unified perspective to identify, extract, and analyze. narrative theory and screenwriting conventions provide conceptual frameworks for describing and capturing these essential structural elements inherent in stories. film studies, narratology @xcite ) and script writing best practices @xcite @xcite decompose narrative structures and elements into different narrative elements. characters including relationships and motivations. plot is the sequence of events in the story. settings involve not only time and place, but other aspects like culture. themes are central ideas and messages. character arcs track the transformation of characters over the course of the story in response to events. dialog collectively is the spoken words and interactions that reveal personality, relationships and advance the plot. classification of narrative elements are flexible. a simpler framework could combine character, character arc and dialog into one broader concept of character. arguably least intuitive, themes are the big ideas and messages that provide deeper meaning, emotional connection and purpose like good vs evil, life finds a way, or love endures. a variety of nlg subfields try to leverage hallucination as a creativity control in story generation @xcite , creative writing @xcite , and screenwriting @xcite . text generation (ctg) is focused on controlling the creative process including more precisely directing the degree and type of hallucinations @xcite . this could enhance human-ai interactions from better human-ai creativity collaboration to more engaging chatbots. a relatively recent and small set of researchers have begun focusing on the positive value llm hallucinations can bring in the form of creativity or 'confabulation' @xcite . this growing perspective warrants a survey of hallucination from a creative perspective @xcite , and new applications are being identified like contrastive dataset generation @xcite . the all rely upon upon semantic distance metrics. the use case of quantifying intellectual property infringement of copyrighted works illustrates the concept of narrative 'similarity'. ip infringement upon written work like movie script involves two tests of 'substantial similarity'. the intrinsic test is an analysis of identifiable properties like character, plot points, and themes. the extrinsic test is a more subjective analysis of whether an "ordinary person" would recognize such similarities @xcite . unlike the high-profile nytimes-openai lawsuit claiming perfect word-for-word reproductions @xcite , most infringement cases have historically fallen in this gray zone of 'substantial similarity'. many more cases may arise either accidentally or intentionally as generative ai becomes a mainstream content creation-and creative collaboration-tool. there is therefore a pressing need to formalize a semantic similarity metric for narratives. the main contributions of this paper are: ## related work semeval22 task 8 evaluated the semantic distances between news stories in order to move to more complex semantic metrics. many entrants used text representations like tf-idf derived from traditional low-level syntax features @xcite , but others used features based on higher-level abstractions like narrative schemas and writing style @xcite . however, many of nlg evaluations using high-level abstractions like empathy and style @xcite and the narrative theory of labov and waletzky @xcite focus on creating novel annotated training datasets @xcite . the 6th annual workshop on narrative extraction from text @xcite survey papers provide a contemporaneous overview of some of the more recent approaches to extracting narrative elements from text @xcite . beyond ai text generation, sota llms like gpt3.5 and gpt4o are increasingly used as proxies for human evaluators in open-ended, referencefree nlg tasks @xcite . they provide benefits of speed, scalability, and cost savings alongside increasingly human-level or better performance @xcite @xcite . this llm-as-judge trend @xcite is evident in various nlp tasks, such as evaluating the quality of generated stories, assessing the effectiveness of adversarial attacks, and grading the comprehensibility of disordered speech transcriptions @xcite . for instance, the mt-bench framework demonstrates a strong 80% agreement between llm evaluations and human judgments in assessing model performance @xcite . for semantic text similarity, llms are shown to be more aligned with humans than any other metric @xcite . however, precautions must be taken to avoid biases like a model's preference for evaluating its own generated content @xcite . moreover, challenges remain in areas of trust and safety (reiter, 2024) and problems exist with human evaluations themselves @xcite . despite these limitations @xcite , llms show promise in augmenting and even replacing certain types of human evaluations given continual advances in ai. at higher levels of abstraction, a variety of research areas relate to text similarity. this includes subfields that rely upon structural elements for automatic story generation (asg) or for automated essay scoring (aes). traditionally, these fields have used a combination of human evaluators, humanannotated references, and more general nlp metrics like coherence @xcite . in addition, more formal structural approaches generate or evaluate more diffuse global features like narrative frameworks @xcite , readability (coherence, fluency, simplicity), and adequacy (faithfulness, informativeness) @xcite . emphasis on story similarity between reference and test works relate to plagiarism detection, intel-lectual property infringement, movie recommendation and search engines, hallucination detection @xcite and measuring creativity in derivative works. aistorysimilarity leverages an abstract structural approach using narrative theory with similarity metrics using sota llms. ## conclusion aistorysimilarity presents a novel story similarity metric and benchmark based upon narratology and best practices in screenplay writing. this benchmark overcomes limitations with traditional text and story similarity metrics and has many potential real-world applications including search/recommendation engines, ip infringement detection, and guided creative ai-collaboration. three comparisons methods are tested and evaluated including 1. preprocessing scripts to extract concise narrative elements (elements), 2. using llm parametric memory with a narrative rubric (genai), and 3. providing full-text scripts with a narrative rubric (scripts). for these famous hollywood films, the genai method proved most aligned with the human expert. however, the other two methods (elements and scripts) may be required for narratives that do not exist in parametric memory or are subject to other practical constraints like cost and privacy. in our test dataset, results demonstrate sota llms have a good innate sense of popular hollywood films, narrative theory, and can produce results in strong alignment with human experts.