Storybranch - generating multimedia content from novels (2025)

Abstract: We present Storybranch - an automated system for generating multimedia content from long texts such as novels and fanfiction.The Storybranch pipeline includes structured information extraction, text parsing and processing, content generation using Gen-AI models and syncronization of different streams (audio, video, background). Our system is highly modular and can efficiently generate three different types of multimodal content: audiobooks, simple animated videos, and visual novel text-and-image-style video games.Storybranch successfully addresses challenges such as generating unique and consistent image and voice for each character and narrator, identifying and generating background images and sounds effects, and syncronizing character expressions and lip movement with text.As part of the Storybranch , we develop and release BookNLP2 - a new open-source library for parsing and extracting information from books, based on the legacy library BookNLP.

## introduction the recent advances of machine learning in the fields of speech @xcite and image @xcite generation have created opportunities in multimedia content creation. different open-source and commercial models @xcite excel at generating images, sound files, and video clips from short texts. however, generating customizable, high-quality multimedia content from long texts, such as novels and books, faces various non-trivial challenges: the system has to keep track of multiple characters, narrator(s), and constantly changing background images and noises. in this paper, we present storybranch : a new system for multimedia content creation from long texts. we extract and process information from novels and books in a unified structured manner and leverage the capabilities of different generative ai models to create content in different output formats: audiobooks, simple animated videos, and video games. figure 1 shows a snapshot from a video generated by storybranch from the book crime and punishment by fyodor dostoyevsky. the scene has two characters: raskolnikov and alyona ivanovna and the background is his room. in addition to being capable of generating complex output in different modalities, storybranch addresses multiple practical challenges: separation of human characters, animals, narrator speech, and the description of background images and sounds; consistent image and voice for each character; dynamic update of context-specific properties of characters, such as change of clothes and age; separate generation of the narrator, background effects and noises; synchronization of different media files. we encourage readers to check the demo video foot_0 . storybranch is highly modular and is structured as a five-step pipeline: multimedia solutions such as audiobooks have become a popular way to enjoy and connect with literature. animations and image-based games can further enhance the experience and make it more interactive and engaging for children and secondlanguage learners. storybranch can generate one or more output types at scale, using the same text-processing and content generation pipeline. as part of the project, we created and released a new text open-source library, booknlp2 2 . ## related work to ensure consistency and overall quality of the generated content, we first perform text parsing. the original booknlp library is inspired by the works of @xcite and @xcite . our baseline and initial experiments use these approaches as well. we extend the toolbox by including the work of @xcite which focuses on parsing fanfiction litrary texts. approaches include supersense tagging @xcite , entity tagging and coreference resolution @xcite . audiobook generation is central to story-branch as both audiobooks and videos require a text-to-speech model as well as generation of special effects. elevenlabs 3 is a highly popular commercial tts system that focuses on the generation of human speech. our work directly takes inspiration from popular tts frameworks such as 2 booknlp2 github 3 elevenlabs: prime voice ai xttsv2 @xcite . the landscape of speech generation has seen rapid growth over the last few years.including work on expressive human-sounding narration @xcite . a current state-of-the-art model for video generation is the work of @xcite . they present a soty video generation paper capable of generating high-quality videos. a key limitation to their work is the scope -they can only generate videos for short timeframes. nonetheless, significant progress is made in improving the art of creating continuous frames @xcite . the evaluation of multimedia generation from long texts is non-trivial, as there are no large-scale datasets for the task. existing benchmarks for llms and nlp models @xcite @xcite are not applicable to the task. we rely on task specific benchmarks for the components of booknlp2 @xcite and perform separate human evaluation on the final multimedia output. ## processing long texts -booknlp2 generating multimedia content from long texts in an end-to-end manner is beyond the capabilities of current open-source models. the complex structure and long-distance dependencies of a book result in high computational cost and low quality output. instead of going end-to-end we take a hybrid divide-and-conquer approach. we first process the book and perform automatic annotation and analysis, converting the raw text into a predefined structured format. then we combine the raw text and the structured data to generate accurate sound files and images for different characters, descriptions, and narrator. finally, we recombine the different streams to obtain a single multimodal output. the problem of parsing long texts has been addressed before in the booknlp (david bamman and shen, 2020) library. it combines various nlp models in a configurable pipeline. unfortunately, the library is outdated and is no longer supported. we developed an updated version of the library called booknlp2. we replicated the original work, updated dependencies, and deployed it on pypl for easy installation across different os. we keep the format of input, output, and usage, enabling backwards compatibility. we have also started replacing the individual nlp models in the pipeline with state-of-the-art models using litbank @xcite for training and evaluation. we design booknlp2 as a configurable modular system that also allows users to finetune the models on specific custom domains and books. in storybranch we use the library with its default pretrained version, without in-domain finetuning, as the first step in the content generation pipeline. we release booknlp2 as open source. ## extracting and structuring data when generating multimedia content from books, a key challenge is to correctly identify the characters, the narrator, and the context of the scene. some aspects of the characters, such as their voice, height, and eye color, remain consistent throughout the book. other aspects, such as their age or clothes, may change from one scene from another. while booknlp2 does a great job extracting information about individual scenes, we need to process and enrich the data further, so that we can account for the dynamic nature of the generation of sound and animations. at the current version of storybranch we have two types of dynamically generated content: audio effects and character animations. our approach identifies the parts of the text that correspond to either of them, extracts and stores the information. we extract the information about the audio effects for each scene in three steps. first, we separate the scenes and perform basic linguistic processing. then, we use zero-shot learning to identify parts of the text that describe effects and background noises. we also ask the model to provide a time duration for the effect. for obtaining an updated description of characters, we also perform a two step approach. first, we use booknlp2 to identify all characters in the book and to store all related information (description, coreference entities, scenes) in a csv format. we use the base description as the foundation of generating animations for the character, that is, the static character information. to account for potential change in the description, for every scene that the character participates in, we use zero-shot learning to dynamically extract information from the text chunk, using structured outputs. here is an example prompt: ## prompt generation after we extract all static and dynamic information for each scene, as described in section 4, we need to convert it into prompts that can be given to generative ai models. we generate the following prompts for each individual scene: ## content generation we use the prompts from section 5 to generate multimedia content, such as audio and image. we follow the format of the prompts and generate individual characters and effects separately. for the audio, we experiment with a variety of tts and tta models. we specifically select models that are open-sourced. where possible, we have finetuned them on voices of individuals that have allowed the usage of their voices. the finetuned models for character speech include xttsv2, styletts2, your-tts, and parler-tts. for sound effects, we used stable-open-audio and audioldm. for the image generation, we also favor open source models. we create "comfyui" workflows and chain multiple publicly available stable diffusion and flux models to generate content. the "comfyui" is key to consistently generating the same character in multiple angles allowing for more dynamic animations for the final video. figure 2 show the same character seen from from multiple angles and with different clothes. ## synchronization once all the components are in place and the dialogue and images are generated for every character, we need to combine all components in a single multimedia output. first, we use the existing meta-data to assign each image and audio to the corresponding scene. then, we compose the audio stream by merging the audio effects with the dialogues. we use the audio as the base to determine the correct timing of the scenes. the images are merged with the audio to generate videos and interactive games. first, we pair the audio with the corresponding character visuals. then we add the background images corresponding to the scene. finally we have the non-speaker character's opacity dropped to hint at who the speaker is. the current video generation is simplified and cartoon-like 4 . we are experimenting with more capable generative models. all of the basic synchronization is currently done using the 'moviepy' python library which is under the mit license. we generate the video in chapters, but we experiment with keeping information 4 demo video of dorian gray between chapters, such as "last known dynamic character description". once the video is generated, we can inspect the output and make iterative improvements to the extraction process, the choice of models, or directly on the csv file, if we want to isolate sources of errors. we conducted experiments with users to assess the best pipeline and llms for the prompt and content creation. we also use that feedback to improve the integration of booknlp and get additional data. ## visual novel game generation we use storybranch to create a new type of content in addition to audio and animation -interactive text-based games. we use the same data that we extracted for the first two steps in the pipeline, and add an additional logic component that allows for user interaction to influence the flow of the story. we use the open-source game library godot @xcite . the library has a visual-novel making component called dialogic that uses character images and dialogues to help make the game. we automatically convert the csv data from sto-rybranch into the dialogic script necessary for the visual novel games. figure 4 shows an example of a dialog script. we are currently developing a method for adding user choices to the game that can lead to new outcomes and interactions. ## storybranch by combining all individual elements in a single system, we have created storybranch -a multimedia generator pipeline able to convert longtext books and novels into multiple output formats. figure 3 shows the overall flowchart of story-branch . we start with two forms of textual analysis -static data extraction using booknlp2 and dynamic data extraction using zero-shot learning. we use the data to first generate an audio book. the audio is then used as a base for creating animations, and the animations are converted in interactive visual-novels using godot. the system is highly flexible and allows for customizing the models, the prompts, or directly modifying the raw data of the csv. we have experimented with several classical novels and achieved good content quality. we performed two experiments using human participants to determine the impact of different models and settings on the final product. we recruited 12 volunteers for the task and asked them to compare audiobooks. in the first experiment, we compared audios generated by storybranch using different generative models. in the second experiment we compared storybranch to existing solutions for audiobook generation. ## conclusions and future work we have created a new system, storybranch , which can convert long texts such as novels and fanfiction text to multiple types of multimedia content with a high degree of success. it provides an immersive approach to experience books by combining narrative, sound effects, and animation design, transforming stories into more dynamic and entertaining format. the audiobooks, animated videos, and visual novel games improve the user experience and accessibility. during the creation of storybranch we addressed several challenges: identifying different scenes, characters, background sounds and images. we had to balance between static properties and in-context dynamic changes in the characters and the environment. the addition of sound effects was well received by users, who rated storybranch above commercial alternatives. to the best of our knowledge, this is the first system that can consistently generate long videos from books and convert them to visual novel games. the animations and novels facilitate book access to children and second-language learners in a engaging and interactive way. we encourage the readers to check the available demonstration videos: usage; video output; lip sync; game. the pipeline approach of storybranch can be used as-is to automatically create multimedia content, but can also be modified and extended to new output formats specific to user needs. our new library booknlp2 facilitates the creation of multimedia formats and can help both researchers and practitioners. the library is open-source and has been released to the community. as a future work, we are still working on improving the system across several different directions: