S tory DB : Broad Multi-language Narrative Dataset (2021)

Abstract: This paper presents StoryDB — a broad multi-language dataset of narratives. StoryDB is a corpus of texts that includes stories in 42 different languages. Every language includes 500+ stories. Some of the languages include more than 20 000 stories. Every story is indexed across languages and labeled with tags such as a genre or a topic. The corpus shows rich topical and language variation and can serve as a resource for the study of the role of narrative in natural language processing across various languages including low resource ones. We also demonstrate how the dataset could be used to benchmark three modern multilanguage models, namely, mDistillBERT, mBERT, and XLM-RoBERTa.

## introduction stories are central to human culture and communication. however, it seems that stories are easier said than generated. despite incredible recent progress in natural language processing generation of longer texts is still a challenge @xcite . @xcite present a machine comprehension corpus for the end-to-end evaluation of script knowledge with 50% of the questions in the corpus that require script knowledge for the correct answer. the authors demonstrate that though the task is not challenging to humans, existing machine comprehension models fail to perform well on the data, even if they make use of a commonsense knowledge base. partially, this challenge could be attributed to the lack of adequate memory models. longer texts demand better memory mechanisms and possible ways to construct such mechanisms are discussed in the literature for the last 25 years. long shortterm memory networks @xcite , neural turing machines @xcite , memory networks @xcite and many other architectures try to tackle this problem. attempts to introduce some form of memory in transformers, such as @xcite or @xcite , could be regarded as the next steps in this long line of work. there are some interesting recent attempts to generate long texts using some form that makes such longer text feasible for a human reader. for example, @xcite generate a diary of a neural network. yet the generation of a narrative is still challenging. for a detailed review of earlier approaches to narrative generation, we address the reader to @xcite . even modern models for narrative generation rely heavily on some form of expert knowledge or some type of hierarchical structure of the narrative. for example, @xcite first generate the predicateargument structure of the text, then generate a surface realization of the predicate-argument structure, finally replace the entity placeholders with contextsensitive names and references. @xcite ; @xcite propose a hierarchical generation framework that first plans a storyline, and then generate a story based on it. present a technique for preprocessing textual story data into event sequences. @xcite develop a model that first generates the most critical phrases, called skeleton, and then expands the skeleton to a complete and fluent sentence. similarly, @xcite provide a mid-level of abstraction between words and a sentence to minimize event sparsity and present a technique for automated story generation whereby the problem is decomposed into the generation of successive events and the generation of natural language sentences from events. finally, @xcite develop an approach, where the user provides the model with such mid-level sentence abstractions in the form of cue phrases during the generation process. however, we should take into consideration that modern natural language processing (nlp) is fun-damentally an experimental discipline, so the lack of dedicated data could be another bottleneck for the development of narrative generation. this paper tries to amend this problem. unfortunately, the majority of available narrative datasets deal with some constrained form of a short plot that is usually called scenario. these scenarios are centered around common activities, i.e. going grocery shopping or taking a shower. these narrative datasets available in the literature are also extremely small and could not be used with the most advanced modern nlp models. regneri et al. (2010) collect 493 event sequence descriptions for 22 behavior scenarios. modi et al. (2016) present inscript dataset that consists of 1,000 stories centered around 10 different scenarios. @xcite provide 200 scenarios and attempt to identify all references to them in a collection of narrative texts. @xcite present a corpus of 50k five-sentence commonsense stories. finally, there is an mpst dataset that contains 14k movie plot synopses, @xcite , and wikiplots foot_0 that contains 112 936 story plots extracted from english language wikipedia. recently @xcite provided a dataset of tv series along with an instrument for narrative arc analysis. these datasets are useful yet as well as a vast majority of the narrative datasets they are only available in english. this paper provides a large multi-language dataset of stories in natural language. the stories have a cross-language index and every story and character are cross-linked if they occur in different languages. additionally, the texts have tags such as a genre or a topic. this is the first story dataset of such magnitude that we know of. we hope that a large dataset of long storylines could be used for various aspects of narrative research as well as to facilitate experiments with end-to-end narrative generation. ## data storydb is motivated by several interesting experiments that used wikiplots -one of the larger english datasets of narratives available for all-purpose narrative research that we have mentioned earlier. seeing various applications that wikiplots dataset found in the nlp community, we believe, that sto-rydb would be even more useful due to multiple languages, advanced filtering that guarantees higher quality of obtained data, and genre tagging. to improve reproducibility and make storydb usable as wikipedia is further updated we publish the data as well as the code for the filtering pipeline foot_1 . the stories that form storydb are extracted from any wikipedia article that contains a sub-header that contains the word "plot" (e.g., "plot", "plot summary", etc.) in a corresponding language. ## evaluation we have used three modern transformer-based architectures for the evaluation: these models are the most widely used multilanguage models to date. the results of the experiments are publicly available at weights and biases 10 , see @xcite . the evaluation was performed on ten largest languages in storydb, namely: english -'en', french -'fr', italian -'it', russian -'ru', german -'de', dutch -'nl', ukrainian -'uk', portuguese -'pt', polish -'pl', and spanish -'es'. we evaluated three tasks: training data consists of sixteen thousand plots in ten languages, with one tenth of the dataset in every language. let us now describe every task in detail. ## discussion we believe that a broad multilanguage dataset of narratives can facilitate several areas of narrative research. ## conclusion this paper presents storydb -a broad multilanguage dataset of narratives. we describe the construction of the dataset, provide the code for the whole pipeline, list the parameters of the resulting dataset, and briefly discuss several areas of natural language processing research, where storydb could be useful for the community. we hope that storydb could be broadened as more plot descriptions are added to various languages. these considerations make storydb a flexible resource that would be relevant for the nlp community as the subfield of quantitative narrative research moves on.