IDN -Sum: A New Dataset for Interactive Digital Narrative Extractive Text Summarisation (2022)

Abstract: Summarizing Interactive Digital Narratives (IDN) presents some unique challenges to existing text summarization models especially around capturing interactive elements in addition to important plot points. In this paper, we describe the first IDN dataset (IDN-Sum) designed specifically for training and testing IDN text summarization algorithms. Our dataset is generated using random playthroughs of 8 IDN episodes, taken from 2 different IDN games, and consists of 10,000 documents. Playthrough documents are annotated through automatic alignment with fan-sourced summaries using a commonly used alignment algorithm. We also report and discuss results from experiments applying common baseline extractive text summarization algorithms to this dataset. Qualitative analysis of the results reveals shortcomings in common annotation approaches and evaluation methods when applied to narrative and interactive narrative datasets. The dataset is released as open source for future researchers to train and test their own approaches for IDN text.

## introduction automatic summarization has often been been studied for domains such as news and scientific reports. while there is some work on narratives like movies and books, there is limited work surrounding automatic summarization of interactive and game narratives. extrapolating idn performance from news article summarization results is non trivial due to longer texts and the existence of elements like characters and plot. idn also differs from movies and books due to the presence of interactivity and game elements that make summarisation of idn different to that of general text and/or linear narratives. unlike novel/movie summarization, idn has the concept of choices, structure and multiple plot lines which also affect the relative importance of sentences. additionally, idn text formats vary significantly and can look like novels, movie scripts, gameplay logs, or a mixture of all three. the idn-sum dataset is generated from fan made transcripts of two narrative games, both sourced from fandom foot_0 -before the storm published by square enix and wolf among us published by telltale games. different simulated playthroughs through the game are generated by implementing a readerbot like the one described in @xcite , assuming a different combination of choices for each playthrough. while these two sources account for only one type of idn (narratives in the form of a gauntlet, see section 3.1), it takes a step towards increasing resources available for research in this area. an analysis of dataset characteristics and performance of some baseline summarisation methods on this dataset is presented. novel contributions of this paper are (a) a new text summarization dataset for idn (idn-sum), with abstractive summaries for overall idn and aligned extractive summaries for multiple @xcite baseline evaluation of standard benchmarks on idn-sum and qualitative analysis of the predictions made by them. ## related work most text summarization work is targeted at news, academic papers and reviews. the most commonly used summarisation dataset is the cnn/dailymail dataset which is a collection of news articles and human written summaries @xcite . summarisation datasets for narratives include datasets with novel chapters and corresponding human written summaries from online guides, @xcite , extractive summaries that read like telegraphs @xcite , stories and summaries from wattpad @xcite , transcripts and summaries of movies @xcite , transcripts of tv shows @xcite and subtitles @xcite . papers on game summarisation are few and usually involve game logs from online games like dota @xcite or commentary from sports @xcite . however, idn text is typically more similar to movie scripts or novels than game logs. the critical role dataset (rameshkumar and bailey, 2020) is a dataset of transcripts and summaries from critial role episodes. this is a transcript of several voice actors playing a table top role playing game and hence captures only one playthrough of a narrative. to the best of our knowledge, idn-sum is the first dataset for idn that captures multiple playthroughs of an idn. unsupervised methods for automatic extractive summarisation use several methods to determine the importance of sentences including statistical methods using features like sentence position and tf-idf, concept based methods that use external databases like wordnet, topic based methods to infer important topics, graph based methods that build intermediate graphs computed through metrics like semantic similarity, semantic methods using techniques like semantic role annotation, optimization methods that involve optimising for constraints (like maximising coverage or minimising redundancy) and fuzzy logic based methods @xcite . supervised methods include different rnns and tranformers, using pretrained models such as bert for summarisation @xcite . variations of bertsum @xcite , summarun-ner @xcite , matchsum @xcite , discobert @xcite , hibert @xcite , banditsum @xcite and neusum @xcite are among the most commonly used baselines for extractive summarisation in the past three years. however, most of these were designed for short documents (cnn/dm). longformer @xcite ) is an adpatation of bertsum for longer documents. there are also summarisation approaches that are specific to the narrative domain @xcite @xcite . ## results ## discussion the main contribution of this piece of work is the generated idn-sum dataset. this is the first dataset for idn that shows different branches that are possible through an interactive story. idn is different from other forms of narrative text due to the presence of choice points that affect how the story unfolds. this dataset captures many different paths through such narratives. it is hence unique compared to other summarisation datasets because the high amount of overlapping text between data points. the dataset was created as a resource that enables us to investigate summarisation approaches for interactive and game narratives. it may also be used to study how summarisation models respond to small changes in text and target summary. capturing important differences between different playthroughs is a significant aspect of idn summarisation. idn is essentially a collection of linked literary documents. summarization of multiple linked literary documents has not been studied previously, although multi-document summarization and plot (literary) summarization have been addressed separately. unlike domains like news where multi document summarization @xcite has been studied, idn documents have a narrative structure and elements (plot, protagonist, emotions, etc) which influence the relative importance of sentences. the nature of differences between documents is different from domains like academic papers where comparative summarization has been studied @xcite . the differences are not solely topical and the links and link texts influences what is different between groups of documents. therefore, this would also be a useful resource to study new nlp problems like comparative plot summarisation. the dataset has 1250 playthroughs per episode and 8 episodes overall, but the code and jsons for the readerbot will also be made available on github foot_8 . this can be used to generate more playthroughs of the game, although they will need to be modified to adapt to different games. there are many types of idn, both in terms of types of text and narrative design. while it is a limitation of this dataset that only one type of idn is included, it takes a step towards making resources available for exploration of some aspects of idn summarisation. we also report and analyse performance of some standard baseline approaches quantitatively and qualitatively. in spite of a smaller number of data points, much longer input documents and difference in domain from cnn/dm, summarunner seems to scale for these longer documents and work well across domains, when considering rouge scores. however, manual inspection reveals several drawbacks of the rouge metric in terms of accurately reflecting summary quality. this is in line with findings from similar experiments performed on summscreen in @xcite where new entity centric evaluation metrics are pro-posed. finding a good evaluation metric to assess summary quality is a known challenge, even in case of the cnn/dm dataset @xcite . for this reason, evaluation strategies usually include a human evaluation step in addition to automated metrics like rouge. however, in the case of narrative datasets, due to the large source length and relatively large reference summaries, human evaluation is resource intensive when compared to datasets like cnn/dm and more subjective since it needs to account for subjective aspects like coverage of plot points. attempts to decrease subjectivity include strategies like judging the ability of the evaluator to answer questions about major plot points from the summary (lapata, 2021). however, interactive narrative summarisation needs to account for interactive elements in addition to plot elements and important differences between playthroughs. future work will augment this dataset with a similar list of plot points and interactive elements like decision points that can be used for evaluation. the human written summaries against which scores are calculated summarise the entire idn and represent variations between playthroughs through sentences like : "if chloe goes along with rachel, she will be suspended. if chloe takes the blame for rachel, she will be expelled." this means that in a playthrough where chloe chose to take blame, there will be keywords relating explusion and in other branches, those relating suspension, but neither branch will have both. hence, even if the model works perfectly, it cannot get a perfect rouge score since some of the keywords in the abstractive summary will not be present in that playthrough. paraphrasing also causes some keywords to not be present in the original text. while these are drawbacks of the automatic evaluation, these scores give insight into relative performance of models and can be put into context by considering the score of the oracle as the upper bound and random-n as the lower bound. these issues are mitigated by also providing rouge f1 scores against the oracle extractive reference summaries in appendix b. the qualitative analysis of the oracle summaries also reveals some characteristics of narrative datasets that makes it worse if only keyword overlap is considered. news articles are structured differently to narrative text and are more likely to have summary sentences in the original text that capture the important information. important in-formation in narrative datasets are spread across several sentences. presence of short sentences and sentences in utterances being broken up to include narration-like sentences in between screenplay-like text produces extracts that have high keyword overlap but are not useful or coherent. while scenelevel summaries might be too large, selecting multisentence extracts instead of single sentence extracts might alleviate this issue to some extent. additionally, sentences with many character names or short sentences with character names get high rouge scores even if they do not contain any relevant information because the reference summary contains them. a version of rouge that gives lower weights to words that are common in the document like the weighted rouge from @xcite might do better in this regard. this study indicates that several aspects of the summarisation approaches that are commonly used for cnn/dm need to be re-examined and potentially redesigned for narrative and interactive narrative datasets, including: 1) the size and nature of extracts 2) automatic methods for conversion of abstractive summary to extractive summary 3) evaluation metrics and methodology. hopefully, this dataset can help aid future research in these directions. ## conclusion in this paper, we present the first summarisation dataset for interactive narratives. this was done by collecting fan made transcripts and abstractive summaries from fandom and generating simulated playthroughs by assuming different combinations of choices. annotation for extractive summarisation were created automatically from the abstractive summaries through greedy selection of extracts that maximised the rouge score with the abstractive summary. even though narrative datasets have less data and longer text, summarunner with document truncation set to 3000 appears to scale when considering rouge scores. however, a qualitative analysis of generated summaries revealed several short comings in the rouge metric and oracle summaries suggesting that even though rouge scores for narrative datasets are comparable to cnn/dm, the summaries are not on the same level qualitatively. we hope that this dataset can be used for future research into better annotation methods, evaluation strategies. and summarisation approaches for interactive digital narratives.