CREATIVESUMM : Shared Task on Automatic Summarization for Creative Writing (2022)

Abstract: This paper introduces the shared task of summrizing documents in several creative domains, namely literary texts, movie scripts, and television scripts. Summarizing these creative documents requires making complex literary interpretations, as well as understanding non-trivial temporal dependencies in texts containing varied styles of plot development and narrative structure. This poses unique challenges and is yet underexplored for text summarization systems. In this shared task, we introduce four sub-tasks and their corresponding datasets, focusing on summarizing books, movie scripts, primetime television scripts, and daytime soap opera scripts. We detail the process of curating these datasets for the task, as well as the metrics used for the evaluation of the submissions. As part of the CREATIVESUMM workshop at COLING 2022, the shared task attracted 18 submissions in total. We discuss the submissions and the baselines for each sub-task in this paper, along with directions for facilitating future work.

## introduction contemporary research in text summarization systems has focused mainly on a select few domains such as news, scientific and legal articles. while summarizing these domains is important, the corresponding documents are limited in length and the diversity of the discourse structure. there is a large body of creative texts available on the web that pose greater challenges for text summarization in nlp. these creative documents like books, movie scripts, and television scripts are usually written by 'subject matter experts', are substantial in length and contain complex inter-dependencies between characters in the plotline @xcite @xcite . summarization of such text requires understanding many different genres, and offers the possibility of improving creating writing platforms @xcite @xcite . such creative documents are often accompanied by a synopses, short-description or a summary, which allows readers to gauge their interest in the corresponding artifact. summaries of creative texts are widely used by students as study guides, and could be useful to experts that need to grade the quality of these documents for a particular audience. furthermore, researchers may be interested in using the challenges posed by these creative texts, to identify the limitations of large language models at understanding complex discourse structure. to further summarization research of creative writing, we identified four particular domains, each with their own set of challenges. building off of datasets recently released in the community, we develop a shared task, composed of four sub-tasks, and encouraged participants to further research in this direction. for sub-task 1, we focus on summarization of chapters from literary texts like books, novels and stories. we curate a combined version of booksum @xcite and novelchapters @xcite for this purpose. for sub-task 2, we use the scriptbase dataset @xcite , which pairs movie transcripts with their corresponding wikipedia summaries. sub-tasks 3 and 4 both relate to summarization of transcripts for tv shows, both derived from summscreen @xcite , which contains two non-overlapping set of tv shows, from different sources. the two sources provide transcripts for two different domains -prime time tv shows and daytime 'soap operas', respectively forming the sub-task 3 and 4. we share the instructions to access the data for each of the four sub-tasks and associated scripts on our github repo foot_0 . we detail the process of evaluation of these subtasks, presenting the metrics as well as the results from the submissions by the shared-task participants alongside several baseline, long-document summarization models. we discuss ways to expand the sub-tasks to include more genres of creative writing for automatic summarization systems in the future. ## datasets in this section we describe the sources and preprocessing steps taken to curate the data from each sub-task of creativesumm. the data samples for each of the sub-tasks are available on the shared task github repo. ## evaluation we received 18 submissions in total for our shared task, with two, eleven, three and two submissions for shared tasks 1, 2, 3 and 4 respectively. we describe the metrics used for evaluating the submitted outputs. participants had 12 weeks to sign up for the shared task and train their models, before we released the test set(s) for the different sub-tasks. we allowed another week after releasing the test set for the submission of system outputs. we encouraged multiple submissions for each sub-task as long as they reflected different models or approaches. ## related work although summarization of newswire text has dominated research for over a decade, there have been key efforts at encouraging summarizing research in other domains. @xcite introduced a workshop on summarizing different genres, media and languages, with the aim of defining new tasks and corporas in these domains. @xcite and @xcite highlight the need for summarization of creative documents, by arguing that summaries may serve as a "lubricant" in the market for creative content, making it easier for consumers to decide which information to consume. tvrecap @xcite introduced a story generation task that requires generating detailed recaps from a brief summary and a set of documents describing the characters involved in the episode. the dataset contains 26k episode recaps of tv shows gathered from fan-contributed websites. past work has demonstrated the importance of character analysis for understanding and summarizing the content of movie scripts @xcite . with increased focus on characters in creative documents like movies and fictional stories, the task of generating character descriptions using automatic summarization been proposed recently @xcite . the accompanying dataset contains 1,036,965 fictional stories and 942,218 summaries. while there has been prior work on summarizing short stories @xcite , more recent methods in summarizing books utilize the hierarchical structure of documents for understanding the complex inter-dependencies in the text. @xcite incorporate human feedback along with recursive task decomposition, using summaries of small sections of the book to produce an overall summary at inference time. @xcite propose a novel top-down and bottom-up inference framework, which is effective on a variety of long document summarization benchmarks, including books. ## discussion this workshop proposes the task of summarizing documents containing creative textual content. we consider sub-tasks focusing on the summarization of book chapters, movie scripts and transcripts from tv shows. each sub-task highlights key challenges in automatic summarization of creative texts in a different genre. we also discuss how other efforts in the literature have attempted to approach this area of research. in its first version as part of coling 2022, our sub-tasks attracted 18 submissions in total. we present the results from these submissions, along with some baselines and compare the performance of the different systems. summarization of creative texts opens the door for the development of computer-based tools to aid authors and marketers in creative industries @xcite . automatic summaries can serve as an important component of screenwriting and book-writing tools, helping grade the quality and gauge interest amongst the consumers @xcite . advances in creative summarization will also assist and augment research in other related tasks and areas. cues from textual summaries and automated content analysis in movie scripts can be helpful in creating movie-image summaries @xcite . there are two main directions to improve efforts in this direction in the future -incorporating more datasets/sub-tasks that relate to creative summarization, and improving metrics/strategies to effectively evaluate these systems. other sources on the web, such as the one used by @xcite can be used for harnessing datasets for summarizing creative texts. similarly, our evaluation scheme can be expanded to include more entity-centric metrics @xcite , which can be crucial for identifying the presence of key characters in summaries of creative texts. jeff wu, long ouyang, daniel m ziegler, nisan stiennon, ryan lowe, jan leike, and paul christiano. 2021. recursively summarizing books with human feedback. arxiv preprint arxiv:2109.10862. shiyue zhang and mohit bansal. 2021. finding a balanced degree of automation for summary evaluation. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 6617-6632, online and punta cana, dominican republic. association for computational linguistics. tianyi zhang, varsha kishore, felix wu, kilian q. weinberger, and yoav artzi. 2020. bertscore: evaluating text generation with bert. in 8th international conference on learning representations, iclr 2020, addis ababa, ethiopia, april 26-30, 2020. openreview.net. weiwei zhang, jackie chi kit cheung, and joel oren. 2019. generating character descriptions for automatic summarization of fiction. in proceedings of the aaai conference on artificial intelligence, volume 33, pages 7476-7483.