S um T itles: a Summarization Dataset with Low Extractiveness (2020)

Abstract: The existing dialogue summarization corpora are significantly extractive. We introduce a methodology for dataset extractiveness evaluation and present a new low-extractive corpus of movie dialogues for abstractive text summarization along with baseline evaluation. The corpus contains 153k dialogues and consists of three parts: 1) automatically aligned subtitles, 2) automatically aligned scenes from scripts, and 3) manually aligned scenes from scripts. We also present an alignment algorithm which we use to construct the corpus.

## introduction as most written communication is held in the form of dialogues, the amount of dialogue data increases over time. this poses a requirement for efficient dialogue mining tools for information extraction, search, and natural language understanding. an attractive path towards efficient information search is a compact representation, e.g., in the form of summarization. although summarization methods could ease processing a large amount of textual data, few are applicable to dialogues. the main reason for this is that most of the datasets used for summarization models are completely different. probably, the most well known corpus for summarization, cnn/dailymail @xcite , comprises news articles, while other cover social media posts @xcite and web documents @xcite . deep summarization methods, both extractive @xcite and abstractive @xcite , show high performance for these datasets. dialogue data poses new challenges: first, a dialogue presents a conversation of two or more people, while news articles or social media posts are written by one person only. this means that multiple points of view can be expressed, and all of them need to be accounted for in the summary. second, the grammar and the structure of the utterances differ drastically: more personal pronouns and colloquial expressions are used. finally, the conventional sentence order is distorted: two consequent sentences may not be semantically related. these challenges limit the application of extractive summarization methods and push towards abstractive ones. this paper is based upon an early work in dialogue dataset construction, namely ami @xcite . ami corpus is small in size (141 dialogues) and does not allow the deep models training. in this paper, we try to overcome this significant flaw of ami by developing a new large-scale dataset, a thousand times larger. we gather movies subtitles and freely available synopses into a dataset we call "corpus for summarization of movie subtitles" (sumtitles). section 3 describes sumtitles in details. the construction of the sumtitles dataset is a challenging task, and we present an algorithm to create alignment for scripts in section 4. to compare sumtitles to the other summarization datasets, we introduce a new methodology, which is described in section 5. the results of the comparison are also provided there. we experiment with state-of-the-art summarization approaches to create summaries for the dialogues @xcite and therefore section 6 contains the description of the baselines and the results achieved for sumtitles dataset and state-of-the-art results for cnn/dailymail dataset for comparison. section 7 conclude the paper and outline the future research directions. the main contributions of this paper are the following. first, we present a novel dataset, sumtitles, which aims at dialog summarization (section 3) and a splitting algorithm which we used to construct the corpus (section 4). secondly, we use multiple state-of-the-art summarization models on the sumtitles dataset to establish baselines (section 6). last but not least, we propose a novel extractiveness coefficient, based on which we conduct the comparison of existing datasets and sumtitles (section 5). ## the sumtitles dataset following @xcite , we use movie scripts as the main source of data for corpus construction. the core concepts used for corpus construction are the following: the plot summaries, synopses, and casts are collected from the open sources, while for the subtitles and partially the scripts we use the existing datasets. sumtitles consists of three parts: 1) subtitles, 2) scripts, and 3) gold. subtitles part has only rough alignment between the whole movie and a plot summary, since there is no information about characters and a scene separation. scripts part comprises scenes which are automatically, but quite accurately, aligned with the synopses, most commonly a sentence per a scene. the last part, which we refer to as gold, is labeled by human experts for an alignment between scenes and synopses. the subtitles part is an extraction from the opensubtitles corpus (2018 version), which is described in @xcite . we use only subtitles in english, and among them, only those which have plot summary available. we additionally filter the subtitles for the movies and series, which are not present in scripts and gold parts. the subtitles in opensubtitles dataset do not contain character names and scene separators. thus we consider the whole subtitle to be a single dialogue of anonymous characters. the sample dialog accompanied with a movie plot is presented at fig. 1 . although the subtitle could be split into several pieces to produce multiple dialogues, in this case, a plot summary will be covered by the split dialogue only partially. the scripts part itself consists of two parts: the movie scripts available from the open sources foot_1 and friends series scripts described in @xcite . we consider the scripts for the movies which have synopses available only, while fortunately, friends series has a synopsis for each episode foot_2 . we developed an algorithm, allowing us to split a synopsis in an automatic fashion to produce the dialogues accompanied with their summaries derived from the synopsis. the detailed description of the algorithm is available in section 4. ## splitting a script a synopsis consists of the sentences, which we consider independent as each sentence describes a separate scene. to dampen the effects of this strong assumption, we develop an algorithm to split and join script scenes and sentences from a synopsis. the algorithm is presented as algorithm 1. pre-processing is conducted in several steps. firstly, we substitute scene speakers with cast character names, listed in the movie description. to this end, we estimate the similarity between scene speakers and character names by means of symbol-level n-gram jaccard similarity. next, we split each scene and each synopsis into separate sentences. then we embed every sentence to get the vector representations. we use the pre-trained universal sentence encoder model, described in @xcite 4 . to implement the algorithm, we use several functions and formulae, which are referred to in a similar manner. we use jaccard similarity to compare sentences and cosine similarity to compare vector representations. merge function is merging the input of scenes list into one scene which collects all the utterances, annotated synopsis sentences, and character lists from the input scenes in the order of appearance. lastsynid and firstsynid are returning the last and first (respectively) synopsis sentence indices from the ones annotated to an input scene. len returns length of an input set, append appends an input element to an input set. max, mean, union, intersect, sort, and argmax function according to their names. there are additional helper functions presented as algorithms: jaccardbest (alg. 3), bestsplit (alg. 5), annot (alg. 4). also there are two functions important for similarity computation: castsimilarity (alg. 6) and textsimilarity (alg. 7). the output of these two functions is a base for the splitting algorithm. their description could be found in the appendix a. also the splitting algorithm is using restricteddtw presented as algorithm 2. it is a modification of classic dynamic time-warping algorithm @xcite . in our case the restriction is that each cluster should contain exactly one synopsis sentence. if necessary, we add padding symbols to fill in the scenes that are speech free and do not have any utterances and actual sentences. the splitting algorithm (alg. 1) has several hyper-parameters: α, β, γ, δ, which are the weight coefficients for different similarity measures computed on the input data. these hyper-parameters are chosen based on the algorithm performance on the held out gold part. ## comparison to the other datasets there are two aspects which could be considered for comparison: the size of the dataset and its extractiveness. the size of the dataset could be measured in different measurement units (number of speakers, tokens, utterances, summary sentences and summary tokens). the collected statistics for the datasets are presented in the we introduce a compositional approach to define what extractiveness is. the existing approaches, such as @xcite , are evaluating different aspects of the extractivity itself such as coverage and fragment. our goal is to capture the complex phenomenon. we use already existing approaches and extend them to achieve the final extractiveness coefficient (ec). ## experiments to better understand our dataset's properties, we evaluate several current summarization models on it, accounting for both extractive and abstractive approaches. we evaluate the baseline model in multiple settings: ## conclusion we target creating a dataset, which will show the limitations of previously presented summarization datasets, which seem to borrow a lot from the original texts. we presented sumtitles dataset, which on the one hand, is significantly larger than the previous low extractive ami/dialsum datasets. on the other hand, sumtitles is comparable in size with recent abstractive datasets, such as cnn/dailymail, which are highly extractive. to compare the summarization datasets, we presented a methodology for extractiveness evaluation. the alignment of scripts and summaries proved to be a challenging task that we could solve with a specialized algorithm. this algorithm could be used to extend the current work and be adopted to other long texts to produce a split in semantically coherent units to facilitate training.