Long Input Dialogue Summarization with Sketch Supervision for Summarization of Primetime Television Transcripts (2022)

Abstract: This paper presents our entry to the CreativeSumm 2022 shared task. Specifically tackling the problem of prime-time television screenplay summarization based on the SummScreen Forever Dreaming dataset. Our approach utilizes extended Longformers combined with sketch supervision including categories specifically for scene descriptions. Our system was able to produce the shortest summaries out of all submissions. While some problems with factual consistency still remain, the system was scoring highest among competitors in the ROUGE and BERTScore evaluation categories.

## introduction this paper represents our submission to the cre-ativesumm 2022 shared task, which itself was subdivided into four distinct summarization tasks and consists of summarization of (i) chapters from novels, (ii) movie scripts, (iii) prime time television scrips and (iv) day-time television scripts. our system focused on summarizing prime-time television show transcripts for task (iii) and aims at producing a brief description for a single episode of its main developments based on the underlying episode script. our system has been trained on the summscreen forever dreaming (fd) dataset @xcite that was released as part of the shared task to produce automatic abstractive summaries of tv show screenplays. the structure of the data and its strong stylistic reliance on dialogues allows us to construct the problem of tv screenplay summarization as a dialogue summarization problem. to solve this, we use a dialogue summarization architecture of abstractive dialogue summarization with sketch supervision introduced by @xcite as a base architecture and adjust it to process large inputs of up to 16384 tokens and handle scene descriptions, which is one important characteristic of this data and makes it different from typical dialogue data. with our architecture, we achieve results which strongly outperform the baseline end-to-end models and result in better performance than our competitors on word-and context-based metrics on the creativesumm 2022 shared task. ## dataset the summscreen dataset contains television show screenplays -one screenplay per episode -and human-written summaries which provide a short description of the story line of the given episode @xcite . it consists of two parts: summ-screen forever dreaming (fd), which contains prime tv shows screenplays, and summscreen tv megasite (tms), consisting of daytime tv show screenplays. in this work, we focus on summscreen-fd as our data source for model training and evaluation. our training data contains 4008 instances of tv series episodes with corresponding hand-written summaries as gold labels, for validation we use additional 337 instances and for testing 459 examples. an example snippet from the dataset can be found in an important characteristic of this data is that the input length of the screenplays much exceeds the typical input length that standard transformerbased language models are designed to tackle: in terms of word count our train data, for example, has on average 7587 words, with a smallest screenplay having 1934 words and the longest one 21435 words. outputs in the training data are much shorter, with average of 111 words, where the longest summary has 822 words and the shortest summary being only 8 words long. this requires an approach which would be capable of processing large inputs, grasping temporal relations and references on long distances, and squeezing them into concise outputs. such large input size can in part be attributed to another characteristic of summscreen-fd, which makes it different from summscreen-tms, which is that it also contains descriptions about environments or characters and their feelings, similar to scene setting descriptions. we exploit this characteristic in our method, as we describe in more detail in chapter 4. ## method to tackle the task of summarizing tv show screenplays, we construct it as a dialogue summarization problem and use a dialogue summarization model proposed by @xcite . before processing the dialogues, a preprocessing pipeline also provided by the authors is applied, which first leo: piper? piper: hm? leo: what are you doing? (leo sits up. piper walks out of the nursery carrying a packet of diapers.) piper: i'm putting the diapers back where they belong, that is what i'm doing. (she puts the diapers on a shelf.) as mentioned in section 2.1, this method introduces the idea of constructing a summary sketch as a step prior to predicting the summary itself, which is one of the main features of this architecture. the summary sketch consists of the keywords extracted by applying syntax-driven sentence compression method @xcite combined with constituency parsing with a self-attentive encoder @xcite , as implemented in the berkeley neural parser foot_2 . the relations between the keywords are modelled according to a predefined utterance intent classification model @xcite . this idea makes this method suitable to our long inputs and intricate conversation structures with scene breaks and important information spread over several not necessarily adjacent utterances, because it helps capture only the core information, excluding the character or plot development turns, irrelevant to the summaries. this dialogue processing pipeline is also easily adjustable due to the flexibility of the predefined utterance intent model. in the original model, the summary sketches are built after the five ws principle, classifying the utterances as to their intent of "why", "what", "where", "when" or "confirm". utterances which do not fall under any of these categories are marked as "abstain". we extend this approach by incorporating the scene setting information by introducing the additional intent of "scene". subsequently, in our pipeline we create summary sketches based on the intent information for each line from the transcripts and the keywords which could be identified, removing noise from the data (see tables 2 and 3 for examples). we limit the amount of utterances which are considered as a part of the sketch to 20 in order to stay below the maximum output limit of 1024 tokens. the gold label is then concatenated to the sketch separated by the token tldr, which together serve as a label which the generator is trained to predict (for an example, see the training has been performed on a single nvidia a40-48c gpu with 48 gb ram with training batch size of 2 and gradient checkpointing. we performed a few experiments, tuning the learning rate and adjusting the maximum epoch size. we could achieve the fastest and most reliable training process training at initial learning rate of 5e-5 (with adam optimizer), maximum epoch size of 40. early stopping ended the training after reaching the best model after 19 training epochs and model not improving for 5 subsequent epochs. the training script and instructions can be found on github foot_4 . figure 1 : workflow of the used framework to summarize tv show transcripts. it processes transcripts (source) and gold label summaries, modelling intent of the source and extracting key phrases based on the overlap with the gold summaries, and creates summary sketches (e.g. ## results the winning model that we then used for producing summaries for the blind test data was determined based on the validation set performance employing early stopping. we include evaluations of the validation rouge-1 and rouge-l scores across all training epochs in figure 2 . the best performing model reached a rouge-1 performance of 34.45 and rouge-l of 28.05 respectively in epoch 19. based on this we chose this model for evaluation on the test set. test set evaluation shows generally lower, but similarly promising results. the litepyramid evaluation shows good performance compared to other competitors, but here we are witnessing that our system is not able to outperform the led_1024 baseline model. lastly in the summac zs scores our system, unfortunately, scored comparatively low, suggesting problems with factual consistency of the generated outputs. to investigate this problem of factual consistency, we compared outputs of the system manually with reference scripts and found this problem to be true. thus, improving factual consistency would be a vital step for improving the overall model in the future. we present an example of the generated summaries of our system in the number of utterances that go into the sketch for each script is another hyperparameter, that can be freely set and tuned by the developer. to further explore the effect of the sketches in the overall pipeline we conducted experiments varying the number of utterances for each individual sketch. utilizing more utterances has however not lead to any significant changes in performance on the dev set. due to the nature of the shared task, we could test set instance ## conclusion in this paper, we presented our submission to the creativesumm 2022 shared task for tackling the problem of summarizing television script based on the summscreen forever dreaming dataset. our system is based on previous work by @xcite and extends the utterance and sketch categories particularly for the task of screenplay summarization. we have shown that this method can improve over baseline led summarization significantly and have achieved good rouge and bertscore performance, which were the highest among submitted systems. at the same time, this method was able to produce the most concise summaries out of the field. the biggest issue of our system is factual consistency, often mixing up specific details and actors in the summarization part, thus creating seemingly good summaries but with factual errors, which would be hard to spot without actual knowledge of the underlying text. improving factual consistency would, therefore, be an important follow-up step in further developing this approach. similarly, the preprocessing of sketches could be optimized by reevaluating utterance categories and trying to further specify relevant utterances for the task of script summarization with less of a focus on spoken dialogue alone to improve the base performance even more. we are confident that with an improvement of the factual consistency our system will be able to also score higher in a human evaluation process, where such discrepancies are weighed much higher than in a pure word-level or representation-level approach.