Two-Stage Movie Script Summarization: An Efficient Method For Low-Resource Long Document Summarization (2022)

Abstract: The Creative Summarization Shared Task at COLING 2022 aspires to generate summaries given long-form texts from creative writing. This paper presents the system architecture and the results of our participation in the Scriptbase track that focuses on generating movie plots given movie scripts. The core innovation in our model employs a two-stage hierarchical architecture for movie script summarization. In the first stage, a heuristic extraction method is applied to extract actions and essential dialogues, which reduces the average length of input movie scripts by 66% from about 24K to 8K tokens. In the second stage, a state-of-the-art encoder-decoder model, Longformer-Encoder-Decoder (LED), is trained with effective fine-tuning methods, BitFit and NoisyTune. Evaluations on the unseen test set indicate that our system outperforms both zero-shot LED baselines as well as other participants on various automatic metrics and ranks 1st in the Scriptbase track.

## introduction the goal of the creative summarization shared task 2022 (ascw@coling'22) is to automatically generate summaries based on long-form creative texts like literature, movie scripts, or tv screenplays. this task is encouraged by practical settings in part by the condition to reflect the information that is realistically available in realworld natural language generation task -realistic texts like movie scripts and plot summaries can be prohibitively long (see movie script and plot summary in figure 1 ). current dominant neural approaches to long document summarization @xcite @xcite mainly embrace neural sequence-to-sequence architectures consisting of an encoder-decoder setup where the entire input sequence is first encoded before decoding the output sequence autoregressively. while the encoderdecoder architecture is triumphant in natural language generation tasks @xcite @xcite , it is not without its challenges, some of which are exacerbated in this shared task. in particular, the challenges in this shared task stem from the inherent characteristics of the corpus, which consists of not only texts (i.e. scripts of 24k tokens) that are much longer than the context length of current state-of-the-art long document summarizers (e.g. 16k in longt5 @xcite ), but also requires that the decoder be exploited of its long-term attention capabilities to an extreme extent and generate up to summaries of 1k tokens. the excessively long decoding time made it impossible to experiment with different model architectures and perform extensive hyperparameter tuning for model selection. therefore, the optimization space during training time is severely limited; and the inference step becomes highly time-consuming for experimentation during the trial and error process. to this end, our system employs a two-stage procedure for movie script to movie plot summarization. in the first stage, we propose to heuristically extract sentences that effectively reduce the average input length from 24k to 8k tokens. next, we propose to improve upon the longformer-based encoder-decoder model (led) @xcite ) by coupling it with two effective fine-tuning methods, i.e., bitfit @xcite where only the bias-terms (0.09% of the parameters) of the model are being updated and noisytune @xcite where employs a matrix-level perturbation strategy to increase the variation amplitude of the parameters. int. church -later the leader, smiling opens his eyes and looks around the group. ## our approach our method is a two-stage summarization method where the 1st stage is a heuristic extraction method (sec. 2.1) and the 2nd stage is neural seq2seq summarization model (sec. 2.2). ## conclusion in this paper, we present the details of our system, which ranks 1st in the scriptbase track on various metrics, including rouge, bertscore, and n-gram diversities. we show that the proposed approach involving a two-stage solution results in competitive and efficient performance for longform text encoding and generation. in addition, we deliver analysis and ablation studies for the components within our proposed techniques, which allows us to draw further conclusions about decoding configurations and vocabulary sampling. lastly, we argue that more work can be done to speed up the model selection process, which impacts the model performance and model training and evaluation. based on the finding of the ablation study on beam size (sec. 3.6), all experiments use @xmath0. among the current sota models: bart, t5 and led, we obtain 6.3 and 8.6 and 8.7 on r-2 f1 scores respectively. for the performance, led outperforms bart and t5 by 2.4 and 0.1 points. the demand of high complexity attention computation in bart limits the maximum input length and thus fails to generalise well on the long-form movie summarization. on the other hand, employing to global sliding window attention, led does not need to shrink its input context length and greatly benefits from 16-bit mixed precision training by which take only 2 hours. in contrast, fine-tuning t5 requires 32-bit training which results in 8× slower than led at training phase. based on the performance and training efficiency advantages, we leverage led for the development of our long-form summarization system on creativesumm'22.