COLING 2022 Shared Task: LED Finteuning and Recursive Summary Generation for Automatic Summarization of Chapters from Novels (2022)

Abstract: We present the results of the Workshop on Automatic Summarization for Creative Writing 2022 Shared Task on summarization of chapters from novels. In this task, we finetune a pretrained transformer model for long documents called LongformerEncoderDecoder which supports seq2seq tasks for long inputs which can be up to 16k tokens in length. We use the Booksum dataset for longform narrative summarization for training and validation, which maps chapters from novels, plays and stories to highly abstractive human written summaries. We use a summary of summaries approach to generate the final summaries for the blind test set, in which we recursively divide the text into paragraphs, summarize them, concatenate all resultant summaries and repeat this process until either a specified summary length is reached or there is no significant change in summary length in consecutive iterations. Our best model achieves a ROUGE-1 F-1 score of 29.75, a ROUGE-2 F-1 score of 7.89 and a BERT F-1 score of 54.10 on the shared task blind test dataset.

## introduction condensing long novel chapters into succinct and easy to digest summaries could be helpful as an informative bookmark to serve as a reminder of what happened in the last read chapter. this is much harder than other summarization tasks like summarizing news articles @xcite @xcite or legal @xcite and scientific documents @xcite . the reason for this is two fold. firstly, the importance of automatic summarization systems for these tasks is diminished by the presence of article headlines, highlights or abstracts, as well as the length of the text to be summarized which is limited to a few hundred words to a few pages. secondly, due to shorter text length and fact heavy content, there is no scope for extensive paraphrasing in the summaries. this is also due to short ranged causal and temporal dependencies and absence of convoluted plot lines. on the other hand, the task of summarizing chapters from novels @xcite introduces all these additional challenges, including processing of long texts, abrupt changes in plot lines, dialogue and narration, and generation of highly abstractive summaries. we present a recursive summary of summaries approach inspired by @xcite , where we decompose the long novel chapters into paragraphs and summarize them separately, thereby reducing computational complexity and noise in the target summaries. this is also similar to the divide and conquer approach used in @xcite . these partial summaries are then combined to obtain an intermediate summary. this intermediate summary is then treated as the long text to be summarized and this process is repeated until either a final summary of a specified length is obtained or there is no significant change in summary length between consecutive intermediate summaries. the model used to generate these summaries is a pretained longformerencoderdecoder model foot_1 (beltagy et al., 2020) finetuned on paragraph alignments obtained from the novel chapters. the datasets used for finetuning are described in section 2 and the models are presented in section 3. we present our results and analysis in section 4. ## dataset some key challenges in long form summarization are computational constraints and limits on input length of pretrained models used for finetuning. to address these challenges, instead of using entire chapter to summary mappings, we use paragraph level alignments obtained for the novel chapters. the paragraph level alignments are computed between paragraphs extracted from chapters and individual sentences of chapter-level summaries, by leveraging paragraph-sentence similarity scores using a sentencetransformer @xcite and a stable matching algorithm as mentioned in the booksum paper @xcite . we use two datasets for finetuning, one containing just the paragraph alignments (we will refer to this as dataset 1) and another containing paragraph alignments along with a subset of the chapter to summary data with maximum chapter length constrained to 500 words (we will refer to this as dataset 2). the maximum chapter length of 500 words is chosen because the maximum encoder and decoder length for the models is set to 512 and this ensures that a very small percentage of the total number of examples exceeds the maximum token length of 512 after tokenization. before training, all chapter and summary text is cleaned by stripping away hyperlinks, multiple consecutive whitespaces and non ascii characters. the number of examples for training and validation splits for both datasets can be seen in ## models the reference summaries for the novel chapters are highly abstractive with high semantic and low lexical overlap. the novel chapters have long range causal and temporal dependencies that can be effectively captured by the self attention component in transformers @xcite , which enables the network to capture contextual information. however, the memory and computational requirements of self-attention grow quadratically with sequence length, making it very expensive for longer texts like novel chapters. longformer (beltagy et al., 2020) is a modified transformer with a self-attention operation that scales linearly with the sequence length, making it a lucrative option for processing long sequences. we use the led-base-16384 3 longformeren-coderdecoder model for finetuning, which is ini-3 https://huggingface.co/allenai/led-base-16384 tialized from bart-base foot_2 @xcite since both models share the exact same architecture. we finetune the pretrained led base model on the two datasets mentioned in section 2 for 10 epochs and evaluate on the validation split after every 3000 steps. model outputs are decoded using beam search with 2 beams and n-gram repetition blocking for @xmath0. the led config min and max length is set to 100 and 512 respectively, with a length penalty of 2.0, early stopping set to false and a batch size of 1 due to computational constraints. the maximum encoder and decoder length is set to 512. in addition to the usual attention mask, led can make use of an additional global attention mask defining which input tokens are attended globally and which are attended only locally, just as in the case of longformer. we follow recommendations of the paper @xcite and use global attention only for the very first token and we ensure that no loss is computed on padded tokens by setting their index to -100. we also disable gradient checkpointing and the caching mechanism to save memory. we use the rouge metric @xcite for evaluation during model training and validation. ## experiments and results we train two models on dataset 1 and 2 (referred to as model 1 and model 2 respectively) and choose the model with the best overall validation score for final submission for the shared task. the mid rouge f-1 scores on the validation set of dataset 1 and 2 for both models can be found in for the final submission for the shared task, the final summaries for input novel chapters are generated using the recursive summary of summaries method described in the previous sections. the novel chapters in the blind test dataset are divided into paragraphs not exceeding 400 words in length, with an overlap of one sentence per chunk. this means that the last sentence from the previous paragraph chunk becomes the first sentence of the new paragraph chunk. if addition of any sentence to a chunk exceeds the chunk size of 400 words, that sentence becomes a part of the next chunk. these chunks are then summarized separately and concatenated in a recursive fashion to get the final summary. we observe that the training data has a mean summary to chapter length ratio of 0.15 and a standard deviation of 0.20 (where length is considered to be number of words in the text). so, during the generation of summaries by the finetuned model, we keep the maximum predicted summary length to be 35% of the input chapter length i.e. mean plus standard deviation of the summary to chapter length ratio of training datasets. this means that the input text is decomposed into paragraphs and intermediate summaries are created by generating individual summaries of these paragraphs and concatenating them until summary length of atmost 35% of the input text is reached or consecutive intermediate summary lengths are within 200 words of each other. the final evaluation metrics of the best performing model i.e. model 1 on the shared task's blind test set can be found in