Abstractive Document Summarization with Summary-length Prediction (2023)

Abstract: Recently, we can obtain a practical abstractive document summarization model by fine-tuning a pre-trained language model (PLM). Since the pre-training for PLMs does not consider summarization-specific information such as the target summary length, there is a gap between the pre-training and fine-tuning for PLMs in summarization tasks. To fill the gap, we propose a method for enabling the model to understand the summarization-specific information by predicting the summary length in the encoder and generating a summary of the predicted length in the decoder in fine-tuning. Experimental results on the WikiHow, NYT, and CNN/DM datasets showed that our methods improve ROUGE scores from BART by generating summaries of appropriate lengths. Further, we observed about 3.0, 1,5, and 3.1 point improvements for ROUGE-1, -2, and -L, respectively, from GSum on the WikiHow dataset. Human evaluation results also showed that our methods improve the informativeness and conciseness of summaries.

## introduction current abstractive summarization models mostly utilize pre-trained language models (plms) @xcite @xcite @xcite . abstractive document summarization requires an encoder to determine the important parts in an input text and a decoder to output a non-redundant summary of the appropriate length relevant to the input. thus, the characteristics required for an abstractive summarization model differ from those required as a language model, and are not usually considered in the pre-training for plms @xcite @xcite . hence, we need to fine-tune a plm with a summarization dataset to treat it as an abstractive summarization model. unlike training a randomly initialized model, this fine-tuning maintains and inherits the parameters learned as an original language model. therefore, to learn an abstractive summarization model by fine-tuning a plm, it is necessary to suppress its characteristics as a language model while enabling it to learn the unique properties of abstractive summarization. for this purpose, we propose two regularization methods for fine-tuneing a plm to learn abstractive summarization. figure 1 shows an overview of our methods. the first method is a regularization method that uses the encoder's hidden states to predict the length of an output summary. when the length is not given for a summary to be generated, we believe it is difficult to determine what volume of important key contents to select from the original document. thus, fixing the length for a summary can make it easier to select key contents for it. we think humans can also create more informative and concise summaries when a summary length is given. the system should also be better trained for selecting key contents in the original document for a summary in case when it can be provided with the length of the summary. the second method provides the decoder with the length predicted by the first method and enables it to learn to output a summary of the length. in addition to regularizing the training of the decoder, this method reduces the search space by searching only for summaries of the appropriate length during generation, and so it is expected to produce a concise and informative summary. although there have been studies on adjusting the output length of summaries, they have focused on controlling the output length for a given desired length @xcite @xcite @xcite . 1 we incorporate a target-length prediction task to the encoder side and then inject the predicted length to the decoder side to generate the final summary. in an evaluation on the wikihow, nyt, and cnn/dm datasets, our methods improve the rouge scores of bart with appropriate lengths of summaries. on the wikihow dataset, the performance improvement reached about 3.0, 1,5, and 3.1 points for rouge-1, -2, and -l, respectively, from gsum. human evaluation results also showed that our methods enable the fine-tuning for a plm to generate informative and concise summaries. our contributions are as follows: (1) we propose a regularization method that uses the encoder's hidden states by predicting the length of a summary. ## our methods we apply our regularization methods to a transformer-based @xcite plm to generate a summary from a given document. ## related work in summary length control, previous work mostly focuses on controlling models for generating summaries with a predefined length @xcite @xcite @xcite . our work is novel because it enables a model dynamically predicts the appropriate summary length from the input text without relying on any predefined length. from the viewpoint of regularization, we can see such a regularization term like l len in recent works of summarization tasks. @xcite ; @xcite in sentence compression and @xcite in extractive document summarization incorporate dependency tree information into the attention @xcite . @xcite integrate extractive and abstractive summarization. matchsum @xcite considers the semantic similarity between a document and its extracted summary. brio @xcite takes multiple similar abstractive summaries into account by contrastive learning in sequence-to-sequence @xcite . different from these works, our approach focuses on summary lengths through l len and can be incorporated into these works by adding l len to their loss function. ## conclusion to fine-tune a pre-trained language model for abstractive document summarization, we proposed a regularization method that uses the encoder's hidden states to predict the length of an output summary. we also proposed lfpe, that focuses on generating a summary with a given target length while keeping pre-trained information of the transformerbased model. we used lfpe to regularize the decoder during training to generate a summary with the predicted length. automatic evaluation results showed that the proposed methods enable bart to generate summaries of appropriate lengths while improving rouge scores. human evaluation results also showed that the proposed methods enable bart to generate more informative and concise summaries.