Long Document Summarization in a Low Resource Setting using Pretrained Language Models (2021)

Abstract: Abstractive summarization is the task of compressing a long document into a coherent short document while retaining salient information. Modern abstractive summarization methods are based on deep neural networks which often require large training datasets. Since collecting summarization datasets is an expensive and time-consuming task, practical industrial settings are usually low-resource. In this paper, we study a challenging low-resource setting of summarizing long legal briefs with an average source document length of 4268 words and only 120 available (document, summary) pairs. To account for data scarcity, we used a modern pre-trained abstractive summarizer BART, which only achieves 17.9 ROUGE-L as it struggles with long documents. We thus attempt to compress these long documents by identifying salient sentences in the source which best ground the summary, using a novel algorithm based on GPT-2 language model perplexity scores, that operates within the low resource regime. On feeding the compressed documents to BART, we observe a 6.0 ROUGE-L improvement. Our method also beats several competitive salience detection baselines. Furthermore, the identified salient sentences tend to agree with independent human labeling by domain experts.

## introduction and related work text summarization is the task of generating a smaller coherent version of a document preserv- * * equal contribution ing key information. typical abstractive summarization algorithms use seq2seq models with attention @xcite , copy mechanisms @xcite , content selection @xcite , pointer-generator methods @xcite and reinforcement learning @xcite . these methods perform well in high resource summarization datasets with small documents such as cnn/dailymail @xcite , gigaword @xcite , etc. however, summarization over long documents with thousands of tokens is a more practically relevant problem. existing solutions focus on leveraging document structure @xcite or do mixed model summarization involving compression or selection followed by abstractive summarization @xcite . however, these methods require large amounts of training data. low resource settings are common in real world applications as curating domain specific datasets especially over long documents and on a large scale, is both expensive and time consuming. a human summarizing a long document would first understand the text, then highlight the important information, and finally paraphrase it to generate a summary. building on this intuition, we present a low-resource long document summarization algorithm (section 2) operating in 3 steps: ## dataset and approach to mimic the real world scenario of summarization over long domain-specific documents, we curate 120 document-summary pairs from publicly avail-able amicus briefs, foot_0 thus simulating the legal domain. the source contains detailed arguments that the court should consider for a case; the target summarizes them. as shown in dataset # (s, t ) avg. |s| avg. |t | cnn/dm 312,084 781 56 amicus 120 4,268 485 to tackle this low resource setting, we use the state-of-the-art abstractive summarizer bart @xcite , pretrained on a dataset from a related domain (cnn/dm). since bart was trained on short documents, it truncates documents longer than 1024 subwords. hence, instead of feeding the whole source document as input to bart, we feed salient sentences extracted using a salience classifier. our salience classification dataset is built using a novel method which grounds summary sentences to sentences in source with language model perplexity scores. our approach (figure 1 ) resembles the extract-then-abstract paradigm popular in prior work @xcite @xcite . ## conclusion we tackle an important real-world problem of summarizing long domain-specific documents with much less training data than previous works. we propose an extract-then-abstract pipeline which uses gpt-2 perplexity and a bert classifier to estimate sentence salience. this sufficiently compresses a document, allowing us to use a pretrained model (bart) to generate coherent & fluent summaries.