Unsupervised Single Document Abstractive Summarization using Semantic Units (2022)

Abstract: In this work, we study the importance of content frequency on abstractive summarization, where we define the content as “semantic units.” We propose a two-stage training framework to let the model automatically learn the frequency of each semantic unit in the source text. Our model is trained in an unsupervised manner since the frequency information can be inferred from source text only. During inference, our model identifies sentences with high-frequency semantic units and utilizes frequency information to generate summaries from the filtered sentences. Our model performance on the CNN/Daily Mail summarization task outperforms the other unsupervised methods under the same settings. Furthermore, we achieve competitive ROUGE scores with far fewer model parameters compared to several large-scale pre-trained models. Our model can be trained under low-resource language settings and thus can serve as a potential solution for real-world applications where pre-trained models are not applicable.

## introduction summarization is a task involving compressing a longer text into a shorter version while preserving the salient information in the original text. when given article-summary pairs, supervised models are able to learn corresponding implicit relationships, for example, where to focus or what to preserve. however, a lack of sufficient training pairs is a common issue in real-world applications. creating such high-quality training pairs can be costly. although large pre-trained models for language generation or summarization may require less data for fine-tuning, they are often trained on english corpus only (e.g., @xcite @xcite and thus are not suitable for low-resource languages. therefore, we seek the possibility of unsupervised summarization methods. our idea is to utilize the frequency of contents in the source text. intuitively, we expect some specific contents to be included in a summary if they frequently occur in the source article. a similar concept of "content units" was first proposed by @xcite . they manually labeled the text by identifying similar text segments to form a content unit, where the contributing text segments of a content unit should have similar semantic meanings. in their results @xcite , of the top 5 most frequent content units in the source documents, 96% appear in a human summary, and high percentages of 92 and 85 are observed for the top 8 and top 12 most frequent content units across 11 input sets. their observation shows that content unit frequency can provide huge hints as to whether a specific unit of content will be selected as a part of a humanwritten summary and therefore supports our idea. we also provide our statistical results on the recent summarization dataset cnn/daily mail @xcite in appendix a.2. instead of manually labeling content units like @xcite , we divide and enumerate all text spans with a fixed-size sliding window. here, we refer to the divided text spans as "semantic units" (sus), as we expect each semantic unit to contain brief semantic concepts in itself. we then argue that a refined summary should at least contain the semantic units frequently occurring in the original articles since the high-frequency semantic units should be the topic or contain key descriptions. in addition, frequency information alone should be possible to retrieve from source documents only. in this work, we propose a model that automatically learns semantic unit frequency. the learned frequency information is then used to discriminate salient parts in source documents for abstractive summarization. in our proposed method, which is shown in figure 1 , the training process is divided into two stages. in the first training stage, our model learns to predict the masked tokens based on the partially masked semantic units. this stage mimics the masked language modeling objectives used in the pre-trained language models @xcite @xcite . in the second training stage, the training goal of the model is to generate fluent text based on the given semantic units. we train the model to reconstruct the original articles in this stage; thus, no human-written summaries are used during training. in the inference stage, semantic unit frequency is obtained using the attention mechanism, which helps the model decide how much to focus on the semantic units when generating text. we first let the model generate text based on all semantic units in a given article and record the attention weights for each semantic unit. the recorded attention weights are used to assign weights to the semantic units. the weights are considered the semantic unit frequency since they represent how much the model has focused on each semantic unit when reconstructing the original article. the weighted semantic units are used to filter the sentences in the source text, and the corresponding weighted semantic units are provided to the decoder to generate a sequence. the generated sequence is considered the final summary. here, we list our contributions: first, our experiments prove that our proposed model discriminates semantic units by frequency and generates summaries from them. second, our model parameters are far fewer than many other pre-trained models, but we can still achieve competitive rouge scores. finally, no single summary is used in our training and inference process; therefore, our proposed method is suitable for real-world applications where human-written summaries are rarely accessible. our code is publicly available at https://github.com/ikmlab/uassu . ## method we briefly introduce the model structure and semantic unit construction in section 3.1. next, in section 3.2, we divide our training strategy into two stages and describe them separately. finally, we explain how we leverage the learned frequency information for unsupervised text summarization in section 3.3. the overview of training and inference stages is also shown in figure 2 . ## conclusion in this work, we propose an unsupervised abstractive summarization model using semantic units. the frequency of semantic units helps determine whether a specific content is more likely to be included in a human-generated summary. our model learns to discriminate semantic units from the source articles by frequency through the proposed two-stage training and the inference workflow. the proposed model can achieve competitive rouge scores without paired data or pre-trained models compared to the large-scale pre-training methods and the methods under the same unsuper-vised settings. our method is a potential solution for real-world scenarios where directly applying pre-trained models or collecting data with the lead bias characteristic is infeasible. since constructing semantic unit embeddings is similar to making span representations, we experiment with three span aggregation methods ( we tested different sliding window sizes c of 5, 7, and 9 when constructing semantic units. this range was determined considering two reasons: it was hard to form a basic meaning (e.g., a subject, an object, and a verb) with only three tokens where the bert subword-level tokenizer @xcite was used in our experiments. furthermore, there are 10.42 tokens, on average, in a clause in the cnn/daily mail training set. a larger sliding window size results in slightly fewer semantic unit embeddings for each article, and the number of semantic units sharing the same tokens also increases. the results are shown in