Discrete Optimization for Unsupervised Sentence Summarization with Word-Level Extraction (2020)

Abstract: Automatic sentence summarization produces a shorter version of a sentence, while preserving its most important information. A good summary is characterized by language fluency and high information overlap with the source sentence. We model these two aspects in an unsupervised objective function, consisting of language modeling and semantic similarity metrics. We search for a high-scoring summary by discrete optimization. Our proposed method achieves a new state-of-the art for unsupervised sentence summarization according to ROUGE scores. Additionally, we demonstrate that the commonly reported ROUGE F1 metric is sensitive to summary length. Since this is unwillingly exploited in recent work, we emphasize that future evaluation should explicitly group summarization systems by output length brackets.

## introduction sentence summarization transforms a long source sentence into a short summary, while preserving key information @xcite . sentence summarization has wide applications, for example, news headline generation and text simplification. state-of-the-art sentence summarization systems are based on sequence-to-sequence neural networks @xcite @xcite , which require massive parallel data for training. therefore, unsupervised sentence summarization has recently attracted increasing interest. cycle-consistency approaches treat the summary as a discrete latent variable and use it to reconstruct the source sentence @xcite . such latent-space generation fails to explicitly model the resemblance between the source sentence and the target summary. randomly swapping two values in the boolean vector yields a new summary that is scored by an objective function that measures language fluency and semantic similarity. if the new summary increases the objective, this summary is accepted as the current best solution. rejected solutions are not depicted. @xcite propose a left-to-right beam search approach based on a heuristically defined scoring function. however, beam search is biased towards the first few words of the source. in this paper, we propose a hill-climbing approach to unsupervised sentence summarization, directly extracting words from the source sentence. this is motivated by the observation that humanwritten reference summaries exhibit high word overlap with the source sentence, even preserving word order to a large extent. to perform word extraction for summarization, we define a scoring function -similar to @xcite and @xcite -that evaluates the quality of a candidate summary by language fluency, semantic similarity to the source, and a hard constraint on output length. we search towards our scoring function by first choice hill-climbing (fchc), shown in figure 1 . we start from a random subset of words of the required output length. for each search step, a new candidate is sampled by randomly swapping a selected word and a non-selected word. we accept the new candidate if its score is higher than the current one. in contrast to beam search @xcite , our summary is not generated sequentially from the beginning of a sentence, and therefore not biased towards the first few words. due to the nature of the search action, our approach is able to explicitly control the length of a summary as a hard constraint. in all previous work, the summary length is weakly controlled by length embeddings or a soft length penalty @xcite @xcite . thus, the generated summaries by different systems vary considerably in average length, for example, ranging from 9 to 15 on a headline corpus (section 4.1). previous work uses rouge f1 to compare summaries that might differ in length. we show that rouge f1 is unfortunately sensitive to summary output length, in general favoring models that produce longer summaries. therefore, we argue that controlling the output length should be an integral part of the summarization task and that a fair system comparison can only be conducted between summaries in the same length bracket. our model establishes a new state-of-the-art for unsupervised sentence summarization across all commonly-used length brackets and different rouge metrics on the gigaword dataset for headline generation @xcite and on duc2004 @xcite . the main contributions of this paper are: • we propose a novel method for unsupervised sentence summarization by hill climbing with word-level extraction. text summarization. the task can be categorized by source text types, such as multi-document summarization @xcite @xcite and single-document summarization @xcite @xcite . traditional approaches are mostly extractive, i.e., they extract entire sentences from a document. recently, sequence-to-sequence (seq2seq) models have been used for abstractive summaries, where the system is able to synthesize new sentences @xcite @xcite @xcite . the copy mechanism @xcite in a seq2seq model can be viewed as word-level extraction in abstractive summarization @xcite . both state-of-the-art extractive and abstractive approaches are usually supervised. sentence summarization yields a short summary for a long sentence. @xcite and @xcite extract single words from the source sentence based on language model fluency and linguistic constraints. they search via dynamic programming with a trigram language model, which restricts the model capacity. the hedge trimmer method @xcite also uses hand-crafted linguistic rules to remove constituents from a parse tree until a certain length is reached. @xcite propose a supervised abstractive sentence summarization system with an attention mechanism @xcite , and they also introduce a dataset for headline generation derived from gigaword. foot_0 subsequent models for this dataset were also supervised and mostly based on seq2seq architectures @xcite @xcite . recently, unsupervised approaches for sentence summarization have attracted increasing attention. @xcite learn a denoising autoencoder and control the summary length by a length embedding. @xcite and @xcite use cycle-consistency @xcite to learn the reconstruction of the source sentence and return the intermediate discrete representation as a summary. @xcite use beam search to optimize a scoring function, which considers language fluency and contextual matching. our work can be categorized under unsupervised sentence summarization. we accomplish this by word-level extraction from the source sentence. ## proposed model given a source sentence @xmath0, x 2 , . . . , x n ) as input, our goal is to generate a shorter sentence @xmath1, y 2 , . . . , y m ) as a summary of x. we perform word-level extraction, in addition keeping the original word order intact. thus, y is a subsequence of x. our word-level extraction optimizes a manually defined objective function f (y; x, s), where the summary length s is predefined (@xmath2. in the remainder of this section, we will describe the objective function, search space, and the search algorithm in detail. ## evaluation framework in this section, we will describe the datasets, evaluation metrics, and a widely used baseline (called lead). additionally, we report the observation that the commonly used evaluation metric, rouge f1, is sensitive to summary length, preferring longer summaries. thus, we propose to group models with similar output length during evaluation for fair comparison. ## conclusion we proposed a novel word-extraction model for sentence summarization that generates summaries by optimizing an objective function of language fluency and semantic similarity. a hard length constraint is also imposed in our objective function. in a controlled experiment, our model achieves better performance than strong baselines on headline generation and duc2004 datasets.