Efficient Unsupervised Sentence Compression by Fine-tuning Transformers with Reinforcement Learning (2022)

Abstract: Sentence compression reduces the length of text by removing non-essential content while preserving important facts and grammaticality. Unsupervised objective driven methods for sentence compression can be used to create customized models without the need for ground-truth training data, while allowing flexibility in the objective function(s) that are used for learning and inference. Recent unsupervised sentence compression approaches use custom objectives to guide discrete search; however, guided search is expensive at inference time. In this work, we explore the use of reinforcement learning to train effective sentence compression models that are also fast when generating predictions. In particular, we cast the task as binary sequence labelling and fine-tune a pre-trained transformer using a simple policy gradient approach. Our approach outperforms other unsupervised models while also being more efficient at inference time.

## introduction in general, the information content of text is correlated with its length. however, for a given text, a shorter version may still convey the essential information while preserving grammaticality @xcite . the definition of essential can change depending on the downstream application, thus models for text compression must be able to adapt based on information about the downstream task. sentence compression models have been used as sub-modules of text and speech summarization @xcite , for headline generation @xcite , subtitle generation @xcite , and summarizing emails @xcite . potential applications also include snippet generation and highlighting for social media, blog posts or search results. given a particular text compression task, relevant evaluation metrics and auxiliary models of compression quality may not be straightforward to formulate as well-behaved differentiable objectives that can be used with standard backpropagation. in addition, ground-truth examples may be difficult to obtain because the annotation task is difficult to fully specify, and metrics which capture different facets of compression quality, such as fluency and optimal sentence length, may be negatively correlated. even in the case where ground-truth examples are available, they are likely to represent only a subset of the possible outputs, so there is a risk of over-fitting or biasing models when relying solely upon a small amount of gold training data for optimization. recent unsupervised sentence compression approaches leverage powerful neural language models to directly optimize objectives such as fluency and faithfulness of compressed sentences, using discrete search strategies, without relying on ground-truth examples @xcite @xcite . however, these search-based methods are very inefficient at inference-time because the search must navigate through a large candidate space while recomputing expensive reward functions. to allow for flexible reward specification, while also enabling efficient inference, we design a simple and effective reinforcement learning (rl) setup: our model is initialized as an unsupervised pretrained language model with an untrained binary classification head (see figure 2 ), and the sentence compression task is framed as sequence labeling, with optimization via policy gradient using a suite of reward functions. sentences are compressed in an instantaneous, one-step fashion, similar to modern part-of-speech tagging or named entity recognition models. this approach simplifies the learning setup while also allowing for high throughput. according to quantitative evaluation on several summarization benchmarks, our approach shows similar or superior performance compared to search-based methods, while also being much faster at inference time. our approach to unsupervised extractive sentence compression has the following benefits: we review related work in section 2. section 3 formalizes the task. section 4 gives a detailed description of the model and reward functions. section 5 presents experimental results, and sections 6 and 7 provide analysis and discussion of our findings. ## task we focus on the specific task of summarizing a sentence by extracting a subset of its tokens in their original order. given an input sentence x consisting of n tokens @xmath0, x 1 , ..., x n ), we aim to produce a sequence of binary labels @xmath1, y 1 , ..., y n ) ∈ {0, 1} n , where each label indicates whether the corresponding input token should be included in the compressed version of a sentence. we further assume an objective function, or reward function r(x, y) that measures how well applying the labels y summarizes the original sentence x. for a particular x, the goal is to find argmax y r(x, y), without access to any groundtruth examples. in general, there are 2 n possibilities to shorten a sentence in this task. a fixed summary length l would reduce this to n l possibilities, peaking at @xmath2. we do not constrain our approach to a fixed length, but we compare it to search-based techniques that are constrained to the n l search space. ## experiments this section presents a detailed analysis and evaluation results for our proposed model. we name our model scrl (sentence compression with reinforcement learning). we make all code, model outputs and data available foot_1 . ## analysis 6.1 summary length and extraction regions the length and compression ratio of summaries produced by scrl is distributed around the desired values, with peakier distributions than in groundtruth summaries (examples in figure 4 ). hc produces exactly the desired value whenever possible, due to the enforced constraint for length or ratio. figure 5 shows how scrl and hc extract tokens from different relative positions within source sentences. scrl has a higher tendency to extract early tokens. we hypothesize that this is a reliable high-reward strategy discovered during training, considering that a milder form of the lead-bias also shows in hc. note that neither method is inherently biased in its design to prefer tokens from certain regions.