SCAR : Sentence Compression using Autoencoders for Reconstruction (2020)

Abstract: Sentence compression is the task of shortening a sentence while retaining its meaning. Most methods proposed for this task rely on labeled or paired corpora (containing pairs of verbose and compressed sentences), which is often expensive to collect. To overcome this limitation, we present a novel unsupervised deep learning framework (SCAR) for deletion-based sentence compression. SCAR is primarily composed of two encoder-decoder pairs: a compressor and a reconstructor. The compressor masks the input, and the reconstructor tries to regenerate it. The model is entirely trained on unlabeled data and does not require additional inputs such as explicit syntactic information or optimal compression length. SCAR’s merit lies in the novel Linkage Loss function, which correlates the compressor and its effect on reconstruction, guiding it to drop inferable tokens. SCAR achieves higher ROUGE scores on benchmark datasets than the existing state-of-the-art methods and baselines. We also conduct a user study to demonstrate the application of our model as a text highlighting system. Using our model to underscore salient information facilitates speed-reading and reduces the time required to skim a document.

## introduction our fast-paced lifestyle precludes us from reading verbose and lengthy documents. how about a system that highlights the salient content for us (as shown in fig. 1 )? we model this problem as the well-known sentence compression task. sentence compression aims to generate a shorter representation of the input that captures its gist and preserves its intent. compression algorithms are broadly classified as abstractive and extractive. extractive compression or deletion-based algorithms only select relevant words from the input, whereas abstractive compression algorithms also allow paraphrasing. figure 1 : an example of a system that highlights the salient content, allowing the user to skim through the document quickly. in the past, compression approaches have revolved around statistical methods @xcite and syntactic rules @xcite . current state-of-the-art methods model the problem as a sequence-to-sequence learning task @xcite . although these methods perform well, they require massive parallel training datasets that are difficult to collect @xcite . recently, unsupervised approaches have been explored to overcome this limitation. @xcite model compression as a denoising task but barely reach the baselines. @xcite propose seq 3 , an autoencoder which uses a gumbel-softmax to represent the distribution over summaries. but a qualitative analysis of their outputs shows that seq 3 mimics the lead baseline. in this work, we present an unsupervised deep learning framework (scar) for deletion-based sentence compression. scar is composed of a compressor and a reconstructor. for each word in the input, the compressor determines whether or not to include it in the compression. a length loss restricts the compression length. the reconstruc-tor tries to regenerate the input using the words retained by the compressor. a reconstruction loss motivates the compressor to include words that aid in reconstruction. however, without an additional loss to govern word masking, the network fails to converge. we introduce a novel linkage loss that ties together the compressor and the reconstructor. it penalizes the network if a) it decides to drop a word but is unable to reconstruct it or b) it decides to include a word which it could reconstruct easily. ## related work early compression algorithms were formulated using strong linguistic priors and language heuristics @xcite @xcite . @xcite use syntactical evidence to condition the output of the model. @xcite prune dependency edges to remove constituents for compression. deep learning-based approaches have gained popularity owing to their success in core nlp tasks such as machine translation @xcite . @xcite propose an rnn based encoder-decoder network for deletion based compression. although this approach achieves superior performance over metric-based approaches, a large amount of paired sentences are needed to train the network. the first attempt to reduce the dependence on paired corpora for deletion based deep learning compression models was made by @xcite . they train separate compressor and reconstruction models, to allow for both supervised and unsupervised training. the compressor consists of a discrete variational autoencoder. the model is trained end-to-end using the reinforce algorithm. however, the reported results still use a sizeable amount of labeled data. recent approaches have sought completely unsupervised solutions. fevry and phang (2018) use a denoising autoencoder (dae) for sentence compression. the input sentence is shuffled and extended to add noise. dae tries to reconstruct the original denoised sentence from the noisy input. an additional signal is needed to specify the output length. at test time, the sentence is fed to the model without any noise. in an attempt to denoise the input, the network generates a compressed output. however, the model often fails to capture the information present in the input and is barely able to reach the baselines. seq 3 @xcite proposes an autoencoder using a gumbel-softmax to represent the distribution over summaries. a compressor generates a summary, and a reconstructor tries to reconstruct the input using the summary. a pre-trained language model acts as a prior, to incentivize the compressor to produce human-readable summaries. an additional topic loss is required to ensure that the summary contains relevant words, making the model non-generic and fine-tuned to the domain. a qualitative analysis of the outputs shows that seq 3 merely mimics the lead baseline and generates compressions by blindly copying a prefix of the input. ## scar scar is composed of two encoder-decoder pairs: compressor c and reconstructor r, as shown in fig. 2 . given an input sentence @xmath0, w 2 ..., w k containing k words, c generates an indicator vector i @xmath1, i v2 , ..., i vk which indicates the presence/absence of each word in the summary. the summary is represented as @xmath2, where represents element-wise multiplication. therefore, words corresponding to i vi ≈ 0 are effectively skipped. the network tries to reconstruct the input sentence from s . formally, the network tries to find an i * v such that the probability p(s|s i v ) is maximized and k @xmath3, jointly. the probability p(s|s i v ) can be decomposed further as shown in eq.( 1 ) for every word in the sentence, we learn a 300-dimensional embedding initialized with glove @xcite . these embeddings are sequentially fed as input to the sentence encoder (e s ), composed of a bi-lstm. the input is fed forwards and backward. the hidden states are a concatenation of the forward and backward states. the sentence representation is obtained from the final hidden state of e s (i.e., h e1 ). the indicator extraction module (iem), a bi-lstm decoder, is initialized using h e1 . the output of this decoder at each time step is passed through a network of two fully connected layers to generate a single indicator value. we intend this value to be close to either one or zero, denoting the presence/absence of each word from the summary. the masked sentence, @xmath4, is encoded using the summary encoder (e s ), composed of a bi-lstm. the summary decoder (d s ), also a bi-lstm, is initialized using the final hidden state of e s (h e2 ). this decoder aims to regenerate the input sentence s from s . this motivates iem to generate i v such that s can be easily reconstructed. the output at each time step in d s is fed to a dense layer, w s , which computes a distribution over the vocabulary from the decoder's hidden states. ## experiments since the test set of the gigaword corpus is small (1.9k samples) and does not capture the true behavior of the models, we report our results on the significantly larger validation set (189k samples). note that scar does not make use of the validation set during training, and it can be treated as a test set. we also test (without retraining) scar on @xcite @xcite bytes. we report average rouge (1,2,l) f1 scores @xcite obtained by all the models in we compare our model with three standard baselines -prefix (first 8 words for gigaword/first 75 bytes for duc), lead50 (50% tokens) and all-text (entire input). to compare with supervised approaches, we train a baseline seq2seq model, similar to @xcite . finally, we compare our model with the recent unsupervised approaches, dae (fevry and phang, 2018) foot_1 , and seq foot_2 (baziotis et al., 2019) 3 . ## conclusion and future work scar addresses a significant limitation of the unavailability of labeled data for sentence compression. it outperforms the existing state-of-the-art unsupervised models. since scar learns to drop inferable components of the input and therefore reduces noise, it can be used as a preprocessing step for machine translation and other information retrieval tasks.