Unsupervised Opinion Summarization with Noising and Denoising (2020)

Abstract: The supervised training of high-capacity models on large datasets containing hundreds of thousands of document-summary pairs is critical to the recent success of deep learning techniques for abstractive summarization. Unfortunately, in most domains (other than news) such training data is not available and cannot be easily sourced. In this paper we enable the use of supervised learning for the setting where there are only documents available (e.g., product or business reviews) without ground truth summaries. We create a synthetic dataset from a corpus of user reviews by sampling a review, pretending it is a summary, and generating noisy versions thereof which we treat as pseudo-review input. We introduce several linguistically motivated noise generation functions and a summarization model which learns to denoise the input and generate the original review. At test time, the model accepts genuine reviews and generates a summary containing salient opinions, treating those that do not reach consensus as noise. Extensive automatic and human evaluation shows that our model brings substantial improvements over both abstractive and extractive baselines.

## introduction the proliferation of massive numbers of online product, service, and merchant reviews has provided strong impetus to develop systems that perform opinion mining automatically @xcite . the vast majority of previous work @xcite breaks down the problem of opinion aggregation and summarization into three interrelated tasks involving aspect extraction @xcite , sentiment identification @xcite , and summary creation based on extractive @xcite or abstractive methods @xcite @xcite . although po-tentially more challenging, abstractive approaches seem more appropriate for generating informative and concise summaries, e.g., by performing various rewrite operations (e.g., deletion of words or phrases and insertion of new ones) which go beyond simply copying and rearranging passages from the original opinions. abstractive summarization has enjoyed renewed interest in recent years thanks to the availability of large-scale datasets @xcite @xcite @xcite which have driven the development of neural architectures for summarizing single and multiple documents. several approaches @xcite @xcite @xcite @xcite have shown promising results with sequence-to-sequence models that encode one or several source documents and then decode the learned representations into an abstractive summary. the supervised training of high-capacity models on large datasets containing hundreds of thousands of document-summary pairs is critical to the recent success of deep learning techniques for abstractive summarization. unfortunately, in most domains (other than news) such training data is not available and cannot be easily sourced. for instance, manually writing opinion summaries is practically impossible since an annotator must read all available reviews for a given product or service which can be prohibitively many. moreover, different types of products impose different restrictions on the summaries which might vary in terms of length, or the types of aspects being mentioned, rendering the application of transfer learning techniques @xcite problematic. motivated by these issues, @xcite consider an unsupervised learning setting where there are only documents (product or business reviews) available without corresponding summaries. they propose an end-to-end neural model to perform abstractive summarization based on (a) an autoencoder that learns representations for each review and (b) a summarization module which takes the aggregate encoding of reviews as input and learns to generate a summary which is semantically similar to the source documents. due to the absence of ground truth summaries, the model is not trained to reconstruct the aggregate encoding of reviews, but rather it only learns to reconstruct the encoding of individual reviews. as a result, it may not be able to generate meaningful text when the number of reviews is large. furthermore, autoencoders are constrained to use simple decoders lacking attention @xcite and copy @xcite mechanisms which have proven useful in the supervised setting leading to the generation of informative and detailed summaries. problematically, a powerful decoder might be detrimental to the reconstruction objective, learning to express arbitrary distributions of the output sequence while ignoring the encoded input @xcite . in this paper, we enable the use of supervised techniques for unsupervised summarization. specifically, we automatically generate a synthetic training dataset from a corpus of product reviews, and use this dataset to train a more powerful neural model with supervised learning. the synthetic data is created by selecting a review from the corpus, pretending it is a summary, generating multiple noisy versions thereof and treating these as pseudoreviews. the latter are obtained with two noise generation functions targeting textual units of different granularity: segment noising introduces noise at the word-and phrase-level, while document noising replaces a review with a semantically similar one. we use the synthetic data to train a neural model that learns to denoise the pseudo-reviews and generate the summary. this is motivated by how humans write opinion summaries, where denoising can be seen as removing diverging information. our proposed model consists of a multi-source encoder and a decoder equipped with an attention mechanism. additionally, we introduce three modules: (a) explicit denoising guides how the model removes noise from the input encodings, (b) partial copy enables to copy information from the source reviews only when necessary, and (c) a discriminator helps the decoder generate topically consistent text. we perform experiments on two review datasets representing different domains (movies vs businesses) and summarization requirements (short vs longer summaries). results based on automatic and human evaluation show that our method outperforms previous unsupervised summarization models, including the state-of-the-art abstractive system of @xcite and is on the same par with a state-of-the-art supervised model @xcite trained on a small sample of (genuine) review-summary pairs. ## related work most previous work on unsupervised opinion summarization has focused on extractive approaches @xcite @xcite where a clustering model groups opinions of the same aspect, and a sentence extraction model identifies text representative of each cluster. @xcite propose a graph-based abstractive framework for generating concise opinion summaries, while di fabbrizio et al. (2014) use an extractive system to first select salient sentences and then generate an abstractive summary based on hand-written templates @xcite . as mentioned earlier, we follow the setting of @xcite in assuming that we have access to reviews but no gold-standard summaries. their model learns to generate opinion summaries by reconstructing a canonical review of the average encoding of input reviews. our proposed method is also abstractive and neural-based, but eschews the use of an autoencoder in favor of supervised sequence-to-sequence learning through the creation of a synthetic training dataset. concurrently with our work, @xcite use a hierarchical variational autoencoder to learn a latent code of the summary. while they also use randomly sampled reviews for supervised training, our dataset construction method is more principled making use of linguistically motivated noise functions. our work relates to denoising autoencoders (daes; @xcite , which have been effectively used as unsupervised methods for various nlp tasks. earlier approaches have shown that daes can be used to learn high-level text representations for domain adaptation @xcite and multimodal representations of textual and visual input @xcite . recent work has applied daes to text generation tasks, specifically to data-to-text generation @xcite and extractive sentence compression @xcite . our model differs from these approaches in two respects. firstly, while previous work has adopted trivial noising methods such as randomly adding or removing words @xcite and randomly corrupting encodings @xcite , our noise generators are more linguistically informed and suitable for the opinion summarization task. secondly, while in @xcite the decoder is limited to vanilla rnns, our noising method enables the use of more complex architectures, enhanced with attention and copy mechanisms, which are known to improve the performance of summarization systems @xcite . ## modeling approach let @xmath0, ..., x n } denote a set of reviews about a product (e.g., a movie or business). our aim is to generate a summary y of the opinions expressed in x. we further assume access to a corpus @xmath1, ..., x m } containing multiple reviews about m products without corresponding opinion summaries. our method consists of two parts. we first create a synthetic dataset @xmath2, y)} consisting of summary-review pairs. specifically, we sample review x i from c, pretend it is a summary, and generate multiple noisy versions thereof (i.e., pseudoreviews). at training time, a denoising model learns to remove the noise from the reviews and generate the summary. at test time, the same denoising model is used to summarize actual reviews. we use denoising as an auxiliary task for opinion summarization to simulate the fact that summaries tend to omit opinions that do not represent consensus (i.e., noise in the pseudo-review), but include salient opinions found in most reviews (i.e., nonnoisy parts of the pseudo-review). ## results automatic evaluation our results on rotten tomatoes are shown in as can be seen, denoisesum outperforms all competing models on both datasets. when compared to meansum, the difference in performance is especially large on rotten tomatoes, where we see a 4.01 improvement in rouge-l. we believe this is because meansum does not learn to reconstruct encodings of aggregated inputs, and as a result it is unable to produce meaningful summaries when the number of input reviews is large, as is the case for rotten tomatoes. in fact, the best extractive model, sentineuron, slightly outperforms meansum on this dataset across metrics with the exception of rouge-l. when compared to the best supervised system, denoisesum performs comparably on several metrics, specifically me-teor and rouge-1, however there is still a gap on rouge-2, showing the limitations of systems trained without gold-standard summaries. ## conclusions we consider an unsupervised learning setting for opinion summarization where there are only reviews available without corresponding summaries. our key insight is to enable the use of supervised techniques by creating synthetic review-summary pairs using noise generation methods. our summarization model, denoisesum, introduces explicit denoising, partial copy, and discrimination modules which improve overall summary quality, outperforming competitive systems by a wide margin. in the future, we would like to model aspects and sentiment more explicitly as well as apply some of the techniques presented here to unsupervised single-document summarization.