Informative and Controllable Opinion Summarization (2021)

Abstract: Opinion summarization is the task of automatically generating summaries for a set of reviews about a specific target (e.g., a movie or a product). Since the number of reviews for each target can be prohibitively large, neural network-based methods follow a two-stage approach where an extractive step first pre-selects a subset of salient opinions and an abstractive step creates the summary while conditioning on the extracted subset. However, the extractive model leads to loss of information which may be useful depending on user needs. In this paper we propose a summarization framework that eliminates the need to rely only on pre-selected content and waste possibly useful information, especially when customizing summaries. The framework enables the use of all input reviews by first condensing them into multiple dense vectors which serve as input to an abstractive model. We showcase an effective instantiation of our framework which produces more informative summaries and also allows to take user preferences into account using our zero-shot customization technique. Experimental results demonstrate that our model improves the state of the art on the Rotten Tomatoes dataset and generates customized summaries effectively.

## introduction the proliferation of opinions expressed in online reviews, blogs, and social media has created a pressing need for automated systems which enable customers and companies to make informed decisions without having to absorb large amounts of opinionated text. opinion summarization is the task of automatically generating summaries for a set of opinions about a specific target @xcite . figure 1 shows various reviews about the movie "coach carter" and example summaries generated by humans and automatic systems. figure 1 : three out of 150 reviews for the movie "coach carter", and summaries written by the editor, and generated by a model following the extract-abstract approach and the proposed condense-abstract framework. the latter produces more informative and factual summaries whilst allowing to control aspects of the generated summary (such as the acting or plot of the movie). the vast majority of previous work @xcite views opinion summarization as the final stage of a three-step process involving: (1) aspect extraction (i.e., finding features pertaining to the target of interest, such as battery life or sound quality); (2) sentiment prediction (i.e., determining the sentiment of the extracted aspects); and general-purpose summarization approaches have recently shown promising results with end-toend models which are data-driven and take advantage of the success of sequence-to-sequence neural network architectures. most approaches @xcite encode documents and then decode the learned representations into an abstractive summary, often by attending to the source input @xcite and copying words from it @xcite . under this modeling paradigm, it is no longer necessary to identify aspects and their sentiment for the opinion summarization task, as these are learned indirectly from training data (i.e., sets of opinions and their corresponding summaries). these models are usually tested on domains where the input is either one document or a small set of documents. however, the number of input reviews for each target entity tends to be very large (150 for the example in figure 1 ). it is therefore practically unfeasible to train a model in an end-to-end fashion, given the memory limitations of modern hardware. as a result, current approaches @xcite @xcite sacrifice end-to-end elegance in favor of a two-stage framework which we call extract-abstract (ea): an extractive model first selects a subset of opinions and an abstractive model then generates the summary while conditioning on the extracted subset (see figure 2a ). the extractive pass unfortunately has two drawbacks. firstly, on account of having access to only a small subset of reviews, the summaries can be less informative and inaccurate, as shown in figure 1 . and secondly, user preferences cannot be easily taken into account (e.g., a user may wish to obtain a summary focusing on the acting or plot of a movie as opposed to a general-purpose summary) since more specialized information might have been removed. in this paper, we propose condense-abstract (ca), an alternative two-stage framework which enables the use of all input reviews when generating the summary (see figure 2b ). the condense model first represents the input reviews as encodings, aiming to condense their meaning and distill information relating to sentiment and various aspects of the target being reviewed. the abstract model then fuses these condensed representations into one aggregate encoding and generates an opinion summary from it. we implement a simple yet effective instantiation of the ca framework, using a vanilla autoencoder as the condense model, and a decoder with attention and copy mechanisms as the abstract model. we also introduce a zero-shot customization technique allowing users to control important aspects of the generated summary at test time. our approach enables controllable generation while leveraging the full spectrum of opinions available for a specific target. we perform experiments on a dataset consisting of movie reviews and opinion summaries elicited from the rotten tomatoes website @xcite ; see figure 1 ). our proposed approach outperforms state-of-the-art models by a large margin using automatic metrics and in a judgment elicitation study. we also verify that our zero-shot customization technique can effectively generate need-specific summaries. more recent work has seen the effective application of sequence-to-sequence models @xcite to various abstractive summarization tasks including headline generation @xcite , single- @xcite , and multi-document summarization @xcite @xcite . closest to our approach is the work of wang and ling (2016) who generate opinion summaries following a two-stage process which first selects/extracts reviews bearing pertinent information, and then generates the summary by conditioning on these reviews. more recent models @xcite ; amplayo and lapata, 2020) perform opinion summarization in an unsupervised way. however, these are mostly done on toy datasets @xcite , typically with a small number of reviews per target entity. our proposed framework works better on realworld datasets with a large number of reviews, since it eliminates the need to rely only on preselected salient reviews which we argue leads to information loss and subsequently less customizable generation. instead, our model first condenses the source reviews into multiple dense vectors which serve as input to a decoder to generate an abstractive summary. beyond producing more informative summaries, we demonstrate that our approach also allows to customize them. recent conditional generation models have focused on controlling various aspects of the output such as politeness @xcite , length @xcite , content @xcite , or style @xcite . in contrast, our zero-shot customization technique requires neither training examples of documents and corresponding (customized) summaries nor specialized pre-processing to encode which tokens in the input might give rise to customization. ## condense-abstract framework we propose an alternative to the extract-abstract (ea) approach which enables the use of all input reviews when generating the summary. figure 2b illustrates our proposed condense-abstract (ca) framework. in lieu of an integrated encoder-decoder, we generate summaries using two separate models. the condense model returns review encodings for n input reviews, while the abstract model uses these encodings to create an abstractive summary. this two-step approach has two advantages for multi-document summarization. firstly, ca-based models are more space-efficient, since the set of n reviews is not treated as one large instance but as n separate instances when training the condense model. and secondly, it is possible to generate maximally informative and customizable summaries targeting specific aspects of the input since the abstract model operates over the encodings of all available reviews. in the following subsections, we explain how we instantiate a model using the ca framework, which we call condasum, with an lstm-based foot_0 vanilla autoencoder (condense model) and a decoder with attention and copy mechanisms (abstract model). ## experimental setup dataset we performed experiments on the rotten tomatoes dataset foot_2 provided in @xcite . it contains 3,731 movies; for each movie we are given a large set of reviews written by professional critics and users and a gold-standard consensus summary written by an editor (see an example in figure 1 ). we report the dataset statistics in training configuration for all experiments, our model used word embeddings with 128 dimensions, pretrained using glove @xcite . we set the dimensions of all hidden vectors to 256 and the batch size to 8. for decoding summaries, we use a length-normalized beam search with beam size of 5. we applied dropout @xcite at a rate of 0.5. the model was trained using the adam optimizer (kingma and ba, 2015) with default parameters and l 2 constraint @xcite of 2. we performed early stopping based on model performance on the development set. our model is implemented in pytorch foot_3 . ## conclusions we introduced the condense-abstract (ca) framework for opinion summarization which eliminates the need to rely only on a small subset of extracted reviews and allows the use of all reviews to generate maximally informative summaries. we presented condasum, an instantiation of this framework and showed in both automatic and human-based evaluation that it is superior to purely extractive models and abstractive models that include an extractive pre-selection stage. we also showed that when an extractive step is not used, our zero-shot customization technique is able to generate need-specific summaries at test time. in the future, we plan to apply the ca framework to other multi-document summarization tasks.