Extractive Opinion Summarization in Quantized Transformer Spaces (2021)

Abstract: We present the Quantized Transformer (QT), an unsupervised system for extractive opinion summarization. QT is inspired by Vector- Quantized Variational Autoencoders, which we repurpose for popularity-driven summarization. It uses a clustering interpretation of the quantized space and a novel extraction algorithm to discover popular opinions among hundreds of reviews, a significant step towards opinion summarization of practical scope. In addition, QT enables controllable summarization without further training, by utilizing properties of the quantized space to extract aspect-specific summaries. We also make publicly available Space, a large-scale evaluation benchmark for opinion summarizers, comprising general and aspect-specific summaries for 50 hotels. Experiments demonstrate the promise of our approach, which is validated by human studies where judges showed clear preference for our method over competitive baselines.

## introduction online reviews play an integral role in modern life, as we look to previous customer experiences to inform everyday decisions. the need to digest review content has fueled progress in opinion mining @xcite , whose central goal is to automatically summarize people's attitudes towards an entity. early work @xcite focused on numerically aggregating customer satisfaction across different aspects of the entity under consideration (e.g., the quality of a camera, its size, clarity). more recently, the success of neural summarizers in the wikipedia and news domains @xcite @xcite @xcite has spurred interest in opinion summarization; the aggregation, in textual form, of opinions expressed in a set of reviews @xcite @xcite @xcite @xcite @xcite . opinion summarization has distinct characteristics that set it apart from other summarization tasks. first, it cannot rely on reference summaries for training, because such meta-reviews are very scarce and their crowdsourcing is unfeasible. even for a single entity, annotators would have to produce summaries after reading hundreds, sometimes thousands, of reviews. second, the inherent subjectivity of review text distorts the notion of information importance used in generic summarization @xcite . conflicting opinions are often expressed for the same entity and, therefore, useful summaries should be based on opinion popularity @xcite . moreover, methods need to be flexible with respect to the size of the input (entities are frequently reviewed by thousands of users), and controllable with respect to the scope of the output. for instance, users may wish to read a general overview summary, or a more targeted one about a particular aspect of interest (e.g., a hotel's location, its cleanliness, or available food options). recent work @xcite @xcite @xcite @xcite has increasingly focused on abstractive summarization, where a summary is generated token-by-token to create novel sentences that articulate prevalent opinions in the input reviews. the abstractive approach offers a solution to the lack of supervision, under the assumption that opinion summaries should be written in the style of reviews. this simplification has allowed abstractive models to generate review-like summaries from aggregate input representations, using sequence-to-sequence models trained to reconstruct single reviews. despite being fluent, abstractive summaries may still suffer from issues of text degeneration @xcite , hallucinations @xcite , and the undesirable use of first-person narrative, a direct consequence of review-like generation. in addition, previous work used an unrealistically small number of input reviews (10 or fewer), and only sparingly investigated controllable summarization, albeit in weakly supervised settings @xcite . in this paper, we attempt to address shortcomings of existing methods by turning to extractive summarization, which aims to construct an opinion summary by selecting a few representative input sentences @xcite . specifically, we introduce the quantized transformer (qt), an unsupervised neural model inspired by vector-quantized variational autoencoders @xcite @xcite , which we repurpose for popularity-driven summarization. qt combines transformers @xcite with the discretization bottleneck of vq-vaes and is trained via sentence reconstruction, similarly to the work of @xcite on paraphrasing. at inference time, we use a clustering interpretation of the quantized space and a novel extraction algorithm that discovers popular opinions among hundreds of reviews, a significant step towards opinion summarization of practical scope. qt is also capable of aspectspecific summarization without further training, by exploiting the properties of the transformer's multi-head sentence representations. we further contribute to the progress of opinion mining research, by introducing space (shorthand for summaries of popular and aspect-specific customer experiences), a large-scale corpus for the evaluation of opinion summarizers. we collected 1,050 human-written summaries of tripadvisor reviews for 50 hotels. space has general summaries, giving a high-level overview of popular opinions, and aspect-specific ones, providing detail on individual aspects (e.g., location, cleanliness). each summary is based on 100 customer reviews, an order of magnitude increase over existing corpora, thus providing a more realistic input to competing models. experiments on space and two more benchmarks demonstrate that our approach holds promise for opinion summarization. participants in human evaluation further express a clear preference for our model over competitive baselines. we make space and our code publicly available. foot_0 2 related work @xcite were the first to make the connection between opinion mining and text summarization; they developed opinosis, a graph-based abstractive summarizer that explicitly models opinion popularity, a key characteristic of subjective text, and central to our approach. follow-on work @xcite adopts a hybrid approach where salient sentences are first extracted and abstracts are generated based on hand-written templates @xcite . more recently, @xcite extract salient opinions according to their polarity intensity and aspect specificity, in a weakly supervised setting. a popular approach to modeling opinion popularity, albeit indirectly, is vector averaging. @xcite propose meansum, an unsupervised abstractive summarizer that learns a review decoder through reconstruction, and uses it to generate summaries conditioned on averaged representations of the inputs. averaging is also used by @xcite , who train a copyenabled variational autoencoder by reconstructing reviews from averaged vectors of reviews about the same entity. other methods include denoising autoencoders (amplayo and lapata, 2020) and the system of @xcite , an encoderdecoder architecture that uses a clustering of the encoding space to identify opinion groups, similar to our work. our model builds on the vq-vae (van den oord et al., 2017), a recently proposed training technique for learning discrete latent variables, which aims to overcome problems of posterior collapse and large variance associated with variational autoencoders @xcite . like other related discretization techniques @xcite , vq-vae passes the encoder output through a discretization bottleneck using a neighbor lookup in the space of latent code embeddings. the application of vq-vaes to opinion summarization is novel, to our knowledge, as well as the proposed sentence extraction algorithm. our model does not depend on vector averaging, nor does it suffer from information loss and hallucination. furthermore, it can easily accommodate a large number of input reviews. within nlp, vq-vaes have been previously applied to neural machine translation @xcite and paraphrase generation @xcite . our work is closest to @xcite in its use of a quantized transformer, however we adopt a different training algorithm (soft em; @xcite , orders of magnitude fewer discrete latent codes, a different method for obtaining head sentence vectors, and apply the qt in a novel way for extractive opinion summarization. besides modeling, our work contributes to the growing body of resources for opinion summarization. we release space, the first corpus to contain both general and aspect-specific opinion summaries, while increasing the number of input reviews tenfold compared to popular benchmarks @xcite @xcite . ## problem formulation let c be a corpus of reviews on entities {e 1 , e 2 , . . . } from a single domain d, for example, hotels. reviews may discuss any number of relevant aspects a @xmath0, a 2 , . . . }, like the hotel's rooms or location. for every entity e, we define its review set r @xmath1, r 2 , . . . }. every review is a sequence of sentences (x 1 , x 2 , . . . ) and a sentence x is, in turn, a sequence of words (w 1 , w 2 , . . . ). for brevity, we use x e to denote all review sentences about entity e. we formalize two sub-tasks: (a) general opinion summarization, where a summary should cover popular opinions in r e across all discussed aspects; and (b) aspect opinion summarization, where a summary must focus on a single specified aspect @xmath2. in our extractive setting, these translate to creating a general or aspect summary by selecting a small subset of sentences s e ⊂ x e . we train the qt through sentence reconstruction to learn a rich representation space and its quantization into latent codes (section 3.1). we enable opinion summarization, by mapping input sentences onto their nearest latent codes and extract those sentences that are representative of the most popular codes (section 3.2). we also illustrate how to produce aspect-specific summa-figure 1 : a sentence is encoded into a 3-head representation and head vectors are quantized using a weighted average of their neighboring code embeddings. the qt model is trained by reconstructing the original sentence. ## the space corpus we introduce space (summaries of popular and aspect-specific customer experiences), a largescale opinion summarization benchmark for the evaluation of unsupervised summarizers. space is built on tripadvisor hotel reviews and aims to facilitate future research by improving upon the shortcomings of existing datasets. it comes with a training set of approximately 1.1 million reviews for over 11,000 hotels, obtained by cleaning and downsampling an existing collection @xcite . the training set contains no reference summaries, and is useful for unsupervised training. for evaluation, we created a large collection of human-written, abstractive opinion summaries. specifically, for a held-out set of 50 hotels (25 hotels for development and 25 for testing), we asked human annotators to write high-level general summaries and aspect summaries for six popular aspects: building, cleanliness, food, location, rooms, and service. for every hotel and summary type, we collected three reference summaries from different annotators. importantly, for every hotel, summaries were based on 100 input reviews. to the best of our knowledge, this is the largest crowdsourcing effort towards obtaining highquality abstractive summaries of reviews, and the first to use a pool of input reviews of this scale (see the large number of input reviews per entity poses certain challenges with regard to the collection of human summaries. a direct approach is prohibitive, as it would require annotators to read all 100 reviews and write a summary in a single step. a more reasonable method is to first identify a subset of input sentences that most people consider salient, and then ask annotators to summarize them. summaries were thus created in multiple stages using the appen foot_3 platform and expert annotator channels of native english speakers. although we propose an extractive model, annotators were asked to produce abstractive summaries, as we hope space will be broadly useful to the summarization community. we did not allow the use of first-person narrative to collect more summary-like texts. we present our annotation procedure below. foot_4 ## evaluation in this section, we discuss our experimental setup, including datasets and comparison models, before presenting our automatic evaluation results, human studies, and further analyses. ## conclusions we presented a novel opinion summarization system based on the quantized transformer that requires no reference summaries for training, and is able to extract general and aspect summaries from large groups of input reviews. qt is trained through sentence reconstruction and learns a rich encoding space, paired with a clustering component based on vector quantized variational autoencoders. at summarization time, we exploit the characteristics of the quantized space, to identify those clusters that correspond to the input's most popular opinions, and extract the sentences that best represent them. moreover, we used the multi-head representations of the model, and no further training, to detect the encoding sub-space that best separates aspects, enabling aspect-specific summarization. we also collected space, a new opinion summarization corpus which we hope will inform and inspire further research. experimental results on space and popular benchmarks reveal that our system is able to produce informative summaries which cover all or individual aspects of an entity. in the future, we would like to utilize the qt framework in order to generate abstractive summaries. we could also exploit qt's multi-head semantics more directly, and further improve it through weak supervision or multi-task objectives. finally, although we focused on opinion summarization, it would be interesting to see if the proposed model can be applied to other multi-document summarization tasks.