Unsupervised Extractive Opinion Summarization Using Sparse Coding (2022)

Abstract: Opinion summarization is the task of automatically generating summaries that encapsulate information expressed in multiple user reviews. We present Semantic Autoencoder (SemAE) to perform extractive opinion summarization in an unsupervised manner. SemAE uses dictionary learning to implicitly capture semantic information from the review text and learns a latent representation of each sentence over semantic units. Our extractive summarization algorithm leverages the representations to identify representative opinions among hundreds of reviews. SemAE is also able to perform controllable summarization to generate aspect-specific summaries using only a few samples. We report strong performance on SPACE and AMAZON datasets and perform experiments to investigate the functioning of our model.

## introduction opinion summarization is the task of automatically generating digests for an entity (e.g. a product, a hotel, a service, etc.), from user opinions in online forums. automatic opinion summaries enable faster comparison, search, and better consumer feedback understanding @xcite @xcite . although there has been significant progress towards summarization @xcite @xcite @xcite , existing approaches rely on human-annotated reference summaries, which are scarce for opinion summarization. for opinion summarization, human annotators need to read hundreds of reviews per entity across different sources for writing a summary, which may not be feasible. this lack of labeled training data has prompted a series of works to leverage unsupervised or weaklysupervised techniques for opinion summarization @xcite @xcite . recent works in this direction have focused on performing opinion summarization in an abstractive setting @xcite @xcite @xcite . abstractive models are able to produce fluent summaries using novel phrases. however, they suffer from problems common in text generation like hallucination @xcite ), text degeneration @xcite , and topic drift @xcite . also, these approaches have been evaluated on small scales (10 reviews per entity or fewer), which does not reveal their utility in the real world where there are hundreds of reviews per entity. to overcome these issues, another thread of works focuses on extractive opinion summarization, which creates summaries by selecting review sentences to reflect the popular opinions corresponding to an entity. a recently proposed extractive summarization approach is quantized transformer (qt) @xcite , which leverages vector quantization @xcite for assigning texts to a latent representation that is supposed to capture a semantic sense. however, a text phrase can encapsulate multiple semantic senses, making this representation learning approach restrictive. building on the framework introduced by qt, we introduce an unsupervised extractive model, semantic autoencoder (semae), which learns a representation of text over latent semantic units using dictionary learning @xcite . similar to qt, semae leverages transformer @xcite for sentence reconstruction to simultaneously learn latent semantic units and sentence representations. however, while qt assigns texts to a latent representation (codebook), semae models text as a combination of semantics and forms a distribution over latent units (dictionary). this allows sentence rep-resentations to capture fine-grained and diverse semantics. unlike qt that relies on identification of aspect-specific head representations, we achieve controllable summarization by utilizing information-theoretic measures (such as relevance, redundancy, etc) on sentence representations. our sentence selection algorithm is more flexible and allows a broader spectrum of controllable summarization. we experimentally show strong performance on two opinion summarization datasets. our main contributions are: ## related work unsupervised opinion summarization can be conducted either abstractively or extractively. abstractive approaches aim to summarize the opinion text using novel phrases. traditional statistical approaches create abstractive summaries using graphical paths @xcite or hand-written templates @xcite . recent neural approaches leverage the encoder-decoder architecture to aggregate information from multiple reviews and generate summaries accordingly @xcite @xcite . in contrast to abstractive approaches, extractive approaches rank and select a subset of salient sentences from reviews to form a concise summary @xcite . saliency computation has been explored using traditional frequency-based approaches @xcite , similarity with the centroid in the representation space @xcite , and lexical similarity with all sentences in a graph-based representation @xcite . weakly supervised approaches @xcite extract opinions based on their aspect specificity, and nature of sentiment polarity. our work is most similar to the extractive opinion summarization qt @xcite as discussed in section 1. it is also similar to neural topic model-based approaches @xcite @xcite ) that use a variant of dictionary learning @xcite to represent text as a combination of specific semantics (e.g. aspect, relationships etc). in contrast to these models, where text from same topics are trained to have similar representations using max-margin loss, semae uses an autoencoder setup to capture diverse latent semantics. ## task description we follow the task setup in @xcite , where given a set of entities (e.g. hotels), a review set r @xmath0, r 2 , . . .} is provided for each entity e, where each review r i is a sequence of sentences {s 1 , s 2 , . . .}. the review set r e covers a range of aspects @xmath1, a 2 , . . .} relating to the domain (e.g. service, location for hotels). we denote s e to be the set of sentences from all reviews for an entity e. semae is evaluated to perform two types of extractive opinion summarization introduced by angelidis et al. ( 2021) ## the semantic autoencoder the intuition behind semantic autoencoder is that instead of representing text as a single latent semantic unit, we represent text as a distribution over latent semantic units using dictionary learning. learning semantic representations over a common dictionary makes them structurally aligned, enabling comparison of sentences using informationtheoretic measures. semantic autoencoder consists of three stages (i) sentence encoding -an input sentence s is converted into a multi-head representation (h heads) using transformer encoder {s h } h @xmath2; (ii) reconstruction -a latent representation of head vectors s h is formed over elements of the dictionary @xmath3, to produce reconstructed representations @xmath4; and (iii) sentence decodinga transformer-based decoder takes as input the reconstructed representations z to produce the output sentence ŝ. semae is trained on the sentence reconstruction task. the overall workflow of semae reconstruction ( ŝ ) × figure 1 : an example workflow of semae. the encoder produces @xmath5, which are used to generate latent representations over dictionary elements. the decoder reconstructs the input sentences using vectors (z h ) formed using latent representations (α h ). ## summarization using latent representations we leverage the latent representations α h generated by semae to perform opinion summarization. foot_0 ## experimental setup in this section, we discuss the experimental setup, results and analysis. ## conclusion we proposed a novel opinion summarization approach using semantic autoencoder, which encodes text as a representation over latent semantic units. we perform extractive summarization by selecting sentences using information-theoretic measures over representations obtained from semae. our experiments reveal that dictionary element representations from semae form clusters, which capture distinct semantics. our model provides finegrained control to users to model surface-level text attributes (like redundancy, informativeness etc.) in the representation space. semae outperforms existing extractive opinion summarization methods on space and amazon datasets. finally, semae representations can be leveraged to explore different forms of control on the summary generation (e.g. multi-aspect sumamrization) using our inference framework. future works can focus on better representation learning systems to handle use-cases with noisy or sparse textual data. dataset λ 1 λ 2 amazon 10 3 5 × 10 -4 space 10 4 5 × 10 -4