Unsupervised Opinion Summarization Using Approximate Geodesics (2023)

Abstract: Opinion summarization is the task of creating summaries capturing popular opinions from user reviews. In this paper, we introduce Geodesic Summarizer (GeoSumm), a novel system to perform unsupervised extractive opinion summarization. GeoSumm consists of an encoder-decoder based representation learning model that generates topical representations of texts. These representations capture the underlying semantics of the text as a distribution over learnable latent units. GeoSumm generates these topical representations by performing dictionary learning over pre-trained text representations at multiple layers of the decoder. We then use these topical representations to quantify the importance of review sentences using a novel approximate geodesic distance-based scoring mechanism. We use the importance scores to identify popular opinions in order to compose general and aspect-specific summaries. Our proposed model, GeoSumm, achieves strong performance on three opinion summarization datasets. We perform additional experiments to analyze the functioning of our model and showcase the generalization ability of GeoSumm across different domains.

## introduction as more and more human interaction takes place online, consumers find themselves wading through an ever-increasing number of documents (e.g., customer reviews) when trying to make informed purchasing decisions. as this body of information grows, so does the need for automatic systems that can summarize it in an unsupervised manner. opinion summarization is the task of automatically generating concise summaries from online user reviews @xcite @xcite . for instance, opinion summaries allow a consumer to understand product reviews without reading all of them. opinion summaries are also † work done during an internship at google research. useful for sellers to receive feedback, and compare different products. the recent success of deep learning techniques has led to a significant improvement in summarization @xcite @xcite @xcite in supervised settings. however, it is difficult to leverage these techniques for opinion summarization due to the scarcity of annotated data. it is expensive to collect good-quality opinion summaries as human annotators need to read hundreds of reviews to write a single summary @xcite . therefore, most works on opinion summarization tackle the problem in an unsupervised setting. recent works @xcite focus on abstractive summarization, where fluent summaries are generated using novel phrases. however, these approaches suffer from issues like text hallucination @xcite that affect the faithfulness of generated summaries @xcite . extractive summaries are less prone to these problems presenting the user with a representative subset of the original reviews. we focus on the task of unsupervised extractive opinion summarization, where the system selects sentences representative of the user opinions. inspired by previous works @xcite , we propose a novel encoder-decoder architecture along with objectives for (1) learning sentence representations that capture the underlying semantics, and (2) a sentence selection algorithm to compose a summary. one of the challenges in extractive summarization is quantifying the importance of opinions. an opinion is considered to be important if it is semantically similar to opinions from other users. using off-the-shelf pre-trained representations to obtain semantic similarity scores has known issues @xcite . these similarity scores can behave counterintuitively due to the high anisotropy of the representation space (a few dimensions dominate the cosine similarity scores). therefore, we use topical representations @xcite , which capture the semantics of text as a distribution over latent semantic units. these semantic units encode underlying concepts or topics. the semantic units can be captured using a learnable dictionary @xcite @xcite . topical representations enable us to effectively measure semantic similarity between text representations as they are distributions over the same support. text representations from reviews lie on a high-dimensional manifold. it is important to consider the underlying manifold while computing the importance score of a review. therefore, we use the approximate geodesic distance between topical text representations to quantify the importance scores of reviews. in this paper, we present geodesic summarizer (geosumm) that learns topical text representations in an unsupervised manner from distributed representations @xcite . we also present a novel sentence selection scheme that compares topical sentence representations in high-dimensions using approximate geodesics. empirical evaluations show that geosumm achieves strong performance on three opinion summarization datasets -oposum+ (amplayo et al., 2021a), amazon @xcite and space @xcite . our primary contributions are: • we present an extractive opinion summarization system, geosumm. it consists of an unsupervised representation learning system and a sentence selection algorithm (section 3). ## task setup in extractive opinion summarization, the objective is to select representative sentences from a review set. specifically, each dataset consists of a set of entities e and their corresponding review set r. for each entity @xmath0.g., a particular hotel such as the holiday inn in redwood city, ca), a review set r @xmath1, r 2 , . . .} is provided, where each review is an ordered set of sentences r @xmath2, s (i) 2 , . . .}. for simplicity of notation, we will represent the set of review sentences corresponding to an entity e as s @xmath3. for each entity, reviews encompass a set of aspects a @xmath4, a 2 , . . .} (e.g., service, food of a hotel). in this work, we consider two forms of extractive summarization: (a) general summarization, where the system selects a subset of sentences o e ⊂ s e , that best represents popular opinions in the review set r e ; (b) aspect summarization, where the system selects a representative sentence subset o (a) e ⊂ s e , about a specific aspect a (e.g., service) of an entity e (e.g., hotel). ## geodesic summarizer (geosumm) in this section, we present our proposed approach geodesic summarizer (geosumm). geosumm has two parts: (a) an unsupervised model to learn topical representations of review sentences, and (b) a sentence selection algorithm that uses the approximate geodesic distance between topical representations, to compose the extractive summary. ## experiments we evaluate the performance of geosumm on extractive summarization. given a set of user reviews the system needs to select a subset of the sentences as the summary. this summary is then compared with human-written summaries. in this section, we discuss the experimental setup in detail. ## related work most work on opinion summarization focuses on generating summaries in an unsupervised setup due to the scarcity of labeled data. these works are broadly classified into two categories based on the type of summaries being generated: abstractive @xcite @xcite or extractive @xcite @xcite human geosumm semae qt all staff members were friendly, accommodating, and helpful. the hotel and room were very clean. the room had modern charm and was nicely remodeled. the beds are extremely comfortable. the rooms are quite with wonderful beach views. the food at hash, the restaurant in lobby, was fabulous. the location is great, very close to the beach. it's a longish walk to santa monica. the price is very affordable. overall we had a nice stay at the hotel. our room was very clean and comfortable. the atmosphere is stylish and the service was great. we ate breakfast at the hotel and it was great. i appreciate the location and the security in the hotel. the food and service at the restaurant was awesome. the hotel is classy and has a rooftop bar. the restaurant is cozy but they have good healthy food. great hotel. the staff is great. the hotel erwin is a great place to stay. the staff were friendly and helpful. the location is perfect. we ate breakfast at the hotel and it was great. the hotel itself is in a great location. the service was wonderful. it was great. the rooms are great. the rooftop bar high was the icing on the cake. the food and service at the restaurant was awesome. the service was excellent. great hotel. we liked our room with an ocean view. the staff were friendly and helpful. there was no balcony. the location is perfect. our room was very quiet. i would definitely stay here again. you're one block from the beach. so it must be good! filthy hallways. unvacuumed room. pricy, but well worth it. our work is most similar to extractive summarization systems @xcite , and qt @xcite . similar to these systems, geodesic summarizer has two components: a representation learning system, and a sentence selection routine. however, unlike these approaches, we leverage pre-trained models to learn topical representations over a latent dictionary and propose a sentence selection mechanism using approximate geodesics to perform summarization. approaches in our work resemble prior works in deep clustering, which considers a similar combination of unsupervised representation learning and sparse structures @xcite @xcite @xcite . in a similar fashion, dictionary learning-like approaches have been combined with deep networks @xcite for various tasks. ## conclusion we present geodesic summarizer, a novel framework for extractive opinion summarization. geo-summ uses a representation learning model to convert distributed representations from a pre-trained model into topical text representations. geosumm uses these representations to compute the importance of a sentence using approximate geodesics. we show that geosumm achieves strong performance on several opinion summarization datasets. however, there are a lot of open questions about the inductive biases of representation learning that are needed for unsupervised summarization. in this work, we show the efficacy of topical representations. however, are there better approaches to capturing language semantics that help us quantify the importance of an opinion? our analysis shows that representations from geosumm span the highdimensional space in a manner that different parts of it capture distinct semantics. this opens up the possibility of leveraging the representation geometry to capture different forms of semantics. future work can explore ways to leverage topical representations from geosumm for tasks where there is a scarcity of labeled data. we propose geosumm, a novel system that learns topical representations of text and uses them to compute the importance of opinion reviews for extractive summarization. one of the limitations of geosumm is that it requires pre-training of the representation learning module using reviews sentences from a similar domain. for this, geosumm requires access to a large collection of review data from the target domain, thereby limiting its applicability in zero-shot or few-shot setups. this can be alleviated by future research on developing foundational models that learn topical representations on large-scale datasets and generalize across different opinion summarization domains.