C onvex A ggregation for O pinion S ummarization (2021)

Abstract: Recent advances in text autoencoders have significantly improved the quality of the latent space, which enables models to generate grammatical and consistent text from aggregated latent vectors. As a successful application of this property, unsupervised opinion summarization models generate a summary by decoding the aggregated latent vectors of inputs. More specifically, they perform the aggregation via simple average. However, little is known about how the vector aggregation step affects the generation quality. In this study, we revisit the commonly used simple average approach by examining the latent space and generated summaries. We found that text autoencoders tend to generate overly generic summaries from simply averaged latent vectors due to an unexpected L_2 -norm shrinkage in the aggregated latent vectors, which we refer to as summary vector degeneration. To overcome this issue, we develop a framework Coop, which searches input combinations for the latent vector aggregation using input-output word overlap. Experimental results show that Coop successfully alleviates the summary vector degeneration issue and establishes new state-of-the-art performance on two opinion summarization benchmarks. Code is available at https://github.com/megagonlabs/coop .

## introduction the unprecedented growth of online review platforms and the recent success of neural summarization techniques @xcite @xcite , spurred significant interest in research on multi-document opinion summarization @xcite @xcite @xcite . the goal of multi-document opinion summarization is to generate a summary that represents salient opinions in the input reviews. research on multi-document opinion summarization is challenging because of the lack of goldstandard summaries, which are difficult to collect at scale. this is in contrast to single-document summarization, where there exists an abundant annotated datasets @xcite @xcite . consequently, the primary approach is to employ text autoencoders for unsupervised opinion summarization @xcite ). text autoencoders, especially variational autoencoders (vaes), are known for the ability to generate grammatical and consistent text by aggregating multiple latent vectors @xcite . unsupervised opinion summarization models leverage this property to generate a summary by first aggregating the latent vectors of input reviews via simple average, and then decoding the summary from the aggregated vector. however, it has not been verified if the simple average is the best choice for summary generation. furthermore, little is known about the relationship between the latent vector and the generation quality. in this paper, we report that text autoencoder models with the simple average vector aggregation tend to generate overly generic summaries, which we refer to as summary vector degeneration. for example, as shown in figure 1 , with simply averaged latent vectors, the generated summaries of two distinct entities are almost identical. we further discovered two factors that cause summary vector degeneration: (1) simply averaged latent vectors cause unexpected l 2 -norm shrinkage, and (2) latent vectors with smaller l 2 -norm are decoded into less informative summaries (e.g., contain only general information.) to address the summary vector degeneration issue, we develop coop, a latent vector aggregation framework. in essence, coop considers convex combinations of the latent vectors of input reviews for better summary generation. more specifi-this is a great place to eat. the food is always fresh and the staff is very friendly. it's a great place to go if you are in the area. the food is always good and the prices are very reasonable. this place is great. the food is good and the service is great. the staff is very friendly and attentive. they have a nice selection of drinks and the food is always fresh. the prices are very reasonable. first time here and it was really good. the service was great, the food was delicious, and the portions were very large. this is a great place to go for chinese food. this is a great place to eat. the food was delicious and the staff was very attentive. the catfish was tender and tasty. the hush puppies were amazing. the mac and cheese was very good. they have a great beer selection as well. ## preliminaries let us denote @xmath0.g., restaurant or product), where each review is a sequence of words @xmath1, ..., x ∥x∥ ) in the text space x . given an entity e and its reviews r e ⊆ r, the goal of the multi-document opinion summarization task is to generate an abstractive summary s e such that the salient opinions in r e are included. ## revisiting simple average approach in this section, we revisit the commonly used simple average approach (simpleavg) and examine the relations between the aggregated latent vector and the quality of generated summaries. taking a simple average is an intuitive way to optimize the aggregated vector in the latent space since it minimizes the total distance between input latent vectors and the aggregated vector. thus, it appears to be a reasonable design choice and has been adopted by multiple unsupervised opinion summarization models as de-facto standard. however, we find that only considering the total distance between the input and the aggregated latent vectors does not always render high-quality summaries. this is because simpleavg is completely ignorant of the decoder performance and the resulting generation. in fact, we observe that simpleavg tends to produce overly generic summaries (as shown in figure 1 ), which we refer to as summary vector degeneration. to gain a better understanding of the summary vector degeneration problem, we conducted further analysis and discovered two factors that cause this problem: (1) simply averaging input latent vectors causes l 2 -norm shrinkage, and (2) latent vectors with smaller l 2 -norm tend to be decoded into less informative generations. ## convex aggregation in latent space as discussed above, there are two limitations with the de-facto standard simpleavg. first, it causes summary vector degeneration. second, it is ignorant of the decoder generation (in the text space x ) for a summary vector (in the latent space z.) to address the issues, we consider an optimization problem that searches for the best combination of the latent vectors of input reviews that maximizes the alignment between input reviews and generated summaries. we restrict the search space to the convex combinations of input review representations, so the contribution of each input review is always zero or positive. this is based on the assumption that each review in the input set should be either ignored or reflected. hence, we refer to the latent representation aggregation problem as convex aggregation. ## analysis in this section, we conduct a series of additional analysis to verify the effectiveness and efficiency of coop. we also provide detailed descriptions of the setups and additional analysis in the appendix. ## rouge-2/l are shown in appendix. most abstractive unsupervised opinion summarization techniques use a two-stage approach that trains an encoder-decoder model based on the reconstruction objective and generates a summary from the average latent vectors of input reviews using the trained model @xcite . amplayo and lapata (2020) and amplayo et al. ( 2021 ) have expanded the approach by creating pseudo review-pairs to train a summarization model. our study revisits this two-stage approach and develop a latent vector aggregation framework, which can be combined with a variety of opinion summarization models. ## conclusions in this paper, we revisit the unsupervised opinion summarization architecture and show that the commonly used simple average aggregation is suboptimal since it causes summary vector degeneration and does not consider the difference in the quality of input reviews or decoder generations. to address the issues, we develop a latent vector aggregation framework coop, which searches convex combinations of the latent vectors of input reviews based on the word overlap between input reviews and a generated summary. the strategy helps the model generate summaries that are more consistent with input reviews. to the best of our knowledge, coop is the first framework that tackles the latent vector aggregation problem for opinion summarization. our experiments have shown that with coop, two summarization models, bimeanvae and optimus, establish new state-of-the-art performance on two opinion summarization benchmarks. the results demonstrate that our aggregation framework takes the quality of unsupervised opinion summarization to the next stage.