Disentangling Text Representation With Counter-Template For Unsupervised Opinion Summarization (2023)

Abstract: Approaches for unsupervised opinion summarization are generally based on the reconstruction model and generate a summary by decoding the aggregated representation of inputs. Recent work has shown that aggregating via simple average leads to vector degeneration, generating the generic summary. To tackle the challenge, some approaches select the inputs before aggregating. However, we argue that the selection is too coarse as not all information in each input is equally essential for the summary. For example, the content information such as “great coffee maker, easy to set up” is more valuable than the pattern such as “this is a great product”. Therefore, we propose a novel framework for unsupervised opinion summarization based on text representation disentanglement with counter-template. In specific, a disentangling module is added to the encoder-decoder architecture which decouples the input text representation into two parts: content and pattern. To capture the pattern information, a counter-template is utilized as supervision, which is automatically generated based on contrastive learning. Experimental results on two benchmark datasets show that the proposed approach outperforms the state-of-the-art baselines on both quality and stability.

## introduction with the unprecedented development of online interactive platforms, opinion summarization has received significant interest in natural language processing communities. unlike other summarization tasks for news, wikipedia, and medical treatment records, opinion summarization pays more attention to user opinions in product reviews, blog journals, and social media texts. opinion summarization has great potential in many application scenarios. due to the lack of large-scale annotated data, opinion summarization is generally formalized as * corresponding author. an unsupervised learning framework. a series of reconstruction models have been developed @xcite @xcite . the training goal is to reconstruct input text through an encoder-decoder architecture such as autoencoders (ae), variational autoencoders (vae). as shown in the upper part of figure 1 , when generating summaries, the encoder is employed to aggregate the text representations from a set of texts via averaging (the 'mean' module in figure 1 ) to obtain a summary representation. the representation is used to generate the summary by the decoder. recently, it has been demonstrated in @xcite that simply averaging tends to generate generic summaries, as shown in the upper part of figure 1 . to tackle the challenge, a 'select' module is incorporated in coop @xcite to strictly select input text. however, we argue that it has such a limitation: coarse-grained selection. there might be some information hidden in the abandoned reviews which is important for summarization. moreover, the kept reviews might contain some redundant information, leading to generic or incorrect sentences, as shown in the middle of figure 1 . in this paper, instead of treating text vectors as a whole, we assume that the text representation consists of content and pattern information. the content information is summary-relevant and related to specific product characteristics, such as aspects, emotions, and opinions illustrated as the orange part of texts in figure 1 . the pattern information marked in blue describes some common patterns that occur frequently in the corpus and can be used to describe most products. both are valuable for reconstruction training. however, when text representations are aggregated to obtain a summary representation, the repeated pattern information is more conspicuous, which squeezes valuable content information and results in the generation of overly generic summaries. to separate the content information from the pattern information, the text representation needs to be disentangled. recently, qin et al. ( 2020 ) projects feature into orthogonal space to disentangle representation. however, it is not straightforward to disentangle in the same way as no supervision labels are available for the pattern information. to create supervision signals of the pattern information, we elaborately design a text template to capture the common semantic patterns shared in the corpus. it is called counter-template since the extracted pattern representation works as a negative guidance against the content information. therefore, we propose a novel framework based on text representation disentanglement with counter-template for unsupervised opinion summarization (trace). a disentangling module is added to the encoder-decoder architecture which orthogonally decouples the text representation into two parts: content and pattern. the content representation is used to reconstruct the input text and the pattern representation is supervised by the counter-template. moreover, to construct the counter-template automatically, a novel approach based on contrastive learning and average strategy is proposed. a counter-template generator is trained to not only preserve the general generation constraints in vae but also make all text representations in the same batch similar to each other using contrastive learning. the counter-template is obtained by decoding the average of several text representations. the main contributions of this paper are summarized as follows: ## related work opinion summarization generally focuses on short and numerous product reviews. according to whether summaries are extracted from original texts, they can be divided into extractive approaches and abstractive approaches. extractive approaches @xcite @xcite capture the opinion of product reviews through emotional polarity or aspect information. some methods attempt to obtain more flexible sentences through graphs @xcite or decision theory @xcite . recently, @xcite leverage vector quantization @xcite to capture a semantic sense. on this basis, semae @xcite uses dictionary learning to capture fine-grained and diverse semantics. as text generation technology continues to evolve, a series of end-to-end abstractive methods have been developed for opinion summarization @xcite ; amplayo and lapata, 2020; @xcite . they use the encoderdecoder architecture and employ the average representation of inputs for summarization. another type of method @xcite @xcite focuses on modeling aspects and emotional information. @xcite use a two-stage approach that first identifies opinion phrases and then uses the phrases to generate smooth sentences. other works use aspect seed words @xcite or implicit aspect code (amplayo et al., 2021b), making the summary generation more controllable. recently, @xcite show that averaging text representations simply tends to generate overly generic summaries. then they use word overlapping to search for a better subset of input texts when inferring a summary. after that, wassos @xcite optimizes the process of summary inference by approximating the wasserstein barycenter to construct a better summary distribution. although both wassos and the proposed trace conduct text representation decoupling, trace differs from wassos in the following two aspects: (1) different purposes. trace aims to tackle the coarse-grained selection problem in opin-ion summarization while wassos employs text disentanglement for the better generation. (2) different ways. trace designs a novel way to separate the content and pattern information via the special design counter-template while wassos employs a general way to disentangle syntactic and semantic spaces via the linearized parse tree sequence and the bag-of-words distribution @xcite . ## methodology in this section, we describe trace, a novel disentangled framework with counter-template guidance for unsupervised opinion summarization. we first present the overview of the summarization model. then, we introduce the counter-template generation approach based on a contrastive learning model and average inference strategy. finally, we describe the detailed components of trace and explain how to train the model. ## experiments to investigate the effectiveness of the proposed method, the best and the average results were performed on two datasets. in addition to the best performance, the mean and standard deviation were reported to evaluate the stability of the model. besides, two analysis experiments were conducted for the counter-template. ## conclusion to avoid generating generic summaries, we propose trace, a novel framework for unsupervised opinion summarization based on text representation disentanglement with counter-template. the additional disentanglement module inside the encoderdecoder architecture decouples the pattern and content information in the text representation under the guidance of the special counter-template. experimental results on amazon and yelp-res show the proposed approach outperforms the state-of-the-art baselines on both quality and stability.