O pine S um: Entailment-based self-training for abstractive opinion summarization (2023)

Abstract: A typical product or place often has hundreds of reviews, and summarization of these texts is an important and challenging problem. Recent progress on abstractive summarization in domains such as news has been driven by supervised systems trained on hundreds of thousands of news articles paired with human-written summaries. However for opinion texts, such large scale datasets are rarely available. Unsupervised methods, self-training, and few-shot learning approaches bridge that gap. In this work, we present a novel self-training approach, OpineSum for abstractive opinion summarization. The self-training summaries in this approach are built automatically using a novel application of textual entailment and capture the consensus of opinions across the various reviews for an item. This method can be used to obtain silver-standard summaries on a large scale and train both unsupervised and few-shot abstractive summarization systems. OpineSum outperforms strong peer systems in both settings.

## introduction abstractive summarization is particularly promising for fluently comparing opinions from a set of reviews about a place or product. however, while language models trained on huge numbers of source-summary pairs have driven summarization performance in domains such as news, it is harder to find such pairs on the web for opinions, and immensely difficult to present tens or hundreds of reviews to human annotators and train them to write informative summaries. this paper presents a new self-training approach that automatically leverages common opinions across reviews, for example as in so far, many abstractive summarization methods for opinions are based on auto-encoders @xcite @xcite , and do not use any supervision from r1 ...very large and clean with a nice size kitchen. the hotel is located right across the street from balboa park and within walking distance of a rite aid drugstore.. r2 ...reserve a refurbished room and get that promise in writing! the location was great for tourists, right across from balboa park. you could walk to the zoo (about 1/4 mi)... r3 ...i decided to stay at the park manor suites hotel since it seemed to be close to san diego zoo. the hotel is conveniently located in front of balboa park, walking distance to san diego zoo,... r4 ...the staff are both pleasant and professional. hotel is across from balboa park on sixth ave. this is the park west area, and features a diverse array of restaurants... r5 ...as other reviewers have said, it's easy to be here without a carbalboa park is just across the road and the airport is a short taxi ride away. ## related work opinion summarization is a widely studied problem, where the role of sentiment, and product aspects (such as 'lens' and 'focus' for a camera) are well documented. we focus on abstractive methods for general-purpose summaries (not aspect-based), and we overview the closest approaches here. unsupervised neural networks. as in other areas of text generation, modern opinion summarization methods are also predominantly neural networks. since large scale training data is largely absent in this domain, many prior methods are unsupervised. common techniques include auto-encoders @xcite and associated generative approaches such as vaes @xcite . the underlying goal in these approaches is also to steer systems towards common information in the source reviews. our work also presents an unsupervised method, but based on encoder-decoder models. self-training methods. some very recent solutions take advantage of recent large pretrained encoder-decoder models via self-training (amplayo and lapata, 2020; @xcite @xcite . these approaches create large number of pairs of source review sets, paired with a pseudo or silver summary as an approximate target. in all these methods, one of the reviews from the source set is taken as the pseudo summary, and other random reviews or topically related reviews to the target is taken as the set of source reviews. this dataset is then used for further training of encoder-decoder transformer models to suit the review domain. these self-trained models are usually better than unsupervised generative ones. while allowing a favorable paradigm shift, and better performance, there are a few limitations of this type of self-training. as pointed out by @xcite , reviews are considerably diverse from one another. so an objective that generates a review from other reviews will need to also predict content not present on the source side, a major difference from actual summaries of reviews. such pseudo-summaries also contain a lot of first person language which is less desirable. in this work, we present a novel self-training method. we also create silver-summaries on a large scale. however, our summaries actually contain propositions from multiple input reviews and in particular those which are reflective of the consensus among the review authors. these summaries are more powerful and faithful signals, and move the training task away from review generation. few-shot learning. with increased use of encoderdecoder models, methods have also been proposed to efficiently augment the training with a small number of human-generated summaries (50 to 100). @xcite train transformer models on a small number of examples and use a ranking approach for inference. in bražinskas et al. (2020a) , a plug-in network is added to predict desired summary properties and thereby augment training signals. @xcite introduce a few additional parameters in the form of adaptors and only these are finetuned instead of the full network, leading to efficient training on few examples. we also demonstrate our self-trained model in few-shot settings. consensus as a goal for summarization. when there are multiple input texts, intuitively the common information across them is one important signal for summary-worthy content. multi-document news summarization has exploited frequency from early times @xcite to most recent ones @xcite . consensus is also used as a goal for summarizing scientific publications around health topics @xcite , and identifying agreement and discrepancies in wikipedia document clusters @xcite . instructions to annotators in multiple annotation efforts for opinion summarization explicitly ask annotators to capture what is common and popular @xcite . most recently, agreement among opinions has been proposed for evaluating opinion summaries @xcite . our self-training approach explicitly captures consensus statements. we acknowledge that majority opinions are only one indicator of useful content, and that others (eg. aspects) will complement it. ## textual entailment to identify consensus among review users we propose a novel approach to create silver source-summary pairs for abstractive opinion summarization. a central idea here is the use of textual entailment to find statements reflecting user consen-sus. we first present our definition of the idea and describe the steps involved in silver data creation. ## datasets we use two types of data in our experiments: unlabelled and eval. the first is an unlabelled review corpus (no gold or human summaries are available for the items). this dataset is used to create silverstandard summaries for self-training. the second type is an evaluation dataset containing a much smaller set of items (not seen during training) and here for each item, the set of source reviews is paired with one or more human summaries. we use publicly available anonymized corpora of reviews and summaries as per their terms. ## experiments in this section, we explain how we trained our abstractive summarization models. ## results we present our findings on the space-eval and amazon-eval testsets ( section 4). we compute the automatic rouge @xcite ) metric, and also confirm the improvements with human raters. for comparison, we included recent systems where the best system outputs were available publicly. ## conclusion we have presented a simple self-training approach which leads to gains on both unsupervised and fewshot abstractive opinion summarization. our work is one of the first to demonstrate how an intuitive idea of review consensus can be incorporated during self-training. it opens up a number of challenges and new problems for future work. in particular, while our silver data contains the provenance for each top proposition-meaning the set of reviews which support each the propositionthis information is only minimally used at the moment. future work could explore how the entailment weights (scores) of each proposition and their links to entailing reviews could be used for further improvements and faithful generation. we also hope that such self-training models could serve as good checkpoints for other tasks in the opinion domain such as review helpfulness or product popularity prediction.