Prompted Opinion Summarization with GPT -3.5 (2023)

Abstract: Large language models have shown impressive performance across a wide variety of tasks, including text summarization. In this paper, we show that this strong performance extends to opinion summarization. We explore several pipeline methods for applying GPT-3.5 to summarize a large collection of user reviews in aprompted fashion. To handle arbitrarily large numbers of user reviews, we explore recursive summarization as well as methods for selecting salient content to summarize through supervised clustering or extraction. On two datasets, an aspect-oriented summarization dataset of hotel reviews (SPACE) and a generic summarization dataset of Amazon and Yelp reviews (FewSum), we show that GPT-3.5 models achieve very strong performance in human evaluation. We argue that standard evaluation metrics do not reflect this, and introduce three new metrics targeting faithfulness, factuality, and genericity to contrast these different methods.

## introduction recent years have seen several shifts in summarization research, from primarily extractive models @xcite @xcite @xcite to abstractive models with copy mechanisms @xcite @xcite to pre-trained models @xcite @xcite @xcite . gpt-3 @xcite @xcite and gpt-4 represent another shift: they show excellent zero-and few-shot performance across a variety of text generation tasks. however, their capabilities have not been extensively benchmarked for opinion summarization. unlike news, where extractive lead baselines are often highly effective, opinion summarization requires balancing contradictory opinions and a higher degree of abstraction to convey all of the viewpoints faithfully. in this paper, we apply gpt-3.5, specifically the text-davinci-002 model, foot_0 to the task of opinion summarization, focusing on reviews of products, hotels, and businesses. applying gpt-3.5 in this setting is not straightforward, as the combined length of the reviews or posts may exceed the model's maximum input length. furthermore, we find that certain styles of inputs can lead to gpt-3.5 simply echoing back an extract of the inputs. to mitigate these issues, we explore a family of pipelined approaches, specifically (1) filtering a subset of sentences with an extractive summarization model, (2) chunking with repeated summarization, and (3) review-score-based stratification. in the context of aspect-oriented summarization, we also explore the inclusion of a sentence-wise topic prediction and clustering step. we show that our approaches yield high-quality summaries according to human evaluation. the errors of the systems consist of subtle issues of balancing contradictory viewpoints and erroneous generalization of specific claims, which are not captured by metrics like rouge @xcite or bertscore @xcite . this result corroborates work calling for a re-examination of current metrics @xcite and the need for fine-grained evaluation @xcite . we therefore introduce a set of metrics, using entailment as a proxy for support, to measure the factuality, faithfulness, and genericity of produced summaries. these metrics measure the extent of over-generalization of claims and misrepresentation of viewpoints while ensuring that summaries are not overly generic. our results show that basic prompted gpt-3.5 produces reasonably faithful and factual summaries when the input reviews are short (fewer than 1000 words); more sophisticated techniques do not show much improvement. however, as the input size all staff with the exception of the front desk were so polite and friendly. the housekeeping staff did a great job keeping the rooms clean. the manager would not register our complaint. the staff was found to be polite and friendly, with special praise given to the housekeeping staff. most reviewers enjoyed their experience. however, one reviewer specifically complained about the manager… the reviews were generally positive about the service, with praise for the housekeeping staff and chefs. some reviewers did find their room damp and dark, but were happy to be upgraded to a better suite. chunkwise summary per aspect final summary per aspect great staff… loved it… the bed… stayed for… terrible… arrived… review sentences grouped by @xmath0, repeated summarization leads gpt-3.5 to produce generalized and unfaithful selections of viewpoints relative to the first round. we demonstrate that using qfsumm @xcite , an extractive summarization model, to filter out sentences prior to gpt-3.5 (instead of multi-level summarization) can slightly help with factuality and faithfulness. the resulting summaries also present a more specific selection of viewpoints but are generally shorter and use a higher proportion of common words. a topicwise clustering and filtering step pre-pended to the pipeline alleviates these issues while relinquishing a portion of the gains on factuality and faithfulness. our main contributions are: (1) we introduce two approaches to long-form opinion summarization with gpt-3.5, namely, hierarchical gpt-3.5 summarization with chunking, and pre-extraction with an extractive summarization model. (2) we establish the strength of these approaches with a human study and demonstrate the need for objective and automatic means of evaluation. (3) we develop three entailment-based metrics for factuality, faithfulness, and genericity that are better suited to evaluate extremely fluent summaries as compared to metrics based on n-gram matching. the relevant artifacts and code for this work are publicly available and can be found at https: //github.com/testzer0/zs-summ-gpt3/ . ## motivation and problem setting review summarization involves the summarization of the text of multiple reviews of a given product or service into a coherent synopsis. more formally, given a set of reviews @xmath1, we define a summarization system s to be a function that takes as input the combined reviews c and then produces k output sentences @xmath2, written as @xmath3, where c ≡ combine(r) is typically obtained by concatenating the review sentences. we use the notation combine to refer to the combination of both sentences and reviews. we can also instantiate this pipeline for aspectoriented review summarization, which involves the summarization of multiple reviews conditioned on an aspect a (such as 'cleanliness'). in particular, the summarization is written as @xmath4. we consider aspect-agnostic review summarization as a special case of aspect-oriented review summarization with the aspect 'none' for notational simplicity. ## new tools for evaluation and analysis enabling fast automatic evaluation of systems will be crucial for the development of future opinion summarizers. furthermore, when a large number of reviews are presented to a system, it may be nearly impossible even for a dedicated evaluator to sift through all of them to evaluate a summary. we investigate the question of how we can automate this evaluation using existing tools. one of the areas where automatic evaluation may help is faithfulness. since faithfulness represents the degree to which a system is accurate in representing general consensus, it requires measuring the proportion of reviews supporting each claim of a summary. a viewpoint with larger support is more popular and, consequently, more faithful. our key idea is to use entailment as a proxy for support. past work @xcite has used natural language inference (nli) models to assess summary factuality by computing entailment scores between pairs of sentences. however, the summaries produced by gpt-3.5 and related pipelines often consist of compound sentences that contrast two viewpoints. in addition, gpt-3.5 prefers to say "the reviewers said..." instead of directly stating a particular viewpoint. we found these artifacts to impact the entailment model. we use a split-and-rephrase step to split these sentences into atomic value judgments by prompting gpt-3.5 as shown in figure 4 . we then use the zero-shot entailment model from sum-mac @xcite to compute the entailment scores for these atomic value judgments. similar to the approach in the summac paper, we observe that a summary statement is factual when strongly entailed by at least one sentence and thus select the top entailment score of each summary sentence as its factuality score, and aggregate this score to produce per-system numbers. the choice of the model as well as that of using gpt-3.5 for the split-and-rephrase step are explained further in appendix b, and the relevant metric of abstractiveness is discussed in appendix d. a system could potentially game this metric by producing relatively "safe" statements (like most reviewers found the rooms clean). we therefore the reviews of the hotel were generally positive, with most people finding the rooms clean and the staff apologetic. however, some found the carpets to be stained, and one reviewer reported dust balls in their room. ## conclusion in this work, we show that gpt-3.5-based opinion summarization produces highly fluent and coherent reviews, but is not perfectly faithful to input reviews and over-generalizes certain viewpoints. rouge is unable to capture these factors accurately. we propose using entailment as a proxy for support and develop metrics that measure the faithfulness, factuality, and genericity of the produced summaries. using these metrics, we explore the impact of two approaches on controlling the size of the input via pre-summarization on two opinion summarization datasets. with the reasonably sized inputs of fewsum, gpt-3.5 and cg produce faithful and non-generic outputs. however, as we move to long-form review summarization, the factuality and faithfulness of these approaches drop. a preextraction step using qfsumm helps in this setting but leads to generally shorter and more generic summaries; a topic clustering step can then make summaries less generic and more relevant at a small cost to faithfulness and factuality. we hope that our efforts inspire future improvements to systems and metrics for opinion summary evaluation.