Noisy Pairing and Partial Supervision for Stylized Opinion Summarization (2024)

Abstract: Opinion summarization research has primarily focused on generating summaries reflecting important opinions from customer reviews without paying much attention to the writing style. In this paper, we propose the stylized opinion summarization task, which aims to generate a summary of customer reviews in the desired (e.g., professional) writing style. To tackle the difficulty in collecting customer and professional review pairs, we develop a non-parallel training framework, Noisy Pairing and Partial Supervision (NAPA), which trains a stylized opinion summarization system from non-parallel customer and professional review sets. We create a benchmark ProSum by collecting customer and professional reviews from Yelp and Michelin. Experimental results on ProSum and FewSum demonstrate that our non-parallel training framework consistently improves both automatic and human evaluations, successfully building a stylized opinion summarization model that can generate professionally-written summaries from customer reviews. The code is available at https://github.com/megagonlabs/napa

## introduction opinion summarization, which focuses on automatically generating textual summaries from multiple customer reviews, has received increasing attention due to the rise of online review platforms. different from single-document summarization tasks (e.g., news summarization), which can easily collect a large amount of document-summary pairs, manually creating summaries from multiple reviews is expensive; it is not easy to collect large-scale training data for opinion summarization. to address this challenge, existing studies build pseudo-reviewssummary pairs in a self-supervised fashion (chu and liu, 2019; amplayo and lapata, 2020; suhara et @xcite @xcite or use a small amount of reviews-summary pairs in a few-shot manner @xcite @xcite to train opinion summarization models. however, existing opinion summarization systems have focused on summarizing important opinions in reviews while not paying much attention to the writing style. they leverage customer reviews as pseudo summaries to train models, which generate summaries in the same writing style as the customer reviews as illustrated in figure 2 . on the other hand, professional reviews, such as michelin guide-a prestigious and popular restaurant guide, use a quite different writing style to describe the same type of information. in this paper, we aim to fill this gap between customer and professional reviews by proposing a new branch of opinion summarization-stylized opinion summarization, where the goal is to generate a summary of opinions in the desired writing style. specifically, besides customer reviews, as the input to the conventional opinion summarization task, we use a few example summaries in the desired writing y x 1 ' x 2 ' step through the serene, bamboo-filled entrance and into this dedicated sushi den. the rooms are dedicated to christmas decorations all year round. i came this steak restaurant for dinner. atmosphere is inviting, intimate and calm. you get a super homey feeling when you step through the door, like you're family. ## self-supervised opinion summarization this section describes the standard self-supervised framework for conventional opinion summarization and then the pseudo-reviews-summary pair construction approach @xcite , which is also used as the pre-training method in §5. opinion summarization is a multi-document summarization problem that aims to generate a textual summary text y that reflects the salient opinions given the set of reviews @xmath0, . . . , x n }. due to the unavailability of a sufficient amount of reference summaries for training, a commonly used approach is to create a pseudo-reviews-summary training pair ( x , ỹ) from a massive amount of reviews and trains an opinion summarization model p θ using negative log-loss minimization, ## evaluation we use prosum and an existing opinion summarization benchmark fewsum @xcite to verify the effectiveness and generalizability of napa . for fewsum, we discarded the source reviews from the training dataset to convert fewsum into a stylized opinion summarization benchmark (i.e., in the non-parallel setting). ## analysis 6.1 importance of partial supervision the experimental results in tables 2 and 3 show that napa without partial supervision-just using noisy reviews-summary pairs-demonstrates solid performance for reference-based automatic evaluation metrics. this is a little bit counterintuitive, and this can be attributed to the positive effect of early stopping against noisy training data @xcite . to analyze this point, we conducted an additional experiment by training napa with and without partial supervision for more training epochs. figure 4 shows the rouge-1 f1 score on the validation set of prosum at different training epochs of the napa model trained with or without partial supervision (orange line and green line). as shown in the figure, we find that in the very early stages of training, both the models improve the rouge scores. in the later stage, napa without partial supervision (green line) shows continuous degradation, while napa with partial supervision (orange line) shows robust performance consistently over the entire training process. this observation is aligned with the literature on noisy supervision, which shows that overparametrized models learn simple patterns in the early stages of training and then memorize noise @xcite . on the other hand, it is also known that early stopping is not sufficient under labeling noise @xcite . we observed that napa without partial supervision generated summaries that were less consistent with the source reviews ( ## conclusions this paper proposes stylized opinion summarization, which aims to summarize opinions of input reviews in the desired writing style. as parallel reviews-summary pairs are difficult to obtain, we develop a non-parallel training framework named noisy pairing and partial supervision (napa ); it creates noisy reviews-summary pairs and then trains a summarization model by focusing on the sub-sequence of the summary that can be reproduced from the input reviews. experimental results on a newly created benchmark prosum and an existing opinion summarization benchmark fewsum demonstrate that our non-parallel training framework substantially outperforms self-supervised and text-style transfer baselines while competitively performing well against supervised models that use parallel training data.