Efficient Few-Shot Fine-Tuning for Opinion Summarization (2022)

Abstract: Abstractive summarization models are typically pre-trained on large amounts of generic texts, then fine-tuned on tens or hundreds of thousands of annotated samples. However, in opinion summarization, large annotated datasets of reviews paired with reference summaries are not available and would be expensive to create. This calls for fine-tuning methods robust to overfitting on small datasets. In addition, generically pre-trained models are often not accustomed to the specifics of customer reviews and, after fine-tuning, yield summaries with disfluencies and semantic mistakes. To address these problems, we utilize an efficient few-shot method based on adapters which, as we show, can easily store in-domain knowledge. Instead of fine-tuning the entire model, we add adapters and pre-train them in a task-specific way on a large corpus of unannotated customer reviews, using held-out reviews as pseudo summaries. Then, fine-tune the adapters on the small available human-annotated dataset. We show that this self-supervised adapter pre-training improves summary quality over standard fine-tuning by 2.0 and 1.3 ROUGE-L points on the Amazon and Yelp datasets, respectively. Finally, for summary personalization, we condition on aspect keyword queries, automatically created from generic datasets. In the same vein, we pre-train the adapters in a query-based manner on customer reviews and then fine-tune them on annotated datasets. This results in better-organized summary content reflected in improved coherence and fewer redundancies.

## introduction online reviews play an important role in purchasing decisions we make. they inform us about customer experiences -what aspects users like and dislike, and ultimately, whether a product or service is worth buying. although significant progress has been made in supervised summarization in nonsubjective single-document context, such as news articles @xcite @xcite , modern deep learning methods rely on large amounts of annotated data, but these are not readily available in the review or opinion summarization domain and are expensive to produce. a key obstacle making such annotation expensive is that annotators need to consider multiple texts when writing a summary, which can be tens or even hundreds for realistic settings. hence, most available datasets have human-written summaries for less than 100 products. the lack of sufficiently large annotated datasets led to a variety of unsupervised abstractive models (e.g., copycat @xcite , mean-sum @xcite ; denoisesum (amplayo and lapata, 2020)) that are trained on large collections of unannotated customer reviews. however, as the models are never exposed to actual summaries, they cannot capture their expected characteristics. this results in generated summaries mimicking the informal style of customer reviews and containing hallucinations and unimportant details. these limitations were addressed by few-shot methods learning from a handful of human-written summaries. @xcite . the first proposed few-shot learning model, fewsum @xcite , relies on various static features capturing differences between a customer review and a summary. while that model is more robust to overfitting, such features require manual, domain-specific engineering and can be sub-optimal for capturing correspondences between texts on the semantic level. we propose a simpler approach -adasum -which is based on adapters @xcite . as we explain next, the adapters are pre-trained on customer reviews in a task-specific manner, and subsequently fine-tuned on gold summaries. adapters. we utilize a pre-trained model with powerful language understanding and @xmath0 leave-one-out pre-@xmath1 leave-one-out pre-training @xmath2 ## related work extractive opinion summarization has been an active area of research @xcite @xcite @xcite . for example, a more recent extractive method of @xcite decouples the summarization procedure into multiple steps with separate models. other earlier approaches @xcite relied on text planners and templates, which, however, restrict the output. abstractive opinion summarization is an emerging branch @xcite @xcite . customer reviews were used to train unsupervised summarizers in amplayo and lapata (2020); bražinskas et al. (2020b); @xcite . the fewshot model fewsum @xcite was also pre-trained on customer reviews before fine-tuning. in this work instead, we focus on pretraining adapters to avoid catastrophic forgetting adasum (5%) this thomas the train costume is cute and functional. the size is perfect for a toddler or 2 year old and the candy pouch is large enough to carry a lot of candy. the costume can be customized with googly eyes, pumpkin patch, spiders, bats, or train tracks to make it more suitable for a child of any age. the fit is comfortable and the fit quality is great. the only complaint i have is the quality of the felt fabric. ## conclusions in this work, we improve few-shot learning for opinion summarization with adapters pre-training on customer reviews in the end task-specific manner. in this way, the model learns in-domain specifics, which reduces semantic mistakes in generated summaries. we show that our approach leads to more than 2.0 and 1.3 rouge-l points improvement over the entire model's fine-tuning on the amazon and yelp datasets, respectively. further, we propose a simple method for few-shot query-based summarization. the queries consist of aspect keywords reflecting potential user interests. we create these queries automatically and show that pre-training is crucial for the end task, significantly improving performance. finally, in human evaluation, we demonstrate that the query-based model generates more coherent and less redundant summaries. and fine-tuning techniques can be applied to a broader set of multi-document summarization domains (e.g., news) and can be considered in the future work. also, while we tested our approach only with bart, we believe that it would work with other pre-trained encoder-decoder models, like pe-gasus @xcite .