PASS : Perturb-and-Select Summarizer for Product Reviews (2021)

Abstract: The product reviews summarization task aims to automatically produce a short summary for a set of reviews of a given product. Such summaries are expected to aggregate a range of different opinions in a concise, coherent and informative manner. This challenging task gives rise to two shortcomings in existing work. First, summarizers tend to favor generic content that appears in reviews for many different products, resulting in template-like, less informative summaries. Second, as reviewers often disagree on the pros and cons of a given product, summarizers sometimes yield inconsistent, self-contradicting summaries. We propose the PASS system (Perturb-and-Select Summarizer) that employs a large pre-trained Transformer-based model (T5 in our case), which follows a few-shot fine-tuning scheme. A key component of the PASS system relies on applying systematic perturbations to the model’s input during inference, which allows it to generate multiple different summaries per product. We develop a method for ranking these summaries according to desired criteria, coherence in our case, enabling our system to almost entirely avoid the problem of self-contradiction. We compare our system against strong baselines on publicly available datasets, and show that it produces summaries which are more informative, diverse and coherent.

## introduction online shopping has become a popular form of purchasing goods even before the most recent acceleration due to the covid-19 pandemic. as ecommerce websites strive to make the shopping process more useful and enjoyable for customers, many interesting challenges arise. one challenge deals with how to surface opinions from product reviews in a concise yet reliable fashion. the research community has addressed this challenge early on, starting from the work of @xcite which defined the task of mining and summarizing customer reviews. more recent advancements have relied on modern deep learning models trained on large collections of unannotated customer reviews @xcite . our first observation relates to the summaries generated by copycat @xcite and fewsum @xcite , two of these sota systems, which tend to mix generic statements such as "would recommend this product to anyone" along with more informative content such as "the sound quality is good" (see large pre-trained transformer-based @xcite models such as openai's @xcite ), google's t5 @xcite , pegasus @xcite , and facebook's bart @xcite have made com-pelling advancements on a host of nlg tasks, including abstractive text summarization. in this work we wish to leverage such models for product reviews summarization, aiming to generally improve the quality of generated summaries, and specifically in terms of their diversity across different products. while we aim to generate humanlike texts, care has to be taken with respect to their correctness. indeed, concerns have been raised regarding the factual consistency of abstractive summaries, i.e., whether the facts conveyed in the summary agree with the source text @xcite @xcite . our second observation relates to this issue of factual consistency in the context of product reviews summarization. our task not only faces the risk of models hallucinating incorrect information, as in traditional abstractive text summarization, but also the risk of generating self-contradicting summaries which are not caused by model hallucinations. the latter can occur when the source documents contradict one another. this situation is quite likely because reviews may disagree on some product aspects or even disagree entirely. for example, review a states a machine is "easy to operate" vs. review b which states it "requires trial and error" (see more examples in our proposed method starts by fine-tuning a strong pre-trained language model for product reviews summarization in a few-shot setup. we then employ an input perturbation method that drops k reviews out of the input and concatenates the remaining reviews in random order. this process, denoted as lko, short for leave k out, produces notable variation between candidate summaries, which increases the model's output diversity. 2 once we have produced a set of candidate 2 diversity here is between candidate summaries for the summaries, we essentially cast our original summary generation problem as a ranking problem. this approach gives us the choice over what kind of summary we are interested in as the final output, i.e. choosing our ranking criteria. as mentioned above, our main concern in this work is producing self-consistent summaries. instead of basing our ranking solely on this criterion, we train a more general coherence summary ranker using human annotated coherence scores @xcite . finally, for each product, we select the top ranked summary as the system's output. we compare our method against strong baselines, comprised of systems introduced in previous work on multi-document opinion summarization, and a t5 language model fine-tuned for abstractive text summarization. we evaluate each over 3 dimensions, of which relevance and coherence are commonly used in summarization @xcite , and our newly introduced metric for cp-diversity. we demonstrate that our method produces high quality summaries which are more informative, diverse and coherent. in summary, the main contributions of this work are: (1) highlight two shortcomings of existing product reviews summarizers, namely low cp-diversity and self-inconsistency, and propose a dedicated metric for the former. (2) propose a method that leverages strong pre-trained models that improve the cp-diversity while significantly reducing the risk of self-inconsistencies. as in traditional summarization, there are two inherently different requirements for the task, a simplified one, in which the goal is to provide an extractive output, i.e., a list of sentences extracted from the review set, or a more advanced one, in which the goal is to provide an abstracsame product, not to be confused with cp-diversity. ## perturb-and-select summarizer in this section, we propose a system that employs a large pre-trained transformer-based model (t5) in a few-shot fine-tuning scheme for multiple reviews abstractive summarization. we aim to leverage the inherent diversity between reviews for a given product to our advantage, by applying systematic perturbations to the model's input during inference. this allows our fine-tuned model to generate multiple different candidate summaries per product, exhibiting variability both in the content being surfaced as well as in the phrasing of said content. we develop a ranking mechanism for selecting the best candidate summary according to desired criteria, which in our case is coherence. we provide an end-to-end diagram of the pass summarizer's components in figure 1 . ## conclusion in this work we highlight two shortcomings of existing product reviews summarization systems, namely low cp-diversity and self-inconsistency. we propose the spr metric to quantify cross prod-tights. these tights are very comfortable and durable. they can be worn with ballet slippers or sandals. the color is beautiful and the fabric is soft. they will last a long time. they are great for transitioning from ballet to ballet. purse. this purse is not as cute as it looks in the picture. it is very small and will not hold a lot of stuff. it would be a great purse if it was a little bigger but it would have been nice to have a purse that would hold more than one purse. protein bar. these bars are a great snack bar. they taste good and have a good amount of protein. they do not have a lot of protein in them so they are not as sweet as some protein bars, but for the price, they are well worth it. tank top. this tank top is well made, fits well, and is comfortable to wear. the only thing is that it runs a little small, so order a size up from what you normally wear. other than that, it's a great top. it's well made and it looks like it will last a long time. love it! pass. these reeboks are great for supporting a high arch and are lightweight and comfortable. they come in a variety of colors and sizes, and are ideal for walking or biking. they are also flexible and well made. t5-ft. these reeboks are a great choice for those with wide feet. they run true to size and the colors are great. they are lightweight and comfortable, yet they are flexible and flexible. they are recommended for people with wide feet. they are also very popular for running and casual wear. fewsum. these running shoes are great! they fit true to size and are very comfortable to run around in. they are light weight and have great support. they run a little on the narrow side, so make sure to order a half size larger than normal. copycat. i love these shoes. they are light weight and comfortable to wear. i have worn them for several months now and they are holding up well. i would recommend them to anyone looking for a comfortable shoe. in future work we plan to investigate the perturb-and-select framework in order to promote summaries with a plethora of desired linguistic characteristics, other than coherence. we shall further explore ways of extending this framework to employ other input perturbation methods and experiment with scenarios of larger scale input. in addition, we plan to further investigate our proposed spr evaluation metric for lexical diversity, by studying its correlation with human judgments. lastly, we believe our proposed framework and evaluation metric may be applicable to other domains of opinion or news summarization. pass. this camera is good to have as a first camera before investing in a dslr. the quality of the pictures is great, and the camera is easy to use. it takes some time to learn about the features and settings, but overall it's a great camera. t5-ft. this camera is a great camera for taking professional photos. it is easy to use and takes excellent pictures. the low light feature is outstanding and will be helpful in museums and other venues where flash is not allowed. the battery is constantly malfunctioning making the camera unusable. the on off button is also malfinctioning. fewsum. this camera is a great camera for the price. it takes great pictures and is easy to use. the only drawback is that the battery life is not as good as the camera that comes with the camera. it would be nice if it had a battery life to last longer. overall, it's a good camera. copycat. this is a great camera for the price. it is easy to set up and use. the only downside is that it takes a while to learn how to use it, but it's not a problem. travel sound conditioner. this is a great product for the price. the sound quality is good and the sound is good. the only problem is that it is not loud enough for a small room. it is loud enough to drown out background noise, but not very loud. overall, it's a good product and would recommend it to anyone. motion sickness tablets. this is a great product at a great price. it is easy to use and easy to take. the pills are easy to swallow and do not take up a lot of space. the price is great for a product that will last a long time. would recommend this product to anyone who suffers from nausea or sickness. digital camera. this camera is a great camera for the price. it takes great pictures and is easy to use. the only drawback is that the battery life is not as good as the camera that comes with the camera. it would be nice if it had a battery life to last longer. overall, it's a good camera. review 1. the machine is very tricky. it requires some trial and errors to make it work right. i do not like to put oil in the dough; however, it appears to me that without oil it is impossible to make tortilla or chapatti. it is useless for me. review 2. fun and easy to use! took me one batch to get my technique worked out, but it was very simple, easy to follow directions. easy clean up too! i would recommend this to anyone looking for an electric tortilla maker! summary. this tortilla maker is a great option for making tortillas but it does require some trial and error to make it work right. it requires some trial and error to make it work right. yes, you should grill them after cooking to get the toasted look. it is easy to use and very easy to clean up. candidate 1. these nugo bars are high quality and they come in a variety of flavors and sizes which make them perfect for serving as a snack or as a replacement for processed foods. they are low glycemic and have a smooth, vanilla-like texture which makes them very good. candidate 2. these nugo bars are high quality and they come in a variety of flavors and sizes which makes them ideal for snacking on the go. the taste is great and the nutritional value is great as well. although they can be a little sweet, they are not too sweet. candidate 3. these nugo bars are high quality and they come in a variety of flavors and sizes. they are low glycemic and have a great taste. while they may be sweet, they can also have a chalky or barky texture. these are great for replacing junk food with healthy snacks. candidate 4. these nugo bars are high quality and they taste great. they are low glycemic, and they contain no added sugar or artificial flavors. these are great for a healthy snack or for a quick breakfast. candidate 5. these nugo bars are very good quality and they come in a variety of flavors. they are high in calories and fiber, and are great for snacking on the go. they are often a bit chewy, but they are definitely worth the money.