A Hybrid Approach to Cross-lingual Product Review Summarization (2022)

Abstract: We present a hybrid approach for product review summarization which consists of: (i) an unsupervised extractive step to extract the most important sentences out of all the reviews, and (ii) a supervised abstractive step to summarize the extracted sentences into a coherent short summary. This approach allows us to develop an efficient cross-lingual abstractive summarizer that can generate summaries in any language, given the extracted sentences out of thousands of reviews in a source language. In order to train and test the abstractive model, we create the Cross-lingual Amazon Reviews Summarization (CARS) dataset which provides English summaries for training, and English, French, Italian, Arabic, and Hindi summaries for testing based on selected English reviews. We show that the summaries generated by our model are as good as human written summaries in coherence, informativeness, non-redundancy, and fluency.

## introduction summarizing product reviews with thousands of reviews is a daunting task. at the same time since this task is extremely time consuming to be done by humans, there are no annotated training datasets available for it. hence, almost all existing approaches rely on unsupervised methods such as latent semantic analysis (lsa) @xcite , lexrank @xcite , mean-sum @xcite , and copycat @xcite to name a few. however, the main two shortcomings of these methods are: (i) they do not provide comparable summaries in terms of coherency and fluency to human written summaries @xcite , and (ii) they cannot be used for cross-lingual summarization (i.e., provide summaries in a target language given the summaries in the source language). 1 1 although it is possible to machine translate the summaries, it will add an extra inference time and may reduce the in order to address both of these shortcomings, we propose a hybrid two-step summarization approach: (i) an unsupervised extractive summarization step to extract the most representative sentences out all the reviews, and (ii) a supervised abstractive summarization step to summarize the extracted sentences into a coherent short summary. the main advantage of this approach is that we can rely on a light-weight unsupervised method for the extractive step to reduce the number of reviews and focus on a more expensive supervised model for the abstractive part (and actually collect human summaries for training it). moreover, we can use state-of-the-art multilingual transformer models to develop a cross-lingual abstractive summarizer relying only on a monolingual extractive step. for extractive summarization, we use lsa @xcite which is computationally efficient and provides a good performance in our application. to create training data for the abstractive step, we select 1000 products, for each product extract 10 most informative sentences out of all the english reviews using lsa, and obtain 3 summaries per product by asking amazon mechanical turkers to write a short summary of the provided 10 sentences in english (further details are provided in section 3). an example datapoint is provided in table a1 in the appendix. for testing, we repeat the same process for another 280 products, but this time we ask turkers to quality by translating name of the products etc. finally, we train multiple transformer based models on cars training data and evaluate their performance on the test data through both automatic and human evaluations. we show that our approach provides informative summaries that are as good as human written summaries and can generalize well to the categories not in the training data (an example of a summary generated by our model is shown in table a1 in the appendix). the main contributions of our work are two fold: 1) introducing a scalable and productionalizable approach for cross-lingual product review summarization capable of producing summaries in english, french, spanish, italian, arabic, and hindi from english reviews, and 2) demonstrating that our abstractive summarization model outperforms stateof-the-art opinion summarization method @xcite . ## related work some of the most widely used extractive summarization techniques are based on latent semantic analysis (lsa) @xcite , in which a matrix that represents the importance of words in sentences is created, and singular value decomposition is used to select the sentences with the highest relevancy; or bayesian topic modeling @xcite , in which a generative model is used to represent documents as mixtures of latent topics, where a topic is a probability distribution over words. other extractive methods make use of graph methods @xcite and machine learning methods @xcite . abstractive summarization methods can be structure-based, in which a data structure is used to generate the new summary (examples include treebased methods @xcite , template-based methods @xcite and rule-based methods). more recently, the use of encoder-decoder architectures in transfer learning frameworks have facilitated the use of pre-trained encoders to generate document representations which are then used to generate a new summary @xcite @xcite . since then there has been increasing efforts to tackle many of the challenges posed by this task: nallapati et al. (2016) adds linguistically motivated embeddings and pointer networks to deal with outof-vocabulary words; paulus et al. (2018) adds reinforcement learning to the training objective to lessen exposure bias; cohan et al. (2018) proposes a hierarchical model to encode the discourse structure research papers and an attention-based decoder to generate the summaries; gehrmann et al. (2018) improves content selection performance on neural summarizers by incorporating an extra attention step to constrain on more likely phrases; desai et al. (2020) incorporates two transformer models to predict the saliency and plausibility of sentence deletion in compressive summarization; mao et al. (2020) introduces token-level constraints to improve factual consistency. to improve faithfulness dou et al. (2021) proposes gsum, a general guided summarization framework that can make use of external guiding policies to ensure faithfulness to the source documents. on the topic of cross-lingual summarization, @xcite proposes a pre-training strategy for natural language generation tasks on both monolingual and multi-lingual settings followed by monolingual fine-tuning on the downstream task, and show that the resulting model can generalize to new languages. @xcite trains low resource cross-lingual summarization systems on automatically translated input and clean references, improving on a standard copy-attention summarizer on low resource languages and also evaluates on an unseen language. the task of opinion or review summarization, which this paper focuses on, is receiving increased attention due to real-world practical usage. @xcite proposes fewsum, a hierarchical framework for few-shot multi-document review summarization that consists on a transformerbased generator followed by plug-in network that switches the generator into a summarizer. few-sum is the main system we will use for comparison throughout this paper. in a recent concurrent work @xcite , the authors create a dataset of product summaries from a set of product reviews and propose to use joint learning to select a subset of reviews and then summarize from them. however, they use professional written summaries from various websites as gold summaries (which may not be based on customer reviews at all, leading to model hallucination). the main advantage of our approach in production is its decoupled design. namely, the the extractive part of our system can always be improved to extract more informative sentences without a need to retrain the abstractive summarizer. finally, @xcite proposes the task of extreme summarization from multiple product reviews by extracting a single sentence that is concise, relevant and supported by multiple reviews. in our work, we propose to extract the most relevant sentences from each set of reviews, rank them, and use the top 10 reviews to create product summaries. nevertheless, our decoupled design allows us to use @xcite 's method as our extractive summarizer and improve our end to end system in the future. ## cars dataset in this section, we describe the steps in creating the cross-lingual amazon reviews summarization (cars) dataset. ## conclusion we introduced a hybrid approach to cross-lingual product review summarization which provides summaries on different target languages by only relying on english reviews. we demonstrated that our approach results in review summaries that are as good as human written ones in english and spanish (and comparable to gold summaries in other languages based on automatic evaluation metrics). we also showed that our pre-fine-tuning plus fine-tuning approach can outperform state-of-theart in few-shot abstractive review summarization. moreover, since our abstractive summarizer is trained on summarizing a few selected (maybe unrelated) sentences, our end to end system can be improved by improving the extractive summarization component only without retraining the more expensive multilingual encoder-decoder architecture that we used for abstractive summarization (which is very desirable and cost saving feature in production systems). the main shortcoming of our work is that it does not provide a mechanism to evaluate the correctness of the generated summaries which is part of our future work.