Aspect-based Summaries from Online Product Reviews: A Comparative Study using various LLM s (2024)

Abstract: In the era of online shopping, the volume of product reviews for user products on e-commerce platforms is massively increasing on a daily basis. For any given user product, it consists of a flood of reviews and manually analysing each of these reviews to understand the important aspects or opinions associated with the products is difficult and time-consuming task. Furthermore, it becomes nearly impossible for the customer to make decision of buying the product or not. Thus, it becomes necessary to have an aspect-based summary generated from these user reviews, which can act as a guide for the interested buyer in decision-making. Recently, the use of Large Language Models (LLMs) has shown great potential for solving diverse Natural Language Processing (NLP) tasks, including the task of summarization. Our paper explores the use of various LLMs such as Llama3, GPT-4o, Gemma2, Mistral, Mixtral and Qwen2 on the publicly available domain-specific Amazon reviews dataset as a part of our experimentation work. Our study postulates an algorithm to accurately identify product aspects and the model’s ability to extract relevant information and generate concise summaries. Further, we analyzed the experimental results of each of these LLMs with summary evaluation metrics such as Rouge, Meteor, BERTScore F1 and GPT-4o to evaluate the quality of the generated aspect-based summary. Our study highlights the strengths and limitations of each of these LLMs, thereby giving valuable insights for guiding researchers in harnessing LLMs for generating aspect-based summaries of user products present on these online shopping platforms.

## introduction the present age of e-commerce provides online consumers with a wide range of online product reviews. these reviews contain crucial information and highlight insights about the products, thereby showcasing the strengths, quality and performance. also, the popularity of user products on these platforms is growing at an unbelievable rate. the influx of people accessing e-commerce websites is attributed due to the presence of written product reviews by the users themselves and not by the origin of brands. this has helped prospective buyers to decide whether to buy the product based on the reviews and experiences shared by the users on these e-commerce platforms as they are considered to have more credibility and trustworthiness by the potential buyers @xcite . summarizing online product reviews is a daunting nlp task. it incurs a lot of computation costs in terms of time and other resources. the goal of summarization goes beyond simply identifying and extracting sentences, synthesizing reviews, and providing an overall summary. it becomes more meaningful when the summary is generated based on specific product features. instead of a generic summary, a more focused summary on particular aspects of the product is preferred. a promising solution to this problem is aspect-based summarization, which identifies key aspects of the product and generates a summary for each extracted aspect. the rise and power of large language models (llms) such as @xcite , llama (ollama, 2023), gemma @xcite , mistral @xcite , mixtral (face, 2024a), qwen @xcite and others has brought a huge revolution in the field of natural language processing (nlp) including text summarization. the capability of these models is not just restricted to understanding the context, structure, and language of the input text, but it has found a huge potential and improvements in language generation or abstraction approaches. the research experiment demonstrated in this paper is inspired by (p bhagat, 2023), which uses the chi-square test statistical measure to automatically calculate the aspect-based polarity of sentiment words in a given domain. the method used in this paper also helps in discovering strong domain-specific polar adjectives that might be missing in universal sentiment lexicons. the proposed work attempts to accurately identify product aspects and the model's ability to extract relevant information and generate concise summaries. the remainder of the paper is organized as follows. section 2 describes the related work studied. section 3 is the titled proposed methodology and explains the process of generating an aspect-based summary from the given set of reviews for a specified product. section 4 explains the implementation details and the datasets used. section 5 elaborates the experimental details and the evaluation metrics used. section 6 presents the results and discussions, and finally, section 7 states the conclusion. ## related work this section provides insight into a survey of various aspect-based summarization methods and different types of large language models (llms). ## proposed methodology the proposed work attempts to identify relevant product aspects from the given set of reviews and the llms ability to extract relevant information and generate concise summaries. our algorithm begins by identifying and extracting reviews specific to a given product. for each product, it filters reviews from a larger set by matching the product identifier called amazon standard identification number (asin). the resulting set contains only reviews belonging to the specified product id. this ensures that subsequent analysis focuses only on the reviews relevant to a particular product. after extracting the product-specific reviews, the algorithm narrows the focus to reviews with the most extreme ratings-either 1-star (negative) or 5star (positive). these reviews are typically the most expressive in terms of customer feedback and sentiment, making them ideal for aspect-based summary and sentiment analysis. the filtered set includes only the 1-star and 5-star reviews for each product. the next step involves cleaning the text of each review. any numeric values or special characters (such as punctuation) are removed. this cleaning process ensures that only meaningful, interpretable words remain in the reviews. this is crucial for improving the accuracy of the subsequent languageprocessing tasks. to further refine the reviews, the algorithm ap-plies two additional filters. firstly, it ensures that only reviews with a word count between 50 and 100 are kept. reviews that are too short may lack meaningful content, while overly long reviews could introduce noise. secondly, if the number of filtered reviews exceeds a predefined limit (max), then only the first max reviews are retained by randomly considering 5000 reviews. at this point, each review is tokenized using part-of-speech (pos) tagging. this process breaks down the review into individual words and labels them with their grammatical roles (e.g., noun, verb). the algorithm identifies and extracts specific aspects (features) of the product from each tokenized review. once the aspects are identified, the algorithm applies stemming, which reduces each word to its root form (e.g., "running" becomes "run"). this helps in normalizing different forms of the same word. after stemming, similar aspects are grouped together or merged to avoid redundant or overly similar aspects for simplifying the analysis work. next, the algorithm calculates how frequently each aspect appears in the reviews. the resulting set contains the frequency of each aspect. the algorithm limits the number of aspects by keeping only the most frequent ones. this helps in focusing on the most commonly discussed features/aspects of the product. using a pre-trained language model (llm), the algorithm generates an overall summary of the reviews, thereby providing general sentiments and feedback from the customers. in addition to the overall summary, the algorithm generates an aspect-based summary for every identified aspect. the generated summary is based only on reviews related to that aspect, thereby providing detailed insights into the feedback on specific features of the product. after generating both the overall and aspectbased summary, the algorithm combines them to form a full summary, which includes both general sentiments as well as specific feedback on individual aspects. finally, the algorithm evaluates the quality of the generated summary. it compares the full summary with a reference summary (amazon's official summary) using predefined evaluation metrics. the resulting evaluation scores indicate the effectiveness and accuracy of the generated summary. to summarize, our proposed methodology provides a structured way to process, clean, analyze, and summarize product reviews, delivering both high-level insights and detailed feedback on specific aspects of the product. ## implementation details and datasets used the experiment is implemented using python programming language @xcite . the text processing tasks are carried out using the natural language toolkit (nltk) library @xcite , spacy @xcite and hugging face transformers @xcite . the dataset used by us is a collection of reviews from amazon.com @xcite . we test our experiment on three distinct domains of data namely, electronics, cell phones and accessories and grocery and gourmet food. ## experimentation details and evaluation metrics the three domains considered for experimentation in our study are 1) electronics 2) cell phones and accessories and 3) grocery and gourmet food. we have considered 5,000 reviews in each of these domains. the quality of the generated aspect-based summaries from the given set of product-specific reviews using the different llms was evaluated using the traditional and gpt-4 criteria evaluation metrics. under the traditional approach of evaluation on different llms, we applied the likes of meteor @xcite , bertscore f1 @xcite , rouge 1, rouge 2 and rouge l @xcite metrics. and, for gpt-4 criteriabased evaluation @xcite ; @xcite , the following parameters @xcite ; @xcite , such as relevance, coverage, impurity, rating and goodness, are considered. the results of each approach are presented in the next section under results and discussions. ## results and discussions through the experimentation process, our results are tabulated in terms of the number of parameters used for language models, namely mediumlanguage models and large-language models. we present our results obtained using the traditional metrics and gpt-4 criteria metrics on mediumlanguage models and large-language models. as shown in in ## conclusion in this paper, we explored the use of various llms such as llama3, gpt-4o, gemma2, mistral, mixtral and qwen2 models for the summarization tasks. our work involved experimenting with the variations of the above-mentioned models on the publicly available domain-specific amazon reviews dataset. the experiment was tested on three different domains, and the results show that each of the llms was able to generate an aspect-based summary on the given set of domain-specific reviews. we postulated the algorithm used in our study accurately identifies and extracts relevant product aspects and generates a concise summary using the various models. we analyzed our experimental results produced by each of these llms with the summary evaluation metrics such as rouge, meteor, bertscore f1 and gpt-4o metrics and evaluated the quality of the generated aspect-based summary. with this work, we were able to highfigure 4: comparison between results of gpt-4 criteria evaluation metrics on large-language models light the strengths and limitations of each of these llms used. our findings show that the gpt-4o model performs well on traditional as well as gpt-4 criteria evaluation metrics on medium as well as large-language models. however, the variations of gemma2, mixtral and qwen2 models possess a lot of potential and fair well for some of the metrics on medium as well as large-language models. the introduction of qwen2audio and qwen2_vl models opens the door for further exploration in multimodal summarization. so, as part of our future work, we would like to fine-tune our models to improve the performance and efficiency of our summarization task. also, we would like to analyze the performance of small-language models with medium and large-language models in terms of computation time and efficiency.