Leveraging Seq2seq Language Generation for Multi-level Product Issue Identification (2022)

Abstract: In a leading e-commerce business, we receive hundreds of millions of customer feedback from different text communication channels such as product reviews. The feedback can contain rich information regarding customers’ dissatisfaction in the quality of goods and services. To harness such information to better serve customers, in this paper, we created a machine learning approach to automatically identify product issues and uncover root causes from the customer feedback text. We identify issues at two levels: coarse grained (L-Coarse) and fine grained (L-Granular). We formulate this multi-level product issue identification problem as a seq2seq language generation problem. Specifically, we utilize transformer-based seq2seq models due to their versatility and strong transfer-learning capability. We demonstrate that our approach is label efficient and outperforms the traditional approach such as multi-class multi-label classification formulation. Based on human evaluation, our fine-tuned model achieves 82.1% and 95.4% human-level performance for L-Coarse and L-Granular issue identification, respectively. Furthermore, our experiments illustrate that the model can generalize to identify unseen L-Granular issues.

## introduction customer feedback plays a crucial role in continuously improving service quality for e-commerce companies. one important piece of information in customer feedback are the product issues that customers encounter during their order experience, such as product-quality defects and undesired product features. although negative product experience occurs rarely in mature e-commerce stores, identifying the product issues presented in customer feedback significantly helps sellers understand customers' concerns and facilitates them to further improve customer order experience. as the volume of customer feedback grows rapidly, an au-tomatic and intelligent issue identification system is needed to support the fast-growing e-commerce business. in this paper, we introduce a machine learning solution for multi-level issue discovery by taking advantage of advanced deep language modeling techniques. we show that our approach not only accurately identifies product issues from customer feedback, but also meets the requirements for our use case, which, we believe, is also common to other e-commerce store business. identifying product issues from customer feedback has its own unique characteristics as compared to common nlp tasks such as text classification, document summarization, entity extraction, and sentiment analysis. first, we target to extract diverse and dynamic product issues, instead of categorizing them into a fixed set of labels. this is mainly due to three varying factors: the product itself, the customer and the context. different categories of products naturally have different kinds of issues. different customers may find different flaws of the same product, depending on their personal taste and preferences; the way they describe issues could vary greatly from one customer to another. in addition, issues are dynamic as different issues emerge and disappear from time to time. for example, a bad local weather can lead to a surge in shipment damage and delivery failure complaints in a short period of time. second, to effectively improve customer experience, multi-level issues with different granularity are needed. higher-level, concise, and consistent issues can be shared with sellers to help them identify trends and hotspots for the flaws in their products; more detailed lower-level issues are needed by business investigators to study root causes of product defects and design treatment accordingly. the third requirement for product issue identification is about supervised information extraction. customer feedback often contains contents that are unrelated to product issue. furthermore, multi-ple product issues can be tangled or be embedded within one phrase or sentence. therefore, uncontrolled excerpts from customer feedback can be confusing and overwhelming to downstream users. so our system should focus on product issues and disentangle them to best serve downstream users. traditional language processing approaches can not meet all the three requirements discussed above. to start with, traditional classification methods can only produce results from a pre-defined, limited set of labels, and therefore cannot satisfy the first requirement. unsupervised methods such as topic modeling and clustering, on the other hand, are able to identify diverse patterns varying from a dataset to another and discover new trends. however, it is hard to steer the algorithms to exercise the control over the patterns or type of information they extract from the data. based on our experience, it is challenging to generate meaningful and highly coherent 'topics' using topic modeling or clustering algorithm based on text embedding techniques. finally, it is not straightforward to adapt these methods for multi-level issue identification unless we develop multiple models to handle each level individually. for operation efficiency, we prefer to have a single model to complete the whole task so as to minimize model deployment and maintenance cost. in this paper, we propose an innovative solution by formulating our problem as seq2seq language generation tasks and leveraging deep learning models with multi-task capability to meet all our requirements. in particular, we choose transformerbased seq2seq models as backbone and fine tune them on our data. a single model is trained to generate both the low-level and high-level product issues. even though the model is trained in a supervised manner with human-annotated data to have better content generation control, as a large-scale language model, the model shows generalization power to discover fine-grained issues from customer feedback that were not seen during training time. to the best of our knowledge, so far it is the most flexible and effective approach for extracting multi-level product issues from e-commerce customer feedback. ## related work there are various approaches to extract key topics and identify issues from customer feedback. prior work in this area can be categorized into four ma-jor groups: (1) document classification, (2) topic modeling, (3) clustering, and (4) text summarization. document classification is one of the most well studied nlp tasks, and many deep learning methods @xcite have been introduced and excelled at it in recent years. @xcite developed a convolutional neural network to extract reason codes from customer complaints. more recently, @xcite leveraged bert-based model to classify different types of fraud elements from internet fraud complaints. instead of assigning labels to the entire texts, they map the labels to each paragraph. these methods are not very applicable in the current context since they require a fixed and pre-defined taxonomy. for topic modeling, latent dirichlet allocation (lda) @xcite is a widely used statistical method for analyzing information in customer reviews and feedback @xcite @xcite . @xcite added pre-existing constraints to lda in order to improve product feature extraction from customer reviews. @xcite proposed a twofold-lda model to produce topics focused on desired aspects. @xcite proposed autoencoded variational inference for topic model (avitm) which yielded much more interpretable topics than lda. while most traditional topic modeling methods use simple document representations such as the bag-of-words, an alternative approach is to cluster similar documents using document embeddings followed by extracting common topics within each cluster. @xcite used glove embeddings and k-means clustering for analyzing different aspects in electronics and restaurants reviews. grootendorst (2020) used bert embeddings and dbscan to create interpretable topics. although topic modeling and clustering are not constrained to a fixed label set, the results can be inconsistent and incoherent due to unsupervised topic extraction. one can also pose this problem as a text summarization problem, where a model can generate a summary to present the key information from the input text. liu et al. (2021b) 3 proposed approach ## results in this section, we report the results and analysis from our study. to our knowledge there are no publicly-available datasets in e-commerce or performance benchmarks for the problem of multilevel product issue identification. thus, we conduct the study using our private customer-feedback dataset. due to confidentiality, we can not report absolute model performance numbers but only relative ones compared to the baseline models. ## conclusion in conclusion, we analyzed the challenges in identifying diverse and multi-level product issues from customer feedback. to overcome these challenges, we creatively formulated our problem as a seq2seq modeling problem and leveraged text-to-text transfer learning framework. we utilized multi-tasking to generate product issues at multiple levels using a single model, which minimizes operational cost. results show that our model performs closely to human on issue identification tasks. we also observed that our approach is label efficient and our model generalizes well to identify unseen l-granular is-sues. our next step is to explore ways to improve our model's performance and generalizability on l-coarse issue prediction. rate of 5e-5. we train this setup over 25 epochs on a p3.16xlarge ec2 instance with distributed model and data parallelism to fine-tune the model. during inference we use beam search @xcite to generate the target text sequence with a beam width of 2 and length penalty α = 2.5 @xcite .