I nsight N et : Structured Insight Mining from Customer Feedback (2023)

Abstract: We propose InsightNet, a novel approach for the automated extraction of structured insights from customer reviews. Our end-to-end machine learning framework is designed to overcome the limitations of current solutions, including the absence of structure for identified topics, non-standard aspect names, and lack of abundant training data. The proposed solution builds a semi-supervised multi-level taxonomy from raw reviews, a semantic similarity heuristic approach to generate labelled data and employs a multi-task insight extraction architecture by fine-tuning an LLM. InsightNet identifies granular actionable topics with customer sentiments and verbatim for each topic. Evaluations on real-world customer review data show that InsightNet performs better than existing solutions in terms of structure, hierarchy and completeness. We empirically demonstrate that InsightNet outperforms the current state-of-the-art methods in multi-label topic classification, achieving an F1 score of 0.85, which is an improvement of 11% F1-score over the previous best results. Additionally, InsightNet generalises well for unseen aspects and suggests new topics to be added to the taxonomy.

## introduction customer reviews provide rich insights for various stakeholders, such as businesses, brands, and customers. they can inform product development, enhance customer experience, track reputation, and guide purchase decisions. however, customer reviews pose several challenges for analysis, such as subjectivity, variation, noise, domain-specificity, volume, and dynamism. existing solutions for extracting structured insights from reviews, such as topic classification @xcite , polarity identification @xcite , and verbatim extraction @xcite , suffer from several drawbacks that limit their effectiveness and applicability. these drawbacks include: (1) low accuracy and reliability in generating and extracting insights from reviews, (2) lack of coherence and clarity in the output, which makes it hard to act upon, (3) high dependency on large amounts of annotated data, which are scarce and expensive to obtain, (4) task-specificity, (5) inability to handle multiple tasks simultaneously, (5) skewed data distribution towards a few dominant aspects, which biases the models' performance (81% of reviews are covered by just 12% of topics, see appendix c.1 for detailed analysis), (6) reliance on predefined aspects and limitations to discover new topics. in this paper, we present three key modules to address the challenges of the existing approaches as follows: (1) autotaxonomy: a method to generate a hierarchical taxonomy of aspects with minimal supervision (section 4.2). this helps to organise the output in a structured and hierarchical form (2) segmentnet: an unsupervised data creation technique using semantic similarity based heuristics to produce labelled data (section 4.3) that contains topic, polarity and verbatim for each review. here, verbatim is the exact segment of the review that describes the topic identified. (3) insightnet: a generative model for insights extraction. we model aspect identification as a multi-task hierarchical classification problem and then leverage the generative model (section 4.1) to classify topic (granular aspect), identify sentiment, extract verbatim and also discover new topics that are not in the current taxonomy. we use t5-base @xcite as a pre-trained large language model (llm) and finetune it with the data obtained from segmentnet. thus, we do not require any manually annotated data to train insightnet. ment analysis, text summarisation, topic modeling, generative modeling. rana and cheah (2015) ; @xcite proposed a rule-based approach to extract insights from reviews. however, these approaches require huge manual efforts and domain expertise to discover patterns, update them frequently as new products launch and create rules. @xcite ; @xcite propose aspect based sentiment analysis methods which first extract aspects and then rate reviews on each aspect. however, aspects obtained using these methods are not granular enough for actionability. @xcite ; @xcite ; @xcite proposed unsupervised approaches for aspect and sentiment analysis from reviews, but these methods suffer from two main limitations : a) redundancy of clusters and b) low interpretability, as the clusters produced are not actionable, structured or intuitive. recently, generative approaches @xcite demonstrated promising performance on wide range of natural language processing (nlp) tasks. @xcite used a seq-to-seq model to generate product defects and issues from customer reviews, but they lack structure. ## problem statement given a customer review c, we aim to extract a set of all the relevant and actionable insights i 1 , i 2 , . . . , i k , where each insight i i is composed of a granular topic t i , a corresponding polarity p i , and a set of verbatims v i associated with it. ## methodology in this section, we present our generative approach, insightnet, for mining insights (topics, polarities, verbatims) from raw reviews (obtanied from amazon us marketplace foot_0 ). next, we describe how we construct a multi-level hierarchical taxonomy from the reviews to help organize topics in a meaningful way. then, we introduce how the labelled data is created using segmentnet. later, we explain how we apply post-processing techniques to eliminate redundant topics and surface new topics that are not covered by the taxonomy. finally, we discuss the experiments that led us to the design of insightnet. ## conclusion we have presented insightnet, a novel multi-task model that extracts granular insights from customer reviews. insightnet jointly performs multi-topic identification, sentiment classification, and verbatim extraction for each review, generates new topics beyond existing taxonomy, and enriches taxonomy with consistent and exhaustive topics. insightnet surpasses the state-of-the-art methods by 11% f1score on overall performance metrics, and achieves 85% f1-score on topic classification. furthermore, insightnet is scalable and can handle various tasks with a structured and hierarchical output. hugo touvron, louis martin, kevin stone, et al. 2023. llama 2: open foundation and fine-tuned chat models. jian-guo zhang, kazuma hashimoto, wenhao liu, chien-sheng wu, yao wan, philip s. yu, richard socher, and caiming xiong. 2020. discriminative nearest neighbor few-shot intent detection by transferring natural language inference. lili zheng. 2021. the classification of online consumer reviews: a systematic literature review and integrative framework. journal of business research, 135:226-251.