Weakly supervised hierarchical multi-task classification of customer questions (2023)

Abstract: Identifying granular and actionable topics from customer questions (CQ) posted on e-commerce websites helps surface the missing information expected by customers on the product detail page (DP), provide insights to brands and sellers on what critical product information that the customers are looking before making a purchase decision and helps enrich the catalog quality to improve the overall customer experience (CX). We propose a weakly supervised Hierarchical Multi-task Classification Framework (HMCF) to identify topics from customer questions at various granularities. Complexity lies in creating a list of granular topics (taxonomy) for 1000s of product categories and building a scalable classification system. To this end, we introduce a clustering based Taxonomy Creation and Data Labeling (TCDL) module for creating taxonomy and labelled data with minimal supervision. Using TCDL module, taxonomy and labelled data creation task reduces to 2 hours as compared to 2 weeks of manual efforts by a subject matter expert. For classification, we propose a two level HMCF that performs multi-class classification to identify coarse level-1 topic and leverages NLI based label-aware approach to identify granular level-2 topic. We showcase that HMCF (based on BERT and NLI) a) achieves absolute improvement of 13% in Top-1 accuracy over single-task non-hierarchical baselines b) learns a generic domain invariant function that can adapt to constantly evolving taxonomy (open label set) without need of re-training. c) reduces model deployment efforts significantly since it needs only one model that caters to 1000s of product categories.

## introduction having correct, complete, and consistent information on the detail page is very important to ensure a world-class customer experience. e-commerce customers often refer to the "customer questions and answers" (cqa) section to seek the information that they deem important before buying. world wide, a leading e-commerce website customers ask and questions in the order of millions foot_0 per week before buying. customers refer to the cqa section primarily to find information that is a) not present on product page or b) is inconsistent across various sections of the product page. therefore, identifying information gaps on product page can help catalog owners improve the quality of catalog, create new attributes to enrich product page. it also helps product owners design better products, understand customer preferences, and improve the overall customer experience. in this paper, we aim to build a scalable solution that extracts topics from customer questions (cq). note: in this problem, we want to extract topics customers are interested in and not the answers to their questions since the objective is to identify information gap on product page using cqs. further, we wish to identify granular and "actionable" topics. an actionable topic clearly conveys the intent behind the question. please refer to identifying topics for cq poses several practical challenges, especially, at the scale of an ecommerce giant where products are spread across thousands of categories. ## related work the lack of availability of training data and a predefined taxonomy is a big challenge in industry. lda @xcite ) can be helpful in mining topics from corpus of text and creating taxonomy. taxogen ( @xcite ) proposed an unsupervised method to derive taxonomy from a large corpus of text. however, these approaches model the text corpus as bag of words (bow) and do not take into account the context of the keywords. there are many models available that model multiclass text classification as a hierarchical classification problem @xcite . however, they do not offer flexibility in adding or removing labels without adding extra parameters and retraining on entire dataset. also, they use deep learning based models which require large amounts of labelled data for training. recently, few-shot learning based approaches have gained popularity in the nlp domain, particularly to address the challenge of large scale label data availability ( @xcite , @xcite , @xcite ). however, these approaches do not take label information into account. recently, bert @xcite ) has shown state-of-the-art performance on many nlp tasks. we use it for both level-1 and level-2 tasks. in this section, we describe the taxonomy and label creation approach in detail (refer to figure 1 ). ## results in this section, we discuss the results of our experiments in detail. we use the top-1 accuracy metric in all the experiments. below is a summary of key observations made from the experiment results.