A Semi-supervised Multi-task Learning Approach to Classify Customer Contact Intents (2021)

Abstract: In the area of customer support, understanding customers’ intents is a crucial step. Machine learning plays a vital role in this type of intent classification. In reality, it is typical to collect confirmation from customer support representatives (CSRs) regarding the intent prediction, though it can unnecessarily incur prohibitive cost to ask CSRs to assign existing or new intents to the mis-classified cases. Apart from the confirmed cases with and without intent labels, there can be a number of cases with no human curation. This data composition (Positives + Unlabeled + multiclass Negatives) creates unique challenges for model development. In response to that, we propose a semi-supervised multi-task learning paradigm. In this manuscript, we share our experience in building text-based intent classification models for a customer support service on an E-commerce website. We improve the performance significantly by evolving the model from multiclass classification to semi-supervised multi-task learning by leveraging the negative cases, domain- and task-adaptively pretrained ALBERT on customer contact texts, and a number of un-curated data with no labels. In the evaluation, the final model boosts the average AUC ROC by almost 20 points compared to the baseline finetuned multiclass classification ALBERT model.

## introduction as machine learning makes rapid advances in the area of natural language processing (nlp), it is becoming more common to aid customer support representatives (csrs) with nlp models. this not only ensures timely and consistent replies to customers, but also reduces operational costs for organizations. we can see successful use cases from organizations such as alibaba @xcite , uber @xcite , square @xcite , at&t @xcite , ibm @xcite , los alamos national laboratory (delucia and moore, 2020), and us navy @xcite . in general, identifying the intents of the coming contacts is the first step in customer support. therefore, accurate intent classification is crucial. intent classification is a broad topic mostly falling under the umbrella of nlp. in this manuscript, we limit our discussion to intent classification in the area of customer support. in the past two decades, researchers have been trying to improve the efficiency of customer support by detecting customer intents with machine learning approaches @xcite @xcite @xcite @xcite @xcite . we can loosely categorize these approaches into text classification @xcite @xcite @xcite , question-answer (qa) system @xcite and automatic speech recognition (asr) @xcite @xcite . in this manuscript, we focus on using text classification methods to classify intents for customer support. to deal with unstructured text data, researchers use handcrafted features @xcite , bag-of-words type of features @xcite , features from topic modeling (delucia and moore, 2020) and vectorization type of features, such as word2vec @xcite and doc2vec @xcite . by consuming these features, classifiers determine the intent of a case and the case can be routed to specialists @xcite @xcite and/or a reply template from the "answer bank" can be provided @xcite @xcite . a general intelligent customer support loop can be seen in figure 1 . ## background the e-commerce website of interest receives many support requests from customers in each second. there is a team of csrs to actively address the requests via phone, online chat, and email channels. identifying appropriate requests and grouping them into categories is not a trivial task. while a deep discussion of the taxonomy building process is out of the scope of this manuscript, it is sufficient to know that we have a taxonomy system that is similar to those described in @xcite , where customized reply templates are pre-compiled for each customer contact intent. this study elaborates our journey building machine learning models to classify the intents. ## methodology since the inception of bert @xcite , an abundance of research in the area of nlp has demonstrated it to be an effective approach to transfer knowledge from pretrained language models to downstream tasks @xcite @xcite @xcite . following bert's architecture, there is a stream of research that achieve comparable or better performance, to name a few @xcite @xcite @xcite . among these bert variants, albert aims to strike a balance between model performance and model size @xcite . therefore, we use albert-base-v2 as the backbone encoder and perform further pretraining and finetuning. the implementation is based on transformers from huggingface @xcite . ## discussion apart from processing the dismissed recommendations with this multi-task setting, there is another heuristic approach that is commonly adopted under this circumstance. we can group all the dismissed recommendations into an extra bucket others @xcite . the advantage of this approach is that we can pose the problem as a straightforward multiclass classification. the disadvantage is that the dismissed recommendations can either be mis-classified and belong to other existing classes, or belong to unknown classes that might be included in the future taxonomy. in the former scenario, the dismissed recommendations create noise for their true class and the others class; in the latter scenario, the dismissed recommendations can seemingly improve performance for current taxonomy, while they can pollute the future training when the unknown classes are launched in the updated taxonomy. in both scenarios, grouping the dismissed recommendations into others can negatively impact the training. in terms of computational cost, both adaptive pretraining and semi-supervision consume a considerable amount of power, since the former is typically trained on the mlm task through a large corpus and the latter is a iterative finetuning and inferencing process where the data for inferencing are often in large volume. in the meantime, the mt strategy is a cost-effective way to improve model performance by considering negative samples. by examining for the sake of easy implementation of the endto-end multi-task training, we only feed the training data related to one task in each batch. in this way, we can keep the loss function for each task separate. it is possible that including data for various tasks in each batch can bring benefits to training. this assumption can be explored in future studies. this study is only concerned with corpus in en-glish. similar modeling strategies can be followed for other high-resource languages which we have ample training data. however, as in the customer service departments of most global organizations, it is common to receive customer contacts in various low-resource languages, in which case the training data is scarce. recent advances in cross-lingual language models, such as mbert @xcite , xlm @xcite , unicoder @xcite and filter @xcite , can shed light on this situation and we plan to investigate it in the future. in the area of customer support, both @xcite and @xcite propose neural networks that combine unstructured text features from customers' messages and structured features describing customers' interaction with the platforms. they empirically demonstrated benefits of including the latter feature group. the next step for our study is to evaluate the influence of the customer-website interaction features, when combining with advanced language models. for the model candidates with multi-task strategy in this manuscript, we train all tasks jointly with an end-to-end multi-task deep learning approach, as described in the left plot of figure 2 . we want to point out the isolating effect of the endto-end training approach. in one experiment, we trained the tasks independently, i.e. we first trained the multiclass classification task with the off-theshelf albert, and then, for the binary tasks, we trained n logistic regression binary classifiers with the classification vector from the multiclass classification task. we still achieved 12.2 and 8.2 points above the baseline in terms of average auc roc and sample-weighted average auc roc. on one hand, this shows that even training simpler models independently can still bring performance lifts, thus emphasizing the powerful signal brought by the negative cases; on the other hand, if compared to the albert + mt model in as in most machine learning applications, the actual model performance is determined by the choice of the operational point for each intent class and the operational point is determined from the precision-recall (pr) curve. for the sake of brevity, we ignore the pr plots because, for each class, the pr curve of the baseline albert model is well under the envelop of the pr curve of the ss mt d/tapt albert model. this is expected due to the large boost presented in figure 4 . we note that the auc roc can be a decent indication of auc pr when the data is not so skewed @xcite . therefore, the ss mt d/tapt albert indeed outperforms the baseline for every choice of operational point. ## conclusion in this manuscript, we demonstrated and discussed the model performance improvement brought by multi-task learning, adaptive pretraining for al-bert, and semi-supervised learning in the application of customer support on an e-commerce website. we observe ∼20 points performance increase in average auc roc when comparing the final model to the baseline multiclass classification model. this paradigm can be particularly helpful when there is a feedback system collecting confirmation from labelers. future studies can extend this paradigm to more complex situations, such as when the intent taxonomy is deeply hierarchical or considering more feedback information than simple "yes" or "no".