Predicting Customer Satisfaction with Soft Labels for Ordinal Classification (2023)

Abstract: In a typical call center, only up to 8% of callersleave a Customer Satisfaction (CSAT) surveyresponse at the end of the call, and these tend tobe customers with strongly positive or negativeexperiences. To manage this data sparsity andresponse bias, we outline a predictive CSATdeep learning algorithm that infers CSAT onthe 1-5 scale on inbound calls to the call centerwith minimal latency. The key metric to maximize is the precision for CSAT = 1 (lowestCSAT). We maximize this metric in two ways. First, reframing the problemas a binary class, rather than five-class problem during model fine-tuning, and then mapping binary outcomes back to five classes usingtemperature-scaled model probabilities. Second, using soft labels to represent the classes. Theresult is a production model able to support keycustomer workflows with high accuracy overmillions of calls a month.

## introduction to address these issues, we have developed and present here an algorithm that infers csat scores on call center calls with high accuracy and low latency. at the moment of writing, our predictive csat feature is fully deployed at scale and has rated over 50 million calls. ## related work there are few research studies on predicting customer satisfaction (csat) scores on contact center conversations using transcripts generated by an automatic speech recognition (asr) model. in @xcite , they developed a system that not only utilizes call transcripts transcribed by an asr model but also other non-textual data such as call duration, queue, in-queue waiting times, utterance level sentiment scores, and various customer data. overall, there are 5,501 features in the training dataset. the author's model is trained to predict a metric called representative satisfaction index (rsi) which is the average of four different survey scores. in the end, their framework involves two models, namely a rank scoring and an isotonic regression model. in a more recent study, @xcite used the net promoter score (nps) to predict customer satisfaction on chat conversations. a promoter score can be defined as a rating that customers give to indicate how likely they are to promote a company. out of a scale of 0 to 10, customers with ratings of 9 or 10 are considered promoters whilst those with ratings of 0 to 6 are considered detractors. nps is calculated as the difference between promoters and detractors and companies want this metric to be positive and as high as possible. they compared macro f1 scores across different classification methods and their best method yielded a macro f1 score of 53.8%, which they noted that is a rather limited performance. other studies that looked at predicting csat on contact center conversations proposed using information extracted from raw audio signals such as acoustic, emotions, and prosodic features. @xcite @xcite contact center managers are usually interested in picking out calls with a low csat score for either coaching purposes or for identifying opportunities to take meaningful interventions in a timely manner to improve customer experience. hence, it's important to identify these calls with a relatively high degree of precision. label smoothing is a regularization technique introduced by @xcite that has been successfully used to improve accuracy of the inception architecture on the im-agenet dataset. in @xcite is noted that label smoothing has been adopted in training procedures of other state-of-the-art image classification models @xcite @xcite . in another domain such as speech recognition, chorowski and jaitly, 2016 used label smoothing to reduce word error rate on the wsj dataset. additionally in machine translation, @xcite was able to slightly improve the bleu score 3 system overview ## limitations