AI Coach Assist: An Automated Approach for Call Recommendation in Contact Centers for Agent Coaching (2023)

Abstract: In recent years, the utilization of Artificial Intelligence (AI) in the contact center industry is on the rise. One area where AI can have a significant impact is in the coaching of contact center agents. By analyzing call transcripts, AI can quickly determine which calls are most relevant for coaching purposes, and provide relevant feedback and insights to the contact center manager or supervisor. In this paper, we present “AI Coach Assis”, which leverages the pre-trained transformer-based language models to determine whether a given call is coachable or not based on the quality assurance (QA) queries/questions asked by the contact center managers or supervisors. The system was trained and evaluated on a large dataset collected from real-world contact centers and provides an efficient and effective way to determine which calls are most relevant for coaching purposes. Extensive experimental evaluation demonstrates the potential of AI Coach Assist to improve the coaching process, resulting in enhancing the performance of contact center agents.

## introduction ai has the potential to revolutionize many industries, including the contact center industry. with the growing demand for high-quality customer service, contact centers are constantly seeking ways to improve their processes and enhance their agents' performance. one way to achieve this goal is by providing effective coaching and feedback to agents, which can help them identify areas of improvement and develop the necessary skills to provide exceptional customer service. as a common practice, contact center managers or supervisors manually select call recordings to listen in, and grade agents' performance using a rubric that contains questions such as "did the agent greet the customer by name" or "did the agent properly resolve the customer issue" to score the call in order to verify if the agent is following the company's preferred protocol. the grades given by the managers along with their comments are then shared with the agents to improve their performance. however, with the large volume of calls that contact centers receive, it is very challenging for managers or supervisors to determine which calls are most important for agent coaching. thus, the traditional approaches to randomly select calls for agent coaching has the following limitations: this is where nlp could be useful. by analyzing call transcripts using nlp models, it could be possible to recommend calls to the contact center managers/supervisors that are most relevant for coaching purposes. this will lead to an improved coaching experience by prioritizing the calls for analysis that are more likely to contain coachable moments, resulting in saving time for the contact center managers as well as improving agent performance, ultimately leading to better customer satisfaction. for the purpose of improving realworld contact centers, we present the ai coach assist system to assist contact center managers or supervisors by suggesting calls that could be more useful for agent coaching. in this paper, we explore the concept of our proposed ai coach assist system, which leverages the advantage of fine-tuning a pre-trained transformerbased language model @xcite @xcite @xcite . moreover, we provide a detailed overview of its development process (implementation and preparation of a balanced dataset to avoid biases), as well as our experimental findings. in addition, we demonstrate how it could be productionized in real-world contact centers to assist managers/supervisors. note that our model does not automate the scoring of employee performance or replace human review. instead, our model is intended to help contact center supervisors by recommending calls for coaching their employees instead of the traditional random sampling of calls. ## related work the significant performance gain achieved via leveraging transformer-based language models @xcite @xcite in a wide range of nlp tasks in recent years has also led to the use of transformer-based models in the contact center industry @xcite . the successful deployment of these models in industries has helped many organizations to enhance their processes, resulting in improved customer satisfaction. in recent years, several studies @xcite have explored the potential of ai-powered call analysis (e.g., entity recognition, sentiment analysis, etc.), along with providing realtime assistance to contact center agents. in addition to these studies, several commercial solutions have been developed that offer aipowered call analysis and ai assistance for agents in contact centers. some of these solutions also offer real-time feedback to agents during calls 123 , allowing them to adjust their behavior and improve their performance in real-time. however, to the best of our knowledge, there is no prior commercial application that assists contact center managers by suggesting calls that could be the most useful to coach agents. one potential approach for this purpose could be the use of automatic call recommendation, where calls are analyzed using nlp techniques and suggested to the contact center managers based on various factors, such as agents' behavior, issue resolution, customer satisfaction, sales success, etc. these suggested calls can then be analyzed by the managers for coaching purposes to provide relevant feedback to agents. in this regard, we propose ai coach assist, a system that leverages the transformer architecture to effectively analyze the full call transcripts in contact centers and recommends contact center managers with calls that are more likely to contain coachable moments for a given query. in the following section, we describe how we construct a dataset, which we denote as qa scorecard, to train and evaluate our proposed ai coach assist system. ## the qa scorecard dataset we collected our data from real-world contact centers. the dataset consists of customer-agent call conversation transcripts generated using automatic speech recognition (asr) systems, along with annotations indicating whether a call is coachable or not. the process of annotating the dataset was carefully designed and implemented, as the annotations were performed by real-world contact center managers and supervisors who analyzed the whole conversation/transcript. in this way, we ensure the high quality of the dataset. the data annotation works as follows, the managers/supervisors assign a score to the call based on the performance of the agent for a particular question. we consider a call as coachable for a particular question if the call achieves less than 50% scores, otherwise, we consider the call for that particular question as not coachable. the dataset was collected over a period of one year and includes a diverse range of call types from different industries, with a variety of customer interactions, reflecting the real-world complexities of the contact center industry. the resulting dataset consists of a large number of call transcripts and annotations, providing a robust representation of real-world customeragent interactions. note that a total of 58 questions are curated, which are distributed among training, validation, and test sets. while constructing the training, validation, and test splits, we observe that the class distribution (whether coachable or not coachable) for many question-transcript pairs was imbalanced. thus, to ensure an unbiased dataset (as well as to avoid model overfitting), for each question, we ensured that the ratio between coachable and not coachable classes (or vice-versa) to be at most 1:2. in ## our proposed approach we treat the ai coach assist model as a text classification model that combines the query/question given by the contact center manager or supervisor with the call transcript to predict whether a given call is coachable or not. due to the recent success of fine-tuning pre-trained transformer models for text classification @xcite @xcite , we also leverage the pretrained language models based on the transformer architecture for this task. as we are doing text classification instead of generation, we give input data to the pre-trained language model as follows (see figure 1 ): at first, we create a text sequence by concatenating the question and the call transcript. then, this concatenated text sequence is given as input to the language model to learn the contextual relationship between the sentences. the pre-trained transformer language model is fine-tuned to output a probability score for each input sequence, indicating the likelihood that the call is coachable or not, for the given question. whether a question-transcript pair is coachable or not coachable is determined based on the probability score of the class having the higher score. since our objective is to build the ai coach assist system for real-world contact centers, we consider the following two cases while selecting the pre-trained language models: (i) utilize a model to ensure high efficiency: we choose distilbert @xcite for this scenario. distilbert is a distilled version of bert @xcite , designed to be smaller and faster while retaining a similar level of performance. despite its smaller size, distilbert has been shown to perform similarly to bert on many nlp tasks, making it a suitable alternative for many nlp applications. this makes it a popular choice for real-world scenarios where computational resources are limited but the preference is to deploy a fast and optimize model in production. on long dialog conversations, having more similarities with our customer-agent conversation dataset. though in comparison to distilbert, the di-alogled model may require higher computational resources for production deployment, it fulfills our criteria of using a model that may provide higher accuracy for being pre-trained on long dialog conversations, mimicking the customer-agent conversations in the real world. in addition, dialogled can also process long text sequences, contrary to the 512-token limit of most transformer-based models @xcite @xcite . this makes dialogled a suitable choice to build the ai coach assist system since the average length of the transcripts in our qa scorecard dataset is longer than 512 words. ## experiments in this section, we first present the experimental settings and the implementation details of our proposed model. then we discuss our experimental findings in detail. ## usage in real world contact centers in this section, we discuss how the ai coach assist system can be used in real-world contact centers. since determining the calls that are coachable is not required in real-time, rather they are required after the call is over, the inference speed of the model may not be an issue in this regard. moreover, for contact centers where the computing resource is not a problem, our dialogled-based model could be used, as it achieves better accuracy than its distil-bert counterpart. since the size of the trained di-alogled model is 648 mb, the distilbert model which takes only 263 mb could be used in scenarios where the computing resource is limited. we also prototype our proposed system for usage in a real contact center. since directly predicting a score to a call might impact the evaluation of agent performance in contact centers, as such metrics could be used by managers for performance evaluation of agents, in our prototype we rather recommend a list of calls to the managers that are highly likely to contain coachable moments for a particular question type. thus, instead of using those calls for a direct performance evaluation of agents, the managers still require to listen to the conversation or read the asr-generated transcript. more particularly, using our proposed ai coach assist, we help the managers with a list of calls that they may use to manually grade agent performance, contrary to the existing methods of random call selection. in this way, the proposed prototype of ai coach assist may not cause any ethical concerns. ## conclusion in this paper, we presented ai coach assist, a transformer-based pairwise sentence classification model that combines the query/question given by the contact center manager or supervisor with the call transcript to determine which calls are most relevant for coaching purposes. the evaluation results demonstrate the potential of ai coach assist to transform the way contact centers coach their agents, providing an efficient and effective method that recommends calls that are the most relevant for coaching purposes. this will help to improve the coaching process and enhance the performance of contact center agents, leading to better customer satisfaction. note that our model is intended to help contact center supervisors to be more effective in coaching their employees by improving over the random sampling of calls. the model does not automate the evaluation of employee performance by replacing human review. in the future, we will study how to improve the performance on the question types where the model performs poorly. we will also study how to utilize other question-answering models @xcite or leverage generative large language models @xcite that can point out the reason for a call being coachable and not coachable.