Probing the Depths of Language Models’ Contact-Center Knowledge for Quality Assurance (2024)

Abstract: Recent advancements in large Language Models (LMs) have significantly enhanced their capabilities across various domains, including natural language understanding and generation. In this paper, we investigate the application of LMs to the specialized task of contact-center Quality Assurance (QA), which involves evaluating conversations between human agents and customers. This task requires both sophisticated linguistic understanding and deep domain knowledge. We conduct a comprehensive assessment of eight LMs, revealing that larger models, such as Claude-3.5-Sonnet, exhibit superior performance in comprehending contact-center conversations. We introduce methodologies to transfer this domain-specific knowledge to smaller models by leveraging evaluation plans generated by more knowledgeable models, with optional human-in-the-loop refinement to enhance the capabilities of smaller models. Notably, our experimental results demonstrate an improvement of up to 18.95% in Macro F1 on an in-house QA dataset. Our findings emphasize the importance of evaluation plans in guiding reasoning and highlight the potential of AI-assisted tools to advance objective, consistent, and scalable agent evaluation processes in contact centers.

## introduction the convergence of contact-center management and artificial intelligence represents a frontier rich with potential for revolutionizing customer service quality and operational efficiency. contact-centers, serving as the primary interface between organizations and their customers, are increasingly seeking sophisticated methods to evaluate and enhance agent performance so as to improve their customer satisfaction @xcite . concurrently, the † equal contribution as first authors. ‡ equal contribution as second authors. did the agent demonstrate active listening ? did the agent address any objections raised by the customer? did the agent ask probing questions to discover the customer's needs? did the agent properly acknowledge customer inquiry? field of natural language processing has witnessed unprecedented advancements with the emergence of large language models (lms) such as @xcite , gemini @xcite , claude @xcite , and their successors. these models have demonstrated remarkable proficiency in understanding and generating human-like text across diverse domains, suggesting promising applications in a variety of natural language processing tasks, such as, machine translation @xcite , sentiment analysis @xcite ), text summarisation @xcite , reasoning @xcite , etc. however, evaluating contact-center agents using these lms presents unique challenges that extend beyond basic linguistic comprehension. effective assessment requires a deep understanding of industry best practices, domain knowledge, and the nuances of customer service communication. let us consider the question -"did the agent demonstrate active listening?" illustrated in figure 1 . this evaluation involves more than just analyzing text; it requires a comprehensive grasp of active listening principles in customer interactions. an accurate assessment must determine if the agent attentively listened without requiring repetition, understood the customer's issue, and asked appropriate followup questions to guide the interaction towards resolution. while some aspects of this evaluation can be explicitly derived from the question, others demand deeper domain understanding (refer appendix d). this includes recognizing the complexities of customer issues, appropriate troubleshooting steps, and the flow of effective customer service interactions. the multifaceted nature of this task highlights the need to integrate sophisticated nlp models with domain-specific expertise for comprehensive contact-center agent evaluations. while research has explored lms in various contact-center applications, their use in quality assurance remains understudied. @xcite examine in-domain fine-tuning for tasks like summarization and question-answering, but does not address holistic agent evaluation. in this paper, we aim to fill this gap with three key contributions: ## problem formulation contact-centers typically have a dedicated quality assurance (qa) team responsible for maintaining high service standards and ensuring customer satisfaction. this team systematically evaluates agent performance across various interactions, focusing on adherence to company policies, compliance requirements, agent behaviour and best practices. as a part of this process, qa analysts meticulously review the agent-customer conversations, identify key events, and evaluate the agent's performance against predefined criteria. maintaining consistency and accuracy of these evaluations poses a significant cognitive overload for qa analysts and is in turn a time-taking process, necessitating a nuanced approach to improve the efficiency and effectiveness of qa processes. the evaluation criteria used by qa teams are often framed as questions, which need to be answered based on the conversation and the effectiveness of the agent's interaction with the customer. these questions cover various aspects of the interaction, such as whether the agent actively listened to the customer, accurately identified and addressed the issue, adhered to the company's communication protocols, etc. (refer to examples of qa questions in figure 1 ). framing the qa evaluation in this manner naturally makes it a question-answering task. importantly, providing just the answer to these questions is often not sufficient. detailed reasoning must accompany each answer to validate the response and offer transparency. this reasoning explains why a particular answer was chosen, highlighting the relevant portions of the conversation that led to the conclusion (refer to figure 4 in appendix a.1 for a sample response to a qa question). this not only enhances the accuracy of the evaluation but also streamlines the process for qa professionals by offering clear justifications for each assessment, making their workflow more efficient and decisions more reliable. formally, we define the qa task as follows: given a conversation c between an agent and a customer, and an evaluation question q designed to assess a specific aspect of the agent's performance, the goal is to generate a detailed reasoning r and an appropriate answer a, such that r logically leads to a. this requires the extraction and synthesis of relevant information from the conversation c, demonstrating a deep understanding of both contact-center domain-knowledge and nuances of the interaction. providing r with a not only validates the response but also offers transparency and clarity, aiding qa analysts in their decision-making process. ## distilling domain-knowledge to small lms given that large lms demonstrate proficiency in contact-center domain-knowledge, we explore the feasibility of transferring this to smaller lms. specifically, we select phi-3-mini-128k-instruct (abdin et al., 2024) as our target model due to its superior performance among the small group. however, our approach is generic enough to be extended to any lm. ## prior work language models (lms) have shown considerable advancements in recent years, demonstrating their ability to generate fluent text across a wide range of inputs @xcite . these advancements have fueled significant interest in applying lms to domain-specific contexts, where fine-tuning general-purpose models with domainspecific data has led to notable performance improvements across various specialized fields such as legal, medical, and finance domains, highlighting their ability to adapt and perform complex tasks. notable examples include biogpt and med-palm in biomedical research @xcite , codet5 and codellama in coding @xcite , and bloomberg-gpt in finance @xcite . research into the knowledge embedded within lms has underscored their vast repository of general information, suitable for diverse applications @xcite . studies have also indicated that the ability of lms to store and effectively use this knowledge scales with their size, enabling them to handle increasingly complex tasks @xcite . nevertheless, the performance of these models in contactcenter environments, particularly in quality assurance (qa), remains relatively unexplored. advanced question-answering techniques, including chain-of-thought @xcite , tree-of-thought @xcite , and program-of-thought @xcite , have demonstrated potential in enhancing the reasoning ability of lms. these methods utilize structured reasoning paths to guide models through multi-step problem-solving processes, thereby enhancing the reliability of their responses. however, these techniques have primarily been explored in contexts such as mathematical, symbolic, and commonsense reasoning. their direct application to leverage the world knowledge embedded in lms for domainspecific question-answering in contact-centers warrants further investigation. over time, enhancing service quality and customer satisfaction have remained focal points of research within the contact-center industry. re-searchers are continuously introducing mechanisms to monitor these in real-time and post-call scenarios. for instance, roy et al., 2016 introduced a real-time quality assurance system employing statistical and rule-based nlp to enable supervisors to monitor ongoing conversations and intervene as needed. quality assurance practices in contactcenters traditionally include sentiment analysis @xcite , emotion recognition @xcite , and compliance management @xcite . moreover, @xcite proposed fine-tuning a roberta-style language model to analyze silences within contact-center conversations, offering proactive feedback to agents and enhancing their performance. however, the integration of lms into contact-center workflows holds significant potential to revolutionize the sector. recent studies explore various methods to transfer reasoning abilities from large models to smaller ones. for instance, deng et al., 2023 experiment with implicit reasoning distilled from a teacher model's hidden states, enabling effective task solving without explicit chain-of-thought reasoning. similarly, li et al., 2023 introduce symbolic chain-of-thought distillation (scotd), enhancing smaller models' performance by training on rationalizations from larger models. additionally, @xcite propose a multi-task learning framework to distill chain-of-thought reasoning, optimizing the integration of reasoning capabilities into smaller models for improved performance. these techniques can be particularly beneficial in resource-constrained environments where deploying large lms may not be feasible. ## conclusion our study evaluates eight language models (lms) for contact-center quality assurance, revealing a strong correlation between model size and performance. claude-3.5-sonnet, from the large group, demonstrated superior proficiency. we propose methods to distill domain knowledge into smaller models, achieving up to 18.95% improvement in macro f1. using evaluation plans generated by more knowledgeable models enhances smaller models' understanding of contact-center conversations. this approach can be further refined through human-in-the-loop feedback, potentially surpassing larger models' capabilities. our ablation study emphasizes the critical role of evaluation plans in guiding smaller models' reasoning. these findings suggest promising avenues for developing ai-assisted evaluation tools in contact-centers, potentially leading to more objective, consistent, and scalable assessment processes. 1. bias and fairness: the underlying asr system utilizes acoustic modeling trained on us-english dialects. to mitigate potential biases: