Developing a Production System for P urpose of C all Detection in Business Phone Conversations (2022)

Abstract: For agents at a contact centre receiving calls, the most important piece of information is the reason for a given call. An agent cannot provide support on a call if they do not know why a customer is calling. In this paper we describe our implementation of a commercial system to detect Purpose of Call statements in English business call transcripts in real time. We present a detailed analysis of types of Purpose of Call statements and language patterns related to them, discuss an approach to collect rich training data by bootstrapping from a set of rules to a neural model, and describe a hybrid model which consists of a transformer-based classifier and a set of rules by leveraging insights from the analysis of call transcripts. The model achieved 88.6 F1 on average in various types of business calls when tested on real life data and has low inference time. We reflect on the challenges and design decisions when developing and deploying the system.

## introduction the purpose of call as we define it is similar to a thesis statement in an argument: it introduces the speaker's intent, the broad theme or topic of a conversation, any key entities, and relevant relationships between them. the purpose of call statement might also include a linguistic signpost -an indication to the listener that the utterance is intended to convey the purpose of the speaker's call. for instance: i'm calling because [signpost] i'm trying to open one of the programs [entity] on my computer @xcite and it's not opening [relation] so i'm hoping i can get some assistance @xcite with that. 1 purpose of call statements in a contact centre setting are usually uttered by the customer in inbound calls, and by the agent in outbound calls. the purpose of call is typically stated near the beginning of the call, is often stated in a single utterance, and does not contain extra information. atypically, we may sometimes see the purpose of call occurring in the middle of a conversation, occurring across several utterances, being implicit, or being uttered by a call recipient rather than a call initiator. the models described below have been implemented in the dialpad contact center product and are running in production. the purpose of call is extracted from an automatic speech recognition (asr) generated transcript in near-realtime and displayed in a dashboard used by call center supervisors to monitor calls taking place. the dashboard shows information about the caller and the agent, the duration of the call, the purpose of call, and customer sentiment. a separate analytics component clusters the purpose of call from all calls in a call center during a time period to provide insights about trends and anomalies, customer pain points, and common problems and knowledge gaps among agents. additional use-cases include showing the purpose of call in a summary of prior calls with a customer, and including the purpose of call in summaries of the conversation. the utterance segment containing the purpose of call is highlighted in the call transcript and the call recording to be easily accessible to agents and call center supervisors. these use-cases are summarized in figure 1 . the purpose of call feature is used to help call center managers to navigate to relevant sections of conversations to identify areas to coach sales and support agents and sample relevant calls. through customer education, we emphasize that the feature should not be used for automated evaluation of agent performance. there are a few challenges that arise when building an automatic system to detect purpose of call. diversity of purpose of call statements. this type of detection system should be an open-class system, since call purposes vary across different domains, industries, and types of calls. robustness to noise. there are challenges related to the fact that purpose of call extraction relies on the output of an asr system. there are two sources of noise in the asr transcripts: language production issues such as false starts, dysfluencies, filled pauses, inconsistency in conversational turntaking @xcite @xcite as well as their representation in the asr system and recognition errors due to acoustic noise. limitations in training data. existing intent detection datasets do not reflect real world settings (e.g. they do not distinguish the purpose of call from other intent-like statements, and are limited to a subset of domains). manually annotating data (e.g. using crowd-sourced annotators) raises privacy concerns since annotators must have access to a full conversation transcript in order to find the best purpose of call. annotation is a complex task that requires highly trained annotators, and is expensive and time-consuming because annotators must consider the larger context of the conversation to make a judgment. computational efficiency. the need for the system to extract the purpose of call statement in real time imposes constraints on memory consumption, latency, and inference speed. this paper describes an end-to-end system to extract a purpose of call statement from the transcript of a business telephone call. our contributions are three-fold: ## related work the concept of a purpose of call statement has its roots in the conversation analysis framework @xcite . within this framework, which combines perspectives from linguistics and sociology, a conversation is understood to be composed of turntaking utterances, with each "turn" being indicated via linguistic and paralinguistic cues. conversational turns often form adjacency pairs such as question-answer pairs or offer-acceptance/refusal pairs. there are other key aspects of a conversation as well. for instance, a conversation is likely to end after a linguistic cue known as a "closing" is given; likewise, there is usually a linguistic indicator that a conversation is being initiated: an opening @xcite @xcite . most work analyzing telephone conversation openings within the framework of conversation analysis has been conducted on english, but similar patterns have been observed in other languages, including german and farsi @xcite . within a contact center environment, the purpose of call is, like openings and closings, an inte-gral aspect of the conversation (i.e. call) between customer and support agent. we propose that a purpose of call is a particular conversational feature that is necessary in contact center calls, and is distinct from the call opening, the body of the call, and the call closing. the first 120 seconds of a customer support call are predictive of that call's outcome @xcite . the purpose of call statement typically occurs within this timeframe, so highlighting a purpose of call in real time could provide agents with additional support in meeting customer needs. ## methodology we formulate the purpose of call detection task as a binary classification problem. each call, after being transcribed by the asr system, is represented as a sequence of utterances, which may consist of one or more sentences. the division into utterances is based on acoustic features such as silent pauses and the length of a speech fragment. for a given utterance, we determine the probability that the utterance is the purpose of call statement for that particular call. we impose the following constraints on this task: (i) for a given call, there is only one most probable purpose of call. (ii) only calls with two call sides (agent and customer) are considered, which excludes multiparty business conversations. (iii) the model should make a prediction as the call is ongoing and therefore will not have access to the full conversation. due to the lack of available annotated data representing the concept of purpose of call, we followed an iterative approach to develop the model, consisting of three steps: (1) computational linguists on-staff conducted extensive linguistic analysis of transcripts to identify the characteristics of purpose of call statements. (2) we then implemented a knowledge-engineered approach to identify these purpose of call statements. (3) we bootstrapped from the knowledge-engineered solution, using it to label training data for a transformer-based approach. we select a transformer-based model as it is the current state-of-the-art in sequence classification and is known to have better generalization power than rule-based models. we evaluate the performance using f1, precision, and hit rate, i.e. the number of calls in which a purpose of call was detected out of all available calls. we measure hit rate in calls at least 30 sec-onds long, based on the observation that shorter calls may not include any content (e.g. because the caller hung up before starting the conversation). the model is tested on an automatically obtained validation set that represents 10% (18k utterances) of the training data, a manually annotated gold test set of 13215 utterances from 909 calls, and unlabeled samples from 600 real-life calls. ## evaluation qualitative analysis was conducted on the gold set: real life user data of 600 samples, and 200 calls with missed hits. false positives mainly correspond to signposting language without mention of the actual purpose of call, and indicate the model's overreliance on lexical features. the model was found to be accurate in assigning utterances to classes, but not always sensitive to the difference between a ## conclusion this paper discusses the development and deployment of a hybrid system to detect a purpose of call statement in business call transcripts for the english language in near-realtime settings. we introduce the concept of the purpose of call, provide in-depth analysis of real life data, and discuss overcoming the absence of available training data by bootstrapping from a knowledge-engineered model to a deep learning one. both the knowledge-engineered and hybrid models demonstrate high precision and hit rate, with the hybrid model showing better performance while maintaining computational efficiency. use. the purpose of call feature is used by call center managers to identify areas to coach sales and support agents. it is recommended to not use this feature for automated evaluation of agent performance. incorrect purpose of call prediction may provide unsatisfactory user experience for the managers as they sample calls but does not present any risk of negative impact for the agents. licensing. we follow the licensing requirements accordingly while using external tools such as huggingface foot_2 and multimodal-toolkit @xcite libraries. mining of business-oriented conversations at contact centers. in proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (emnlp-conll), pages 458-467, prague, czech republic. association for computational linguistics. carmen taleghani-nikazm. 2002. a conversation analytical study of telephone conversation openings between native and nonnative speakers. journal of pragmatics, 34:1807-1832. piotr żelasko, jan mizgajski, mikołaj morzy, adrian szymczak, piotr szymański, łukasz augustyniak, and yishay carmiel. 2019. towards better understanding of spontaneous conversations: overcoming automatic speech recognition errors with intent recognition.