An End-to-End Dialogue Summarization System for Sales Calls (2022)

Abstract: Summarizing sales calls is a routine task performed manually by salespeople. We present a production system which combines generative models fine-tuned for customer-agent setting, with a human-in-the-loop user experience for an interactive summary curation process. We address challenging aspects of dialogue summarization task in a real-world setting including long input dialogues, content validation, lack of labeled data and quality evaluation. We show how GPT-3 can be leveraged as an offline data labeler to handle training data scarcity and accommodate privacy constraints in an industrial setting. Experiments show significant improvements by our models in tackling the summarization and content validation tasks on public datasets.

## introduction an integral part of salespeople daily routine is summarizing sale calls. the summarization process aims to distill salient information from sales dialogues into succinct highlights, which are then leveraged by salespeople for productivity and coaching purposes. manually curating a call summary is considered as one of the biggest time wasters for b2b sellers @xcite . it distracts salespeople from nurturing the relationship with their next customer. recently, this practice has become more demanding due to the emerging landscape of remote selling where virtual meetings become the new norm @xcite . dialogue summarization induces a variety of unique challenges compared to summarization of documents such as news or scientific papers @xcite . information density is a key challenge in dialogue text; information is scattered over multiple utterances and participants, leading to frequent coreferences and topic alternations. spoken dialogues, usually transcribed by speech recognition engines, impose additional challenges such as redundancies and misrecognized words. the length of these dialogues, e.g. 50k tokens in a 45 minutes call, imposes another challenge to state-of-the-art summarization models as it exceeds their input limits @xcite . figure 1 illustrates parts of the challenges imposed by automatically transcribed sales dialogues. developing a production system which is both fully automatic and agnostic to the input text genre is an extremely difficult task given the current state-of-the-art technology. to this end, we present a pragmatic solution that enables users to interactively edit machine-generated summary for customer-agent sales calls as appears in figure 2 . our solution summarizes the call into a collection of abstractive highlights to accurately capture the various details of the call. the machine-human interaction is enabled through a designated human-in-the-loop user experience @xcite . it enables users to modify the generated summaries, yet, the intervention is designed to be minimal so that the overall time consumption of users is significantly reduced. overall, our contributions are listed as follows: 1. dialogue summarization system. we introduce an innovative production summarization system for summarizing call transcripts with a human-in-the-loop setting. our system uses an advanced summarization model to generate abstractive summaries for dialogues. additionally, it employs a novel model for quantifying the coherence of the summaries to compensate for the summarization model limitations. ## method while most existing methods summarize a call transcript as a single paragraph, our system provides a collection of sentences that summarize the entire dialogue in a chronological order. given a call transcript, the system utilizes word embeddings to break the transcript into semantically coherent segments @xcite . each segment is summarized independently capturing key information such as: customer's issue, agent's solution or the underlying topic of the discussion. finally, the grammatical coherence of highlights is analyzed using a dedicated model before suggesting them to the user. figure 3 provides a high-level overview of the system's flow. next, we introduce the key components of our dialogue summarization system in details. ## gpt-3 as an offline labeler training a dialogue summarization model requires a large amount of labeled examples. manually annotating data for abstractive summarization is a time-consuming and labor-intensive process, let alone the data privacy constraints. in this work, we provide a method to automatically pseudo label examples to overcome these challenges. we leverage gpt-3 model @xcite to generate pseudo labels in a zero-shot setting for each call segment. gpt-3 is a large auto-regressive language model with 175 billion parameters that achieves promising results on various nlp tasks, including question answering. we treat the problem of label generation as a question answering task. for each segment, we concatenate a question-based prompt with the segment's content while expecting the gpt-3 model to provide the answer as appears figure 4: input representation of dialogbart's encoder. in figure 5 . these answers are used as pseudo labels for the corresponding segments. this formulation provides the flexibility of defining multiple questions per segment to summarize the segment from different perspectives. finally, these pseudo labels, combined with proprietary human labeled data, are used to fine-tune the dialogbart model on conversational text. ## custom evaluation metric common evaluation metrics for text summarization task, i.e. rouge and meteor, have salient limitations as both metrics track lexical overlap between the summary and the original text. this kind of assessment falls short when the summary content perfectly aligns with a reference text but does not necessarily contain any lexical overlap, e.g. abstractive summaries. in an industrial setting, one needs to consider various quality perspectives to guarantee that the summary's quality does not introduce productivity blockers for users or negatively affect business decisions. we introduce a custom evaluation metric, sumsim, that relies on lexical overlap as well as other quality aspects to ensure that summaries are fluent, coherent and factually reliable. sumsim aims to cover the following quality perspectives: ## experimental results in this section we evaluate the performance of our proposed models on various datasets: dialogsum @xcite , samsum @xcite , cola @xcite and a proprietary data from the sales domain. we also show the potential of sumsim metric compared to traditional evaluation metrics on the text summarization task. we use huggingface transformers @xcite as a training framework in all of our experiments. ## conclusions in this paper, we present an end-to-end system for abstractive summarization of agent-customer calls. we employ a two-stage strategy to summarize long call transcripts by (i) splitting the dialogue into semantically coherent segments, and (ii) generating summaries using our dialogbart summarization model. we demonstrate how a pragmatic solution that combines a content selection model with a human-in-the-loop user experience can help compensate on generative models' limitations. we show how gpt-3 model can be leveraged as an offline data labeler to train lighter models and accommodate data privacy constraints. experiments show that the introduced embedding parameters combined with fine-tuning on in-domain data significantly improve the quality of the generated summaries with respect to publicly available bartbased summarization models. we emphasize the need for non-standard evaluation metrics and show how common metrics fall short when evaluation of abstractive summaries is considered.