CS - BERT : a pretrained model for customer service dialogues (2021)

Abstract: Large-scale pretrained transformer models have demonstrated state-of-the-art (SOTA) performance in a variety of NLP tasks. Nowadays, numerous pretrained models are available in different model flavors and different languages, and can be easily adapted to one’s downstream task. However, only a limited number of models are available for dialogue tasks, and in particular, goal-oriented dialogue tasks. In addition, the available pretrained models are trained on general domain language, creating a mismatch between the pretraining language and the downstream domain launguage. In this contribution, we present CS-BERT, a BERT model pretrained on millions of dialogues in the customer service domain. We evaluate CS-BERT on several downstream customer service dialogue tasks, and demonstrate that our in-domain pretraining is advantageous compared to other pretrained models in both zero-shot experiments as well as in finetuning experiments, especially in a low-resource data setting.

## introduction the introduction of large-scale pretrained transformers such has bert and gpt-2 (1; 2) has led to sota results in a variety of nlp tasks, such as question answering (qa) (3; 4; 5), named-entity recognition (ner) @xcite , or text summarization @xcite . the pretraining of these models is performed on large unlabeled text datasets with self-supervised objective functions: auto-regressive language modeling (2; 5; 8), masked language modeling (1; 9), replaced token detection @xcite , or next sentence prediction (1; 11), to name a few. afterwards, the pretrained model can be adapted to a specific task by finetuning on a much smaller, labeled, and taskspecific dataset, thus eliminating the need for huge labeled datasets while still maintaining the power of large-scale transformer models. most of the published pretrained transformer encoders have been pretrained on large text corpora covering general topics and domains, such as the wikipedia corpus or the books corpus. hence, they do not represent the structure that is inherent to a dialogue, i.e., two or more speakers conversing on a topic, exchanging information, or working towards solving a problem. recently, a few pretrained transformer encoders for dialogue tasks have been released, i.e., conversational-bert (12) for chit-chat dialogues, and tod-bert (13) for goal-oriented dialogues. however, dialogue models for customer service are very domain-specific (e.g. check order status, request a refund), and as a result, even the few existing dialogue-specific pretrained models exhibit a language domain mismatch between the pretraining and the downstream task. this language mismatch leads to inefficiencies in the knowledge transfer from pretraining to finetuning, resulting in worse downstream task performance, and requiring larger amounts of labeled finetuning data. in this work, we focus on developing retrievalbased task-oriented dialogue systems, which have been successfully deployed in real-world customer service applications @xcite . in these systems, the dialogue model ranks a pre-defined pool of agent responses, given a customer's input and the conversation history. one of the biggest challenges of developing such models is the limited amount of training data @xcite : high-quality annotated dialogue data is needed, which is expensive to obtain. as one of the most effective solutions, in-domain pretraining has been shown to lead to performance gains under the low-resource setting @xcite . more specifically, for customer service dialogue modeling, @xcite showed that pretraining on a large, unannotated and noisy dataset with subsequent finetuning on a small, high-quality labeled dataset yields improved performance as compared to only training with the small labeled dataset from scratch. however, the authors only evaluated their pretraining-finetuning approach using a rather small pretraining dataset of ca. 380k dialogues, and using a small-scale transformer architecture. in addition, they did not consider advanced pretraining objectives such as masked language modeling (mlm) used in bert, but their pretraining was done using the same training objective as the downstream task. in this paper, we present cs-bert, a bert model pretrained on customer service dialogues, which efficiently reduces the need for expensive highquality labeled dialogue task training data. in particular, we present the following novelties: the paper is structured as follows: section 2 provides a literature overview of bert and other available pretrained encoders. section 3 describes the pretraining strategies that we used to develop cs-bert. in section 4 we describe the datasets and model training details for pretraining, as well as for finetuning on three specific downstream tasks. section 5 follows with experimental results and discussions. we conclude the paper with a summary in section 6. ## related literature in this section, we review related literature. we first introduce bert, then we discuss bert variations that were developed to improve the original model. lastly, we present the available pretrained models for dialogue tasks. ## cs-bert pretraining strategies in this section, we review the different pretraining strategies that we use to train cs-bert. in particular, we use three pretraining strategies: (1) we use the same pretraining objectives as bert, i.e., mlm and nsp/ntp (denoted as cs-bert), (2) we add an additional objective, i.e., masked speaker prediction (msp), to bert's original pretraining objectives (denoted as cs-bert-msp), and (3) we use the same pretraining objectives as bert, but use a different segment embedding strategy (denoted as cs-bert-ssr). ## experiment setup for all experiments in this paper, we use dialogues collected from the customer service of a large ecommerce company. we only use "chat" dialogues, i.e., written interactions between customer service agents and customers (as opposed to email con-versations or transcribed spoken interactions, e.g., via phone support). we will distinguish between two dialogue datasets types:"non-annotated" dialogues and "annotated" dialogues. "non-annotated" dialogues are free text conversations between a customer and an agent, and are inherently noisy. "annotated" dialogues are conversations between customers and specialized agents, who provide annotations and only use a limited pool of approved agent responses @xcite . these annotated dialogues are of higher quality and are less noisy, and thus make it easier for a model to learn from. however, they are also more expensive to obtain. ## summary in this work, we present three cs-bert models pretrained on real-world customer service dialogues. we show that in zero-shot response ranking tasks, the cs-bert models are superior to publicly available pretrained models, and benefit from our dialogue-specific objective function msp. we evaluate cs-bert on three different downstream tasks, and demonstrate that cs-bert consistently outperforms other non-cs-bert models on the response ranking and failure detection task, especially in a low-resource data setting. we observe that cs-bert is superior compared to bert, as well as compared to two models pretrained on dialogue data, i.e., conversational-bert and tod-bert, thus demonstrating the advantage of in-domain pretraining. however we also observe that the performance gains diminish with the size of the finetuning datasets, with some models reaching parity with cs-bert when using all available finetuning data. due to customer data privacy concerns, we will not publicly release our cs-bert models nor the datasets used in this work. however, we hope to inspire other researchers to experiment with pretraining their own, domain-specific bert models. our experiments demonstrate that these domainspecific pretrained models can improve performance on in-domain downstream tasks and can reduce the need for large amounts of expensive labeled datasets.