Improved Named Entity Recognition for Noisy Call Center Transcripts (2021)

Abstract: We explore the application of state-of-the-art NER algorithms to ASR-generated call center transcripts. Previous work in this domain focused on the use of a BiLSTM-CRF model which relied on Flair embeddings; however, such a model is unwieldy in terms of latency and memory consumption. In a production environment, end users require low-latency models which can be readily integrated into existing pipelines. To that end, we present two different models which can be utilized based on the latency and accuracy requirements of the user. First, we propose a set of models which utilize state-of-the-art Transformer language models (RoBERTa) to develop a high-accuracy NER system trained on custom annotated set of call center transcripts. We then use our best-performing Transformer-based model to label a large number of transcripts, which we use to pretrain a BiLSTM-CRF model and further fine-tune on our annotated dataset. We show that this model, while not as accurate as its Transformer-based counterpart, is highly effective in identifying items which require redaction for privacy law compliance. Further, we propose a new general annotation scheme for NER in the call-center environment.

## introduction the recent promulgation of new privacy laws, such as the european union's general data protection regulation (gdpr) and the california privacy rights act has drawn additional attention to the need for businesses to effectively identify and redact sensitive personally identifiable information (pii) in their business records. for the call center industry, which frequently collects transcripts of customer calls, this task is particularly tricky as transcripts tend to be an extremely noisy data source. this noise results from several factors, including errors in automated speech recognition (asr), the use of single-channel audio (recordings that do not differentiate between users), and speech disfluencies. in this paper, we explore the use of a named entity recognition (ner) task to identify those items in transcripts which should be redacted for privacy law compliance. most existing ner models are trained on well-formed written text. due to the noisy data in call center transcripts and domain language differences the use of such models in our task is not ideal. additionally, although our task is quite similar to traditional ner, such as the @xcite , the set of entities which we need to identify for privacy law compliance is significantly larger than that used in many ner tasks. for example, traditional ner label sets do not contain tags for entities such as telephone numbers, account numbers, or email addresses. to address this issue, we propose a new ner label set designed to meet the requirements of privacy law compliance in the call center industry. our ner model also lends itself to uses such as identifying product mentions for call routing. given the recent dominance of large pretrained transformer language models such as bert @xcite in nlp, we explore the application of transformer lms to the task of ner on call center transcripts. we demonstrate that fine-tuning these models on unlabeled, in-domain data, then further fine-tuning on a custom annotated set of call center transcripts, results an effective model for entity identification. further, we explore the use of these models to label a large number of transcripts, which we then use as training data for a smaller bilstm-crf sequence prediction model. this model, while not as accurate as its transformerbased brethren, can allow for less computationally expensive inference. the key contributions of this paper are demonstrating the efficacy of transformer lm-based ner models for use in the call center domain and the use of said models to generate training data for smaller models which may be better suited to production environments. we demonstrate that a bilstm-crf using flair @xcite contextual string embeddings is approximately 7 times slower than our best-performing bilstm-crf model using fixed pretrained subword embeddings, with only marginal f1-score improvement. additionally, the flair-based lstm model (0.73 micro-f1) is 3 times slower than our best-performing robertabased model (0.82 micro-f1) and 7 times slower than a distilroberta model (0.81 micro-f1). ## previous work ner has long been a popular task in the nlp community, and many public benchmark datasets exist to test the performance of various sequence labeling algorithms on the ner task. initially developed as part of the message understanding conferences @xcite , the task was further popularized by the conll 2003 shared task on ner @xcite . the conll 2003 dataset and the ontonotes corpus @xcite remain popular benchmarks for sequence labeling algorithms. the conditional random field @xcite was, and remains, a popular method for sequence labeling, including ner @xcite . @xcite achieved success in the ner task by combining the crf with the bidirectional lstm @xcite . similarly, @xcite achieved state-of-the-art ner results with a bilstm-cnn-crf model. more recently, with the advent of pretrained transformer language models such a bert @xcite and roberta @xcite , attention has shifted to using these models for the ner task. @xcite use an ner sequence labeling task as an example of the power of bert. additionally, @xcite combine bert with a crf layer to achieve impressive results for ner in portuguese. in this paper, we utilize both transformer-based ner and bilstm-crf models for ner on a custom annotated dataset of call center transcripts. the use of automated methods for text anonymization has a long history in both nlp and the biomedical domain. for example, @xcite report on the use of regular expressions for removing pii from sms messages. gorinski et al. (2019) and pérez-díez et al. (2021) inves-tigate the use of ner for anonymizing medical records. most relevant to the present work is kaplan (2020), who use a bilstm-crf model for pii redaction in call center transcripts. ## data our primary dataset consists of 1422 transcripts of call center recordings generated by a commercial asr provider, with an average length of 464.4 tokens; the entire dataset contains 660,358 tokens. the dataset is split into a train set containing 594,947 tokens and a validation set containing 65,411 tokens. our held-out test set on which all models are evaluated consists of 44 transcripts containing 41,961 tokens. the dataset contains a total of 21,537 entity spans. all tokens are converted to conll 2003 bio format for training our sequence prediction models. an example of our data can be found in appendix a. in addition to our labeled data, we use a larger set of 99,796 unlabeled call transcripts for domain adaptation of our pretrained language models, as well as for generation of synthetically labeled training data. ## models we propose two classes of model trained on our annotated dataset. the first class applies state-of-theart transformer lms to the ner task, while the second class uses a bilstm-crf architecture. our initial transformer model harnessed bert, however early testing indicated that roberta generally performed better for our target task. as our goal was to create an efficient model for production use, we chose to not explore model ensembling, though this is a potential direction for future work. our transformer-based models were built using huggingface's tranformers package foot_1 for pytorch. our best performing model uses a roberta-base pretrained model to encode input token sequences, which are then passed to a linear layer for dimension reduction, and finally to a crf layer (implemented using pytorch crf foot_2 ) to make final label predictions. our bilstm-crf model, which is implemented in pytorch, consists of a 4-layer bidirectional lstm encoder with 300 hidden units per layer. as with our transformer model described above, encoder output is passed to a linear layer prior be being passed to a crf layer for final prediction. input to the model consists of 100dimension pretrained byte-pair subword embeddings generated using bpemb @xcite . we also tested a bilstm-crf model which uses flair embeddings @xcite as input, similar to the model proposed in @xcite . ## experiments to directly compare the performance of our various models, we train each model on our final annotated training dataset and evaluate on our held-out test set. additionally, we analyze the perfomance of our bilstm-crfs relative to the transformer model, as it is significantly faster and requires fewer computational resources. we test two different pretrained models from the transformers library: roberta @xcite , and distilroberta @xcite . all models are fine-tuned on our annotated training data for 6 epochs using the adamw @xcite ) optimizer, with an initial learning rate of 5e-5 and epsilon of 1e-8. we use linear warmup to reach the initial learning rate over the first training epoch. we select the model checkpoint with the best validation micro-f1 for further evaluation. we also train each model with and without a sliding context window of 99 tokens on either side of the target input, as described in @xcite . the size of the context window was determined by testing various context window sizes; we found 99 tokens to provide the most improvement in model performance. next, we test the performance of our bilstm model, which we train with two types of input token embeddings as described in section 4. as with the transformer models, we use the adamw optmizer with an initial learning rate of 5e -5 and epsilon of 1e -8. we also experimented with the adam optimizer with a lr of 0.001, with nearly identical results. we implement early stopping on validation f1 with a patience of 6. our bestperforming model triggered early stopping after 17 epochs. given that our lstm models are significantly faster than our transformer models on cpu, we wished to test an lstm model trained on additional roberta-labeled data. we therefore use our best-performing roberta model to label 20,000 unlabeled transcripts, which we then use to train a bilstm-crf model. once validation f1 stops all model training is conducted using an nvidia geforce gtx 1080 gpu with 8 gb of memory. inference is performed on both gpu and cpu to test inference latency. f1 and total test runtime of each of our best-performing models are shown in ## discussion our experiments demonstrate that pretrained transformer language models, such as roberta, can lead to significant performance improvements for ner labeling of call center transcripts when compared to previously published methods. while this is not terribly surprising, given the number of nlp tasks in which transformer lms have come to dominate, this paper is the first application of said models to the domain of call transcripts. the only work which has focused on ner for call center transcripts, kaplan (2020) used a bilstm-crf model only; while we are unable to replicate their work due to the unavailability of their data foot_3 , our experiments using a similar model design show that our proposed transformer models achieve improved performance on this task. our lstm model uses pretrained, rather than custom, flair embeddings; however, given the improvements for custom embeddings reported by @xcite we would not expect training custom embeddings to result in a model which outperforms our best transformerbased model. as previously mentioned, call center transcripts are a particularly noisy type of input data; noise sources include disfluencies in spoken language, poor audio quality, and asr errors. given this fact, it can be challenging for both annotators and a model to correctly identify entity boundaries. as previously mentioned, all f1 scores reported for model performance use "strict" evaluation. however, such precision is not necessarily required for a model to be effective for redaction of pii. for example, if only the majority of an email is captured by the model, the resulting redaction may still be sufficient. of note is the fact that our best performing model achieves an "entity type" f1 score of 0.90; this metric requires that entity types be labeled correctly, but that span boundaries need only overlap. this score indicates that the model is correctly labeling the large majority of spans, even if its predicted boundaries do not exactly match those of the annotator. ## conclusion this paper presents the first application of pretrained transformer language models to ner for the call transcript domain. this is a unique domain in many ways, particularly due to the large amounts of noise present in the data. we demonstrate that transformer lms, namely roberta and distil-roberta, outperform the previous state of the art model for this domain on our custom dataset. additionally, we show that these models are significantly faster to run than the previous state of the art model. finally, we show that a transformer lm model can be used to generate training data for a smaller, less computationally greedy lstm model, resulting in substantial performance improvement for the smaller model. this finding provides additional choice to end-users wishing to develop an ner system for production use in a call center environment, and potential future directions for improving model efficiency and performance. a.2 example model output thank o you o for o calling o eastbank b-organization . o my o name o is o william b-person yee i-person . o i o have o your o first o and o last o time o . o john b-person lyon i-person . o and o mislead o me o i o have o your o 51 o digit o account o number o or o your o social o . o social o i'll o give o it o to o you o zero b-id seven i-id eight i-id zero i-id five i-id one i-id one i-id two i-id zero i-id . o thank o you o . o and o how o may o i o help o you o today b-datetime . o well o i'm o t o trying o to o see o if o they o gave o me o the o credit o that o they o were o supposed o to o give o me o the o store o . o okay o and o i o know o you're o calling o for o you o to o actually o funded o your o account o today b-datetime which six do you prefer . my mother's maiden name . can you help me with your mom's maiden name . clayton . ma'am i need the three digits on the back of your card please . 0 0 0 . okay thank you so much . just give me two moment here . thank you for patiently waiting . there you go information has been completely updated . miss blue many help you with your account . i was trying to find out the balance and didn't make a payment but it went straight to a payment and i wanted some clarity first . mm hmm . it didn't okay communicate . . okay . we apologize for the inconvenience ma'am . if you wanted to check do you wanted to check the statement balance . ma'am that is due on the current due date or the whole balance including recent charges . whole balance . member current balance is four thousand two hundred twelve and 28 cents . four thousand . say it again . four thousand two hundred twelve to 2 3 5 . and and twenty my credit and twenty eight . cents man . my credit limit was worth . six thousand . okay can i make a four hundred dollar payment . yes of course ma'am . since you're having issues with the automated system let me just help you with the payment . just one time free of charge for tonight ma'am . okay . because customer service normally charge @xmath0400 payment using your checking account ending insert a 0 0 0 . again $10 if he has been waived . so there you go . miss blue you will are if you have questions or reached the modified as payment please call us at the number on your statement . this confirms that today march 9 two thousand twenty you authorize us to make a single that a chinese debit of seven eighty yeah dollars from your bank to pay your west credit card account on march 9 . two nineteen twenty . is this correct . thank you . payment will successfully process and it will be dated today and it will post within 13 hours under your account for other payment options . you can pay at any local west store truly automated system ensure . the website . all of them are for free . do you need your confirmation number . can you write this one down please . it's going to be 0 0 0 0 0 0 0 0 0 to . okay . there you go . would there be anything else that i could further check man . i just wanted to make sure they haven't missed any payments on my account of . he was a very my credit . with my account is in good standing . that's in good standing . no need to worry ma'am . that is correct . thank you . you're welcome as . green . there are no more concerns ma'am . i'm glad i was able to assist and . thank you for calling card services . you ever get one ma'am . we value your business .