May I Ask Who’s Calling? Named Entity Recognition on Call Center Transcripts for Privacy Law Compliance (2020)

Abstract: We investigate using Named Entity Recognition on a new type of user-generated text: a call center conversation. These conversations combine problems from spontaneous speech with problems novel to conversational Automated Speech Recognition, including incorrect recognition, alongside other common problems from noisy user-generated text. Using our own corpus with new annotations, training custom contextual string embeddings, and applying a BiLSTM-CRF, we match state-of- the-art results on our novel task.

## introduction when a call center says "a call may be recorded", they are often collecting a transcript. these transcripts are the output of speech recognition systems, and while they are redacted for payment card industry (pci) compliance, they often contain other information about the caller such as their name and internal id number. this data can be helpful for quality assurance and future customer care. new privacy laws, such as the general data protection regulation (gdpr) in the eu, define rules and regulations for everything from how data is collected and stored to the rights of a person to retract their consent to the use of their data @xcite . in the face of these new laws, it is important to be able to identify non-public personal information and personally identifiable information (npi/pii) in call transcripts in order to comply with regulations without compromising the data these companies rely on. we use named entity recognition (ner) to find instances of npi/pii, remove them from a transcript, and replace them with a tag identifying which type of information was removed. for example, a transcript containing "this is john doe reference number 12345" would become "this is @xcite reference number @xcite ". this problem is unique in a call center for a few reasons. firstly, call transcripts are organic human conversations and present many of the common problems of user-generated data, including false starts, incomplete sentences, and novel words. secondly, the text provided in a transcript is the output of an automatic speech recognition (asr) system, which is prone to error as described in section 3.1. while modern asr systems are reliable, our input audio is from phone calls, which are usually very lowquality and often contain a lot of background noise. this low-quality audio results in poor asr, which then outputs sentences that may not be grammatical. this makes it difficult to understand the semantics of the call or to pick up on many of the features that are critical to most ner systems such as context or part of speech. additionally, production-level call transcripts, or those that are used by quality assurance agents and data scientists, are missing capital letters, numeric digits, and accurate punctuation, which are features that are crucial to the classic approaches to ner. moreover, traditional ner systems use labels for proper nouns, like people's names, but have no way to handle emails, spellings, or addresses, making bootstrapping from pretrained ner models impossible. in this paper, we apply the current state-of-theart neural architecture for sequence labeling, a bilstm-crf, to our novel call center transcripts in search of npi and pii as identified by a human. we match state-of-the-art performance for standard datasets on our novel problem by using our model in conjunction with annotated data and custom contextual string embeddings. ## previous work ner became popular in the nlp community at the message understanding conferences (mucs) during the 1990s @xcite . in 2003, the conll2003 shared task focused on language independent ner and popularized feature based systems @xcite . the ontonotes corpus, released in 2006, has also been fundamental to ner research @xcite . after conll, the highest performing models were based on a crf @xcite which requires the manual generation of features. more recently, research has used neural techniques to generate these features. huang et al. (2015) found great success using bidirectional long short term memory models with a crf layer (bilstm-crf) on both the conll2000 and conll2003 shared task datasets. ma and hovy (2016) used a bilstm-cnn-crf to do ner on the conll2003 dataset, producing state-of-the-art results. similarly, chiu and nichols (2015) used a bilstm-cnn, with features from word embeddings and the lexicon, which produced very similar results. ghaddar and langlais ( 2018 ) used embeddings for the words and for entity types to create a more robust model. flair, proposed by @xcite , set the current state of the art by using character based embeddings, and built on this with their pooling approach in 2019 @xcite . crossweigh, a framework introduced by @xcite , makes use of flair embeddings to clean mishandled annotations. in 2006, sudoh et al. used the word confidence scores from asr systems as a feature for ner on the recordings of japanese newspaper articles. in 2018, @xcite conducted a similar experiment on french radio and tv audio. unlike our data, neither of these tasks used spontaneous conversation. additionally, the audio was probably recording-studio quality, making asr a reliable task. ## data our dataset consists of 7,953 training, 500 validation, and 534 test samples. each sample is a complete speaker turn from a call taken by a call center that deals with debt collection. for our purposes, a speaker turn is defined as the complete transcription of one speaker before another speaker starts talking, as illustrated in figure 1 . the training set is a random sample of turns from 4 months of call transcripts from the client, but was manually curated to contain examples of npi/pii to compensate for its relatively rarity in call center conversation. the transcripts were made using a proprietary speech recognition system, which is set to provide all lowercase transcripts and omits punctuation and numeric digits. we used spacy foot_0 to convert each turn to a document that starts with a capital letter and ends with a period. this is due to the default configurations of spacy-in order to make use of entities, we needed to add in a sentencizer module, which defaults to this capital letter and period set up. ## model design we implemented a standard bilstm-crf model in pytorch. the basic model implementation is adapted from a github repository foot_1 . we wrote our own main.py to better allow for our spacy preprocessing, and we also adapted the code to handle batch processing of data. after this preprocessing, we trained the model with the training set and used the validation set for any model tuning. all reported numbers are on the test set and occur after all tuning is completed. a visualization of our model is found in figure 2 . ## discussion the results shown in interesting variability also occurred in ad-dress entities. with address, both models that used flair and fasttext embeddings strongly outperformed the models that used flair on its own, but standard flair embeddings strongly outperformed the pooled flair embeddings. neither version of the flair only model identified addresses in which house numbers or zip codes were shown as "[redacted]" but both models that utilized fast-text had no issue with these examples. ## conclusion and future work by using a bilstm-crf, in conjunction with custom-trained flair embeddings, we match current state-of-the-art ner performance on our novel call center conversation dataset with unique entity types. we also reinforce the importance of training word embeddings that fully capture the nuances of the data being used for the task. while we cannot release any data for privacy reasons, we have shown that current state-of-the-art techniques successfully carry over to more non-traditional datasets and tasks. in the future, we'd like to assess the contribution of this model with the call transcripts from other industries. additionally, we'd like to investigate the success of these strategies on other usergenerated conversations, such as chats and emails.