An Effective, Performant Named Entity Recognition System for Noisy Business Telephone Conversation Transcripts (2022)

Abstract: We present a simple yet effective method to train a named entity recognition (NER) model that operates on business telephone conversation transcripts that contain noise due to the nature of spoken conversation and artifacts of automatic speech recognition. We first fine-tune LUKE, a state-of-the-art Named Entity Recognition (NER) model, on a limited amount of transcripts, then use it as the teacher model to teach a smaller DistilBERT-based student model using a large amount of weakly labeled data and a small amount of human-annotated data. The model achieves high accuracy while also satisfying the practical constraints for inclusion in a commercial telephony product: realtime performance when deployed on cost-effective CPUs rather than GPUs. In this paper, we introduce the fine-tune-then-distill method for entity recognition on real world noisy data to deploy our NER model in a limited budget production environment. By generating pseudo-labels using a large teacher model pre-trained on typed text while fine-tuned on noisy speech text to train a smaller student model, we make the student model 75x times faster while reserving 99.09% of its accuracy. These findings demonstrate that our proposed approach is very effective in limited budget scenarios to alleviate the need of human labeling of a large amount of noisy data.

## introduction we describe a named entity recognition (ner) system that identifies entities mentioned in english business telephone conversations. the input to the ner system is transcripts produced by an automatic speech recognition (asr) system. these transcripts are inherently noisy due to the nature of spoken communication and due to the limitations of the asr system. the transcripts contain dysfluencies, false starts, filled pauses, they lack punctuation information and have incomplete information about case. because there was no pre-existing annotated data set publicly available that matched the characteristics of the asr transcripts in the domain of business telephone conversations @xcite , the ner model is required to be trained on a large dataset containing telephone conversations to effectively detect named entities in such noisy data. moreover, the ner model needs to provide realtime functionality in a commercial communicationas-a-service (caas) product such as displaying information related to the named entities to a customer support agent during a call with a customer. the deployed system was therefore required to be fast (less than 200ms inference time) but economical (able to operate on cpu, rather than more expensive gpus). to address the above issues, in this paper, we present a simple yet effective method, distill-thenfine-tune, to transfer knowledge from a large and complex model to a small and simple model while reaching a similar performance as the large model. more specifically, we fine-tune a state-of-the-art ner model, luke @xcite , on our limited amount of noisy telephone conversations and predict the labels of a large amount of unlabeled conversations, denoted as distillation data. the smaller model is then trained on the distillation data using pseudo-labels. we conduct extensive experiments with our proposed approach and observe that our distilled model achieves 75x inference speed boost while reserving 99.09% f1 score of its teacher. this makes our proposed approach very effective in limited budget scenarios as it does not require the annotation of a huge amount of noisy data that would otherwise be required to finetune simpler transformers on downstream tasks. ## related work ner is often framed as a sequence labeling problem @xcite where a model is used to predict the entity type of each token. previously, various models based on the recurrent neural network architecture have been widely used for this task. in recent years, pre-trained language models have been employed to perform the ner task where a new prediction layer is added into the pre-trained model to fine-tune for sequence labeling @xcite . more recently, @xcite proposed a new approach to provide the contextualized representations of words and entities based on a bidirectional transformer. in their proposed model, luke, they treat words and entities in a given context as independent tokens, and output the contextualized representations of them. the luke model achieved impressive performance in various entityrelated tasks. however, this model is inherently slow due to its complex architecture and so it is not applicable for usage in production environments in a limited computational budget scenario. in scenarios where the computational budget is limited, using a smaller model that can mimic the behaviour of the large model can be used. knowledge distillation @xcite is one such technique where a large model is compressed into a small model. one prominent approach for knowledge distillation that has been used in recent years is the work of @xcite , where they proposed a task specific knowledge distillation method to show that using an additional unlabeled transfer dataset can augment the training set for more effective knowledge transfer. however, most prior work that leveraged such knowledge distillation techniques focused on typed input, whereas the amount of work that leveraged knowledge distillation for noisy texts (e.g., telephone conversation transcripts) is very limited @xcite . motivated by the advantages of knowledge distillation, in this work, we also leverage knowledge distillation to address the computational issues that occur while utilizing large state-of-the-art language models in a limited computational environment, while minimizing the amount of noisy data that must be human-annotated for use during fine-tuning. ## datasets in this section, we first introduce the in-domain training data (noisy human-to-human conversations) that we sampled and annotated to train the teacher model. then, we describe the data used for knowledge distillation of the student model. ## our proposed approach in this section, we first describe the architectures of the teacher and student models. we then detype # examples positive utterances 347,412 negative utterances 136,354 utterances containing person tags 179,495 utterances containing prod/org tags 97,857 utterances containing location tags 138,989 ## experiments in this section, we describe our experimental settings and results. ## conclusion in this paper, we introduce the distill-then-fine-tune method for entity recognition on real world noisy data to deploy our ner model in a limited budget production environment. by generating pseudolabels using a large teacher model pre-trained on typed text while fine-tuned on noisy speech text to train a smaller student model, we make the student model 75x times faster while reserving 99.09% of its accuracy. these findings demonstrate that our proposed approach is very effective in limited budget scenarios to alleviate the need of human labeling of a large amount of noisy data. in the future, we will explore how to apply knowledge distillation to other tasks @xcite containing noisy data.