Improving Named Entity Recognition in Telephone Conversations via Effective Active Learning with Human in the Loop (2022)

Abstract: Telephone transcription data can be very noisy due to speech recognition errors, disfluencies, etc. Not only that annotating such data is very challenging for the annotators, but also such data may have lots of annotation errors even after the annotation job is completed, resulting in a very poor model performance. In this paper, we present an active learning framework that leverages human in the loop learning to identify data samples from the annotated dataset for re-annotation that are more likely to contain annotation errors. In this way, we largely reduce the need for data re-annotation for the whole dataset. We conduct extensive experiments with our proposed approach for Named Entity Recognition and observe that by re-annotating only about 6% training instances out of the whole dataset, the F1 score for a certain entity type can be significantly improved by about 25%.

## introduction we describe a named entity recognition (ner) system that needs to provide realtime functionality in a commercial communication-as-a-service (caas) platform such as displaying information related to the named entities to a customer support agent during a call with a customer. the primary focus for this ner system is to identify product and organization type entities that appear in english business telephone conversation transcripts. since these transcripts are produced by an automatic speech recognition (asr) system, they are inherently noisy due to the nature of spoken communication as well as limitations of the asr system, resulting in dysfluencies, filled pauses, and lack of information related to punctuation and case @xcite . these issues make the annotation of such noisy datasets very challenging for the annotators, making the annotation job for such data more difficult than datasets that contain typed text @xcite . to our best knowledge, the publicly available ner datasets mostly contain typed text. moreover, there are no existing ner datasets that match the characteristics of the asr transcripts in the domain of business telephone conversations @xcite . thus, training an ner model in the context of business telephone transcripts requires a large annotated version of such data @xcite consisting of product or organization type entites that usually appear in business conversations @xcite . however, the difficulties to understand noisy data may lead to annotation errors even in the human annotated version of such data. to address the above issues, in this paper, we present an active learning @xcite framework for ner that effectively sample instances from the annotated training data that are more likely to contain annotation errors. moreover, we also address a very challenging task that is not widely studied in most of the existing ner datasets, distinguishing between organization and product type entities. in addition, since the ner model needs to provide realtime functionality in a commercial caas product, we show how data re-annotation through our active learning framework can even help smaller models to obtain impressive performance. ## related work in recent years, transformer-based pre-trained language models have significantly improved the performance of ner across publicly available academic datasets leading to a new state-of-the-art performance @xcite @xcite . however, there remain several issues in these existing benchmark datasets. for instance, most of these datasets are constructed from articles or the news domain, making these datasets quite well-formed with punctuation and casing information, along with having rich context around ## dataset construction the dataset used in this paper is constructed from transcripts produced by an asr system (see for data annotation, at first, we sampled 78,983 utterances containing human to human business telephone conversation transcripts and sent to appen foot_0 for annotation. we asked the annotators in appen to label four types of entities: person name, product, organization, and geopolitical location. the detailed statistic of the dataset labeled by appen is shown in the initial selection criteria for the annotators is that they were required to be fluent in english. moreover, the annotators had to pass a screening test where they were given some sample utterances to annotate the named entities. based on their performance in the screening test, they were selected for the annotation job to ensure better quality for data annotation. ## proposed active learning framework suppose, there is a sequence @xmath0, s 2 , ..., s n containing n words. for each token s i , the sequence tagging model will assign the most relevant tag t j (based on the highest probability score predicted by the model) from a list of m tags @xmath1, t 2 , t 3 , ..., t m . while constructing a sequence tagging dataset (i.e., ner dataset) from our business conversation data, we ask the annotators to annotate each token with the most relevant tag. due to the nature of our dataset, there is a high risk of annotation errors. thus, in this paper, algorithm 1 the active learning framework f olds: {1, 2, 3, 4, 5} reannotationset:{} t : t hreshold_v alue 1: for f old in f olds do 2: p redictionset ← samplep redictiondata() 3: t rainingset ← samplet rainingdata() 4: m odel ← trainm odel(t rainingset) 5: reannotationset.add(utterance) 12: n-fold experiments: given a dataset containing n examples, m fold experiments can be run in the following way to select some samples from the training set that are more likely to contain annotation errors. in each fold, use x% data from the training set as the prediction data (without replacement). also, the prediction set in each fold should only contain distinct examples (i.e., the examples that do not appear in any other fold's prediction set) such that combining all the prediction sets together covers all the training instances. in algorithm 1, we present our framework via demonstrating a 5 fold experiments. the sampled data in each fold will be as follows: 80% data in each fold will contain the training set while 20% data in each fold will contain the prediction set. moreover, the prediction data in each fold should contain those examples only that do not appear in any other prediction set. for model training, we fine-tune a bert-basedcased model on each fold of the training data. in this way, we fine-tune 5 bert-based-cased models on the training data of 5 different folds. then each trained model is utilized to predict the ner tags (p_tag refers the predicted tag in algorithm 1) on the prediction data in their respective folds. finally, we select some instances from the prediction set for re-annotation based on the following method. probability thresholding: we utilize predicted probabilities to select instances from the prediction set for re-annotation. for an entity in an example utterance in the predicted set of the training data, if the predicted tag is p_tag, with the probability score predicted by the ner model for that tag is p_prob_score and the predicted probability score for the gold entity is g_prob_score, then if the probability score difference between p_prob_score and g_prob_score is more than a threshold t (where @xmath2, then we add that utterance in our data re-annotation set. human in the loop: once the data reannotation set is constructed, it can be sent to the annotators for re-labeling. at first, the annotators can decide whether the given tags for an utterance are correct or not. if they think it is incorrect, then they are asked to re-annotate the utterance. in this way, we speed up the annotation process. note that for data re-annotation, labelbox foot_1 was used. ## model architecture we use two models in two stages to run our experiments. in stage 1, we fine-tune a bert-base-cased model @xcite on each fold of the dataset to identify the instances that are more likely to contain annotation errors. after re-annotating those instances, we run another experiment (i.e., stage 2) in the updated version of the training set containing the instances that are selected for re-annotation along with other instances (i.e., the instances that were not selected for reannotation). note that the model trained on stage 2 is the one that is used for production deployment. for this reason, we choose the distilbert-basecased @xcite model for stage 2 since it is more efficient than bert while also being significantly smaller, making it more applicable for industrial scenarios. a general overview of our proposed approach is shown on figure 1 . ## results and analyses we conduct experiments in the original version of the dataset as well as the re-annotated version of the dataset using distilbert. during experiments, we run 5 epochs with the training_batch_size set to 64. the learning_rate was set to 1e -4 with the max_sequence_length being set to 200. we show our experimental results in ## conclusion in this paper, we propose an active learning framework that is very effective to fix the annotation errors in a noisy business conversation data. by sampling only about 6% of the training data for re-annotation, we observe a huge performance gain in terms of precision, recall, and f1. moreover, re-annoating the data using the proposed technique also helps the ner model to better distinguish between product and organization type entities in noisy business conversational data. these findings further validate that our proposed approach is very effective in limited budget scenarios to alleviate the need of human re-labeling of a large amount of noisy data. we also show that a smaller-sized dis-tilbert model can be effectively trained on such data and deployed in a minimum computational resource environment. in the future, we will investigate the performance of our proposed technique on other entity types, as well as on other tasks @xcite similar to ner @xcite containing noisy data.