It is better to Verify: Semi-Supervised Learning with a human in the loop for large-scale NLU models (2021)

Abstract: When a NLU model is updated, new utter- ances must be annotated to be included for training. However, manual annotation is very costly. We evaluate a semi-supervised learning workflow with a human in the loop in a produc- tion environment. The previous NLU model predicts the annotation of the new utterances, a human then reviews the predicted annotation. Only when the NLU prediction is assessed as incorrect the utterance is sent for human anno- tation. Experimental results show that the pro- posed workflow boosts the performance of the NLU model while significantly reducing the annotation volume. Specifically, in our setup, we see improvements of up to 14.16% for a recall-based metric and up to 9.57% for a F1- score based metric, while reducing the annota- tion volume by 97% and overall cost by 60% for each iteration.

## introduction natural language understanding (nlu) models are a key component of task-oriented dialog systems such as as amazon alexa or google assistant which have gained more popularity in recent years. to improve their performance and extend their functionalities, new versions of the nlu model are released to customers on a regular basis. in the classical supervised learning approach, new training data between model updates is acquired by sampling utterances from live traffic and have them annotated by humans. the main drawback is the high cost of manual annotation. we refer to this conventional workflow as human annotation workflow. in this paper, we propose a new workflow with the aim to reduce the annotation cost while still maintaining high quality nlu models. we refer to it as the human verification workflow. the proposed workflow uses the previous (current) version of the nlu model to annotate the new training data before each model update. the predicted annotation produced by the nlu model, which we refer to as nlu hypothesis or interpretation, is then reviewed by humans. if the nlu hypothesis is assessed as correct, the nlu hypothesis is used as the ground-truth annotation of the utterance during training. if the nlu hypothesis is assessed as incorrect, the utterance is sent for human annotation before being ingested for training. with the proposed workflow, only utterances for which the hypothesis of the nlu model was assessed as incorrect are annotated by humans, thereby reducing the annotation volume drastically. since verifying is faster and cheaper than annotating, a cost reduction is achieved. we investigate the adoption of this workflow once the system has reached a certain maturity, not from the start. while these two workflows would provide the same annotation for any utterance in an ideal world, the results may differ in the real world depending on the presence of annotation or verification errors. in this paper, we would like to answer the following fundamental question: in terms of human annotation errors, human verification errors and model performance, is it better to manually verify or annotate in order to iteratively update nlu systems? to answer this question, we investigate the impact of human annotation vs. verification in a large scale nlu system. to this end, we consider two model architectures utilized for nlu models in the current production systems, a conditional random field (crf) @xcite for slot filling and a maximum entropy (maxent) classifier @xcite for intent classification as well as a transformer based bert architecture @xcite . we evaluate the proposed workflow both explicitly by measuring annotation quality as well as implicitly by comparing the resulting model performance. our experimental results show that the human verification workflow boosts the model performance while reducing human annotation volumes. in addition, we show that human annotation resources are better spent on utterances selected through active @xcite @xcite . ## related work using a model to label data instead of humans is an approach that has been studied extensively since human labelling is costly while unlabelled data can be acquired easily. under the term semisupervised learning (ssl) @xcite many different approaches to leverage unlabelled data emerged in the literature. ssl aims at exploiting unlabelled data based on a small set of labelled data. one approach is self-training, also referred to as self-teaching or bootstrapping @xcite . in self-training labels are generated by feeding the unlabelled data in a model trained on the the available labelled data. typically, the predicted labels for instances with high confidence are then used to retrain the model and the procedure is repeated. for neural networks, @xcite suggested pseudo-labelling which optimizes a combination of supervised and unsupervised loss instead of retraining the model on pseudo-labels. self-training has been applied to several natural language processing tasks. to name only a few examples, yarowsky (1995) uses selftraining for word sense disambiguation, @xcite to identify subjective nouns. in @xcite self learning is used for parsing. the two main drawbacks of self-training are that instances with low confidence scores cannot be labelled and that prediction errors with high confidence can reinforce itself. to mitigate the latter issue strategies to identify mis-labeled instances have been discussed. an exhaustive review is beyond the scope of this paper, we just name a few examples. @xcite use local information in a neighborhood graph to identify unreliable labels, @xcite add a distance based uncertainty weight for each sample and propose min-max features for better between-class separability and within-class compactness. in this paper we suggest to use human verification to ensure the ingested predicted labels are reliable. in addition, we rely on human annotation for those utterances that the model cannot interpret correctly. the goal is to mitigate the two afore-mentioned problems of self-training. a so called human-in-the-loop approach has been investigated for different applications. @xcite investigate a human-in-the-loop approach for image segmentation and annotation. @xcite examine the use of suggestion models to support human experts with seg-mentation and classification of epistemic activities in diagnostic reasoning texts. @xcite suggest active learning from weak and strong labelers where these labelers can be humans with different levels of expertise in the labelling task. @xcite show that a human expert is not always needed but that user behavior is valuable feedback that can be collected more easily. the contribution of this paper is two-fold: first, we propose a ssl approach with a human in the loop for large-scale nlu models. second, we show this workflow boosts the performance in a production system while reducing human annotation significantly. active learning (al) @xcite @xcite proposes to label those instances that promise the highest learning effect for the model instead of blindly labelling data. since the proposed workflow reduces the human annotation volume, we spend some of these freed up resources on annotation of al data. ## setup and approach in this section, we briefly discuss the nlu model, the used metrics, the concept of iterative model updates and evaluation. ## datasets for training, we start with a dataset of unlabelled utterances representative of the user engagement with a dialog system. the dataset spans over a large number of intent and slots representative of multiple functionalities. high level statistics are listed in in order to have the same annotation and verification quality as in the production system, we requested the support from professional annotators. trained and experienced annotators mimicked both workflows. for each utterance, one annotator of the team followed the human annotation workflow, while another followed the human verification workflow. for each training utterance, we also have the corresponding nlu hypothesis from the production model when the utterance was sampled. as a result two labelled training datasets were generated from one unlabelled dataset following each workflow. the overall training dataset has been built over multiple nlu releases as explained in section 3.3. the two training sets are then used to re-train or fine-tune each of the considered architecture. as a test set, we also consider a dataset of utterances representative of the engagement of the users with a voice assistant (see ## experiments this section describes the conducted experiments to evaluate both workflows and provides more details about how we selected utterances for annotation through al. ## results this section discusses all obtained results. we first evaluate the annotation quality for both workflows and quantify the possible cost reduction for the proposed workflow, see sections 7.1 and 7.2. second, we compare the performance of the nlu models when trained on data labeled through the respective workflow. results are shown in section 7.3. ## conclusion with the aim of reducing annotation costs, we test a methodology where mature nlu models are iteratively updated by ingesting labelled data via a human verification instead of a human annotation workflow. our findings show that the proposed verification workflow not only cuts annotation costs by almost 60 %, but it also boosts the performance of the nlu system for both considered architectures. this is in line with the annotation quality evaluation we performed, where we found that the human error rate for verification is lower than the human error rate for annotation yielding more consistent training data in the former. our findings have an important practical implication: verifying is better than annotating for mature systems. moreover, a fraction of the annotation savings should be utilized to annotate more impactful data, for instance al data, which generated a large performance gain in the proposed workflow with a minimal increase in annotation volume.