Identifying and Resolving Annotation Changes for Natural Language Understanding (2021)

Abstract: Annotation conflict resolution is crucial towards building machine learning models with acceptable performance. Past work on annotation conflict resolution had assumed that data is collected at once, with a fixed set of annotators and fixed annotation guidelines. Moreover, previous work dealt with atomic labeling tasks. In this paper, we address annotation conflict resolution for Natural Language Understanding (NLU), a structured prediction task, in a real-world setting of commercial voice-controlled personal assistants, where (1) regular data collections are needed to support new and existing functionalities, (2) annotation guidelines evolve over time, and (3) the pool of annotators change across data collections. We devise an approach combining information-theoretic measures and a supervised neural model to resolve conflicts in data annotation. We evaluate our approach both intrinsically and extrinsically on a real-world dataset with 3.5M utterances of a commercial dialog system in German. Our approach leads to dramatic improvements over a majority baseline especially in contentious cases. On the NLU task, our approach achieves 2.75% error reduction over a no-resolution baseline.

## introduction supervised learning is ubiquitous as a form of learning in nlp @xcite @xcite , but supervised models require access to high-quality and manually annotated data so that they perform reasonably. it is often assumed that (1) such annotated data is collected once and then used to train and test various models, (2) the pool of annotators is fixed, and (3) annotation guidelines are fixed @xcite @xcite . in real-world nlp applications e.g., voice-controlled assistants such as google home or amazon alexa, such assumptions are unrealistic. the assistant is continuously evolving and extended with new functionalities, and hence, changes to annotation guidelines are frequent. the assistant also needs to adapt to language variations over time, both lexical and semantic. therefore, annotated data needs to be collected regularly i.e., new collections of data at different time points, where the same utterance text can be re-annotated over time. additionally, the set of annotators might change across collections. in this work, we tackle the problem of resolving annotation conflicts in a real-world scenario of a commercial personal assistant. to minimize annotation conflicts, the same data point is often labeled by multiple annotators and the annotation with unanimous agreement, or the one with majority votes is deemed correct @xcite @xcite . while such measures ensure the quality of annotations within the same batch, they cannot ensure it across batches at different time points, particularly when the same data point is present in different batches with inevitable changes to annotation guidelines. for detecting and resolving conflicts, two main methodologies have been explored; bayesian modeling and training a supervised classification model @xcite @xcite @xcite . both methodologies make certain assumptions about the setting, for example, annotation guidelines and the pool of annotators are fixed, which is not the case for our use case. additionally, while bayesian modeling is reasonably efficient for small datasets, it is prohibitively expensive for large-scale datasets with millions of utterances. we adopt a combination of information-theoretic measures and a classification neural model to detect and resolve conflicts. nlu is a key component in language-based applications, and is defined as the combination of: (1) an intent classifier (ic), which classifies an utterance into one of n intent labels (e.g. playmusic), and (2) a slot labeling (sl) model, which classifies tokens into slot types, out of a predefined set (e.g. songname) @xcite . an example utterance is shown in figure 1 , with two conflicting annotations. in this paper, we consider the task of nlu for personal assistants and assume that utterances arrive at different points in time, and that the annotation guideline evolves over time. the same utterance text, e.g., the one shown in figure 1 , often occurs multiple times across collections, which gives the opportunity to conflicting annotations. moreover, changes to the annotation guidelines over time lead to more conflicts. given an nlu dataset with utterances having multiple, possibly conflicting annotations (ic and sl), our goal is to find the right annotation for each such utterance. to this end, we first detect guideline changes using a maximum information gain cut (section 3.3). then we compute the normalized entropy of the remaining annotations after dropping the ones before a guideline change. in case this entropy is low, we simply use majority voting, otherwise, we rely on a classifier neuralbased model to resolve the conflict (section 3.4). our approach is depicted in figure 2 . we evaluate our approach both intrinsically and extrinsically, and show improved performance over baselines including random resolution or no resolution in six domains, as detailed in section 4. ## related work annotation conflicts could emerge due to different reasons, be it imprecision in the annotation guideline @xcite , vagueness in the meaning of the underlying text @xcite @xcite , or annotators being careless or inexperienced @xcite . @xcite report, on the wsj part-of-speech (pos) corpus, that 28.0% of pos tagging errors stem from imprecise annotation guideline that caused inconsistent annotations, while 15.5% of the errors are due to wrong gold standard, which could be attributed to careless or inexperienced annotators. in our case, conflicts could occur due to changes to the annotation guidelines and having different, possibly inexperienced, annotators within and across data collections. past work on conflict resolution has assumed that data is collected once and then used for model training and testing. consequently, the proposed methods to detect and resolve conflicts are geared towards this setting @xcite @xcite @xcite @xcite . in our scenario, we deal with an ever-growing data which is collected across different data collections at different time points. this increases the likelihood of conflicts especially with frequent changes to the annotation guideline. in @xcite , an approach is proposed to automatically detect annotation errors in gold standard annotations for pos tagging using n-gram tag variation i.e., looking at n-grams occurring in the corpus with multiple tagging. bayesian modeling is often used to model how reliable each annotator is and to correct/resolve wrong annotations @xcite . in @xcite , they propose mace, an item-response based model, to identify spammer annotators and to predict the correct underlying labels. applying such models is prohibitively expensive in our case due to the large amount of utterances we deal with. additionally, our annotator pool changes over time. a different line of work has explored resolving conflicts in a supervised classification setting, similar to our approach for resolving high normalized entropy conflicts. @xcite use an ensemble of two off-the-shelf parsers that re-annotate the training set to detect and resolve conflicts in dependency treebanks. @xcite use a similar approach on out-of-domain treebanks. finally, @xcite introduce the inter-annotator agreement loss to ensure consistent annotations for pos tagging. intent classification and slot labeling are two fundamental tasks in spoken language understanding, dating back to early 90's @xcite . with the rise of task-oriented personal assistants, the two tasks got more attention and progress has been made by applying various deep learning techniques @xcite ; given conflicting annotations, we first use the max information gain (ig) cut to detect changes in annotation guidelines. then, low entropy conflicts are resolved using majority voting. high entropy conflicts are resolved using a classifier lstm-based model. @xcite @xcite . while we focus on resolving annotation conflicts for nlu with linear labeling i.e., intent and slot labels, our approach can be still used for other more complex tree-based labeling e.g., labeling dependency parses or ontology trees @xcite , with the minor change of replacing the task-specific neural lstm-based classification model. we plan to investigate this in the future. ## experiments in this section we evaluate our method both intrinsically and extrinsically. ## conclusion in this paper, we tackled the problem of annotation conflicts for the task of nlu for voice-controlled personal assistants. we presented a novel approach that combines information-theoretic measures and an lstm-based neural model. we evaluated our method on a real-world large-scale dataset, both intrinsically and extrinsically. although we focused on the task of nlu, our conflict resolution pipeline could be applied to any manual annotation task. in the future, we plan on investigating how the choice of the task-specific classification model affects performance. moreover, we plan to study annotation conflict resolution for other nlp tasks e.g., pos tagging and dependency parsing.