Inconsistencies in Crowdsourced Slot-Filling Annotations: A Typology and Identification Methods (2020)

Abstract: Slot-filling models in task-driven dialog systems rely on carefully annotated training data. However, annotations by crowd workers are often inconsistent or contain errors. Simple solutions like manually checking annotations or having multiple workers label each sample are expensive and waste effort on samples that are correct. If we can identify inconsistencies, we can focus effort where it is needed. Toward this end, we define six inconsistency types in slot-filling annotations. Using three new noisy crowd-annotated datasets, we show that a wide range of inconsistencies occur and can impact system performance if not addressed. We then introduce automatic methods of identifying inconsistencies. Experiments on our new datasets show that these methods effectively reveal inconsistencies in data, though there is further scope for improvement.

## introduction slot-filling is a key component of task-driven dialog systems, providing a way for systems to extract key properties from user queries. for example, a slot-filling model can extract the tokens "new york" as a to location slot in the query "book a flight to new york". standard slot-filling models train or finetune on large datasets of carefully-annotated data that is domain specific. this means there is an ongoing need for annotating new datasets for new domains. typically, data annotation is completed via small tasks with brief instructions completed by non-expert crowd workers. collecting high quality data in this setting is challenging, involving multiple rounds of pilot annotations and analysis to develop suitable instructions @xcite . figure 1 shows examples of inconsistencies in annotations performed by crowd workers. training on data with these issues will lead to lower quality models, which in turn decrease the effectiveness of the overall dialog system. most research on improving data quality has focused on mechanisms such as aggregation @xcite , worker filtering @xcite , and attention checks @xcite . these all raise costs and primarily address clear inconsistencies (such as in examples 1, 4, 5, and 6) but not more subtle cases like the inclusion of "dollar" in examples 2 and 3. additionally, annotation-asa-service (e.g., scale.ai) is being widely used by developers but often cannot be customized by the developer to add these kinds of mechanisms. if we could identify all of these inconsistencies then we could fix them without expending effort on examples that were correctly annotated and we could improve task instructions to reduce further inconsistencies. in this paper, we catalog a typology of annotation inconsistencies that occur in slot-filling data and present several automatic methods for identifying inconsistencies. to demonstrate our ideas, we introduce and analyze three new crowd-annotated datasets. 1 by analyzing the data with our typology, we find that the distribution of errors varies substantially depending on the domain. we also measure the impact of these inconsistencies on trained models and examine the relative impact of each inconsistency type through a controlled experiment where we artificially inject errors. we also propose several inconsistency identification methods that require no additional annotation, using only the data already being collected. we evaluate by comparing to manually checking every example, measuring effort required and the quality of models trained on the resulting data. the approaches have different strengths and weaknesses. one reduces effort by 16-31% and produces models within 1.6 f 1 . another reduces effort by 50-87%, but leads to weaker models. these results indicate that inconsistency identification is possible, but there is scope to further improve the tradeoff between effort and data quality. this work provides the basis for a new direction in research for addressing inconsistencies in crowdsourcing by defining a typology of inconsistency types, collecting new benchmark datasets, and exploring directions for automatic identification of inconsistencies. ## inconsistencies in this section, we define six inconsistency types, measure their frequency in three new datasets, and evaluate their impact on model performance. we developed the set of inconsistencies by manually inspecting crowd-annotated data and looking for patterns in the common mistakes. ## automatic inconsistency detection addressing the challenge of inconsistent annotations usually involves a combination of two approaches: improving annotation instructions and manual post-processing. developing better instructions is a challenge, as it requires either foreseeing all scenarios where there may be inconsistency in annotation conventions, or looking at enough annotated data to identify inconsistencies across examples. even if comprehensive instructions are developed, there is no guarantee that annotators will follow the conventions perfectly. manual post-processing can address these issues, but without any means of guiding effort to potentially problematic examples it involves a substantial amount of effort. collecting multiple annotations of each sample can reduce these issues, but increases costs and will not resolve all cases as the majority annotation for one example may be inconsistent with the majority for another example. in this section, we explore methods of automatic inconsistency detection. these could be used either to assist in refinement of instructions or to guide post-processing to focus effort. we propose a range of automatic inconsistency detection algorithms that are a first step towards addressing this challenge. the setting we consider is when a new dataset has been annotated by crowd workers with one annotation per example. there is no in-domain gold standard annotated data available and no other data with the same annotation scheme available. this matches the application described in section 1: developing a dialog system for a new domain with its own set of slots. ## conclusion understanding the nature and frequency of different types of annotation inconsistencies in noisy slotfilling corpora is important for the development of robust task-driven dialog systems. this paper presents a typology of inconsistency types. applying the typology to three new crowdsourced datasets, we find the overall inconsistency rate is high, and the slot format and omission types are the most common. we show that correcting inconsistencies improves the quality of models and propose several methods of automatically detecting inconsistencies. by evaluating our approaches we find no clearly dominant choice, and that there is scope for further work to balance detection recall with the overall number of samples flagged as inconsistent. by detecting inconsistencies we can fix issues in annotations, improve task instructions, and better understand the challenges of slot-filling.