Improving Human-Labeled Data through Dynamic Automatic Conflict Resolution (2020)

Abstract: This paper develops and implements a scalable methodology for (a) estimating the noisiness of labels produced by a typical crowdsourcing semantic annotation task, and (b) reducing the resulting error of the labeling process by as much as 20-30% in comparison to other common labeling strategies. Importantly, this new approach to the labeling process, which we name Dynamic Automatic Conflict Resolution (DACR), does not require a ground truth dataset and is instead based on inter-project annotation inconsistencies. This makes DACR not only more accurate but also available to a broad range of labeling tasks. In what follows we present results from a text classification task performed at scale for a commercial personal assistant, and evaluate the inherent ambiguity uncovered by this annotation strategy as compared to other common labeling strategies.

## introduction ground truth datasets for supervised learning are commonly collected through crowdsourcing annotation projects, either on public platforms (e.g. yandex.toloka, amazon mechanical turk, figure eight ) or on in-house counterparts. in both scenarios, the annotation project is conducted by first providing workers with annotation guidelines and then assigning each one with a set of annotation requests. each unique annotation request is usually assigned to more than one worker, to increase the reliability of the results. subsequently, some post-processing technique is used to create a consensus label that is recorded as the ground truth. however, since human annotations are intrinsically noisy and subject to the influence of external factors, such a process is unlikely to produce accurate ground truth labels for 100% of the requests. moreover, the practice of using a singular consensus label for multi-annotated requests mandates a one and only correct interpretation for every request in any given task, effectively denying the existence of potential ambiguities. this implicit denial of ambiguity, while acceptable for certain tasks such as arithmetics, can be problematic in tasks such as natural language understanding @xcite . perhaps because of these difficulties, estimating the latent accuracy of any annotation process has been a relatively under-examined subject. in this paper, we first introduce a scalable methodology to estimate the latent inaccuracy of any crowdsourcing annotation process. the estimation method does not require any preexisting ground truth dataset, and is therefore immune to the gold question inference type of attacks described by @xcite . we construct an agent-based model to examine the impact of different annotation strategies on overall annotation accuracy. inspired by the findings of our simulation experiments, we propose and implement a novel process of dynamic automatic conflict resolution (dacr), designed to improve the accuracy of the annotation output with limited incremental workload while collecting sufficient annotation input for requests with ambiguity. we present annotation results from a dacr project used in the context of a text classification task, implemented at scale for a commercial personal assistant. the primary contributions of this paper are: 1. latent inaccuracy estimation methodology: we propose a simple, robust, scalable methodology and theoretical framework to study the latent inaccuracies of any crowdsourcing annotation process; ## related work this work relates to recent developments in three areas of research: (1) efficiency and accuracy of human annotation (in particular, inter-annotator agreement), (2) quality assurance in crowdsourcing, and (3) inherent ambiguity in natural language annotation tasks. we survey these in turn below. ## latent inaccuracy estimation method (liem) this section presents a mathematical formula for measuring the inaccuracy of a human annotation task. ## a look into different annotation strategies with this result in hand, we examine three commonly used annotation strategies and evaluate their respective performances through agent based simulations. we compare these results to our novel methodology, dynamic automatic conflict resolution (dacr), which we introduce immediately below. we establish that dacr outperforms other common approaches to human annotation. in a subsequent section we will show that dacr indeed outperforms these other approaches in an actual annotation task. ## exploring ambiguity with dacr as a part of a greater effort to build an intelligent virtual assistant, we have long been working on a text classification task. the text classifier predicts user intent for a given utterance according to a predetermined taxonomy, as in user utterance intent class davos local time time status for aa 166 flight compute 37 times 30 math kesha album rainbow music ## conclusion in this paper, we have introduced a method to estimate the latent inaccuracy of any crowdsourcing annotation process and shown its validity both via formal analysis and simulation experiments. we also investigated the performances and efficiencies of different annotation strategies with an agent-based model. based on our findings, we further propose a novel annotation strategy that dynamically allocates additional annotation resources to difficult (or ambiguous) requests; it is found to be able to preserve multiple conflicting grades for ambiguous requests and outperform comparable annotation strategies while requiring significantly less annotation resources. we share insights from a dacr-based annotation project on a classification task and discover patterns among the in-conflict examples that corroborate the existence of ambiguity in the given task. our goal is to allow the practitioners in the field of machine learning to easily estimate the latent accuracy of any crowdsourcing annotation process. we further hope many will find dacr to be a better alternative to their existing annotation strategies and start thinking about how we should deal with inherent ambiguity.