O pt SLA : an Optimization-Based Approach for Sequential Label Aggregation (2020)

Abstract: The need for the annotated training dataset on which data-hungry machine learning algorithms feed has increased dramatically with advanced acclaim of machine learning applications. To annotate the data, people with domain expertise are needed, but they are seldom available and expensive to hire. This has lead to the thriving of crowdsourcing platforms such as Amazon Mechanical Turk (AMT). However, the annotations provided by one worker cannot be used directly to train the model due to the lack of expertise. Existing literature in annotation aggregation focuses on binary and multi-choice problems. In contrast, little work has been done on complex tasks such as sequence labeling with imbalanced classes, a ubiquitous task in Natural Language Processing (NLP), and Bio-Informatics. We propose OptSLA, an Optimization-based Sequential Label Aggregation method, that jointly considers the characteristics of sequential labeling tasks, workers reliabilities, and advanced deep learning techniques to conquer the challenge. We evaluate our model on crowdsourced data for named entity recognition task. Our results show that the proposed OptSLA outperforms the state-of-the-art aggregation methods, and the results are easier to interpret.

## introduction crowdsourcing @xcite is a popular platform to annotate massive corpora inexpensively. it has bred lots of interest in machine learning and deep learning tasks. however, when workers provide annotations, the results may be noisier comparing with labels provided by experts. thus, it becomes essential to conduct truth inference from the noisy annotations. one common annotation aggregation approach is majority voting (mv) @xcite , in which annotation with the highest number of occurrences is deemed as truth. another naive approach is to regard an annotation as correct if a certain number of workers provide the same annotation. the concern with these methods is that they assume all workers are of the same quality, which is usually invalid in practice. in this paper, we study the annotation aggregation problem for sequential labeling tasks, a common nlp task. many existing crowdsourcing label aggregation methods may suffer from performance loss because they assume that data instances are independent @xcite . new approaches are recently proposed to handle the particular characteristics of sequential labeling tasks, where tokens in one sentence have complex dependencies @xcite @xcite . in this line of approaches, probabilistic models are adopted to model the workers' labeling behavior and to model the dependencies between adjacent tokens. there are some drawbacks to the probabilistic models. first, they have strong statistical assumptions when modeling the sequence annotations, limiting the flexibility of the models. second, these models need to infer complex parameters, making it hard to interpret the relations between worker's quality and token's true labels. third, these aggregation methods can not fully unleash the power of deep learning in sequential labeling tasks. to address these challenges, we propose an optimization framework to improve aggregation performance. our method optsla estimates workers' reliability and models the label dependencies to infer the true labels from noisy annotations. opt-sla handles complex sequential label aggregation problem with fewer parameters comparing the state-of-the-art and produces easy-to-understand results. we further incorporate the state-of-the-art deep learning approach into optsla, where the deep learning component and the aggregation component can maturely enhance each other. to ensure high-quality training data, optsla chooses sentences with high confidence from the aggregation component. the deep learning model is incrementally trained with the iteratively updated aggregation results. the three models mentioned above are probabilistic models with significantly more parameters to tune and are harder to interpret than optimizationbased methods @xcite . moreover, the existing methods do not fully unleash the power of deep learning approaches in sequential labeling tasks. in this paper, we propose an optimizationbased aggregation method to address the interpretability challenge, and further include the deep learning module to boost the performance. ## methodology the sequential label aggregation task aims to combine the annotations provided by different workers to infer the ground truth sequential labels. in this section, we describe our approach, an optimization-based sequential label aggregation method (optsla), which aggregates multiple workers' annotations with deep learning results by estimating the reliability of workers and modeling the dependencies among tokens in the sentences. ## conclusion and future works in this paper, we propose an innovative optimization-based approach optsla for sequential label aggregation problem. our model jointly considers different factors in the objective function, including the workers' annotations, workers' reliability, the deep learning model, and the characteristics of sequential labeling tasks. our experimental results illustrate that optsla outperforms the state-of-the-art sequential label aggregations methods, such as crf-ma, hmm-crowd, and bayesian sequence combination (bsc) in terms of f1 score. for the future work, we will evaluate more factors such as the task assignment that may affect the aggregation performance from the deep learning model and the workers' behaviors.