Crowdsourcing Learning as Domain Adaptation: A Case Study on Named Entity Recognition (2021)

Abstract: Crowdsourcing is regarded as one prospective solution for effective supervised learning, aiming to build large-scale annotated training data by crowd workers. Previous studies focus on reducing the influences from the noises of the crowdsourced annotations for supervised models. We take a different point in this work, regarding all crowdsourced annotations as gold-standard with respect to the individual annotators. In this way, we find that crowdsourcing could be highly similar to domain adaptation, and then the recent advances of cross-domain methods can be almost directly applied to crowdsourcing. Here we take named entity recognition (NER) as a study case, suggesting an annotator-aware representation learning model that inspired by the domain adaptation methods which attempt to capture effective domain-aware features. We investigate both unsupervised and supervised crowdsourcing learning, assuming that no or only small-scale expert annotations are available. Experimental results on a benchmark crowdsourced NER dataset show that our method is highly effective, leading to a new state-of-the-art performance. In addition, under the supervised setting, we can achieve impressive performance gains with only a very small scale of expert annotations.

## introduction crowdsourcing has gained a growing interest in the natural language processing (nlp) community, which helps hard nlp tasks such as named entity recognition @xcite , part-of-speech tagging @xcite , relation extraction @xcite , translation @xcite , argument retrieval @xcite , and others @xcite to collect a large scale dataset for supervised model training. in contrast to the gold-standard annotations labeled by experts, the crowdsourced annotations can be constructed quickly at a low cost with masses of crowd annotators @xcite . however, these annotations are relatively lower-quality with much-unexpected noise since the crowd annotators are not professional enough, which can make errors in complex and ambiguous contexts @xcite . previous crowdsourcing learning models struggle to reduce the influences of noises of the crowdsourced annotations @xcite @xcite . majority voting (mv) is one straightforward way to aggregate high-quality annotations, which has been widely adopted @xcite @xcite , but it requires multiple annotations for a given input. recently, the majority of models concentrate on monitoring the distances between crowdsourced and gold-standard annotations, obtaining better performances than mv by considering the annotator information together @xcite @xcite . most of these studies assume the crowdsourced annotations as untrustworthy answers, proposing sophisticated strategies to recover the golden answers from crowdsourced labels. in this work, we take a different view for crowdsourcing learning, regarding the crowdsourced annotations as the gold standard in terms of individual annotators. in other words, we assume that all annotators (including experts) own their specialized understandings towards a specific task, and they annotate the task consistently according to their individual principles by the understandings, where the experts can reach an oracle principle by consensus. the above view indicates that crowdsourcing learning aims to train a model based on the understandings of crowd annotators, and then test the model by the oracle understanding from experts. based on the assumption, we find that crowdsourcing learning is highly similar to domain adaptation, which is one important topic that has been investigated extensively for decades @xcite @xcite . we treat each annotator as one domain specifically, and then crowdsourcing learning is essentially almost a multi-source domain adaptation problem. thus, one natural question arises: what is the performance when a state-of-the-art domain adaptation model is applied directly to crowdsourcing learning. here we take ner as a study case to investigate crowdsourcing learning as domain adaptation, considering that ner has been one popular task for crowdsourcing learning in the nlp community @xcite @xcite . we suggest a state-of-the-art representation learning model that can effectively capture annotator(domain)-aware features. also, we investigate two settings of crowdsourcing learning, one being the unsupervised setting with no expert annotation, which has been widely studied before, and the other being the supervised setting where a certain scale of expert annotations exists, which is inspired by domain adaptation. finally, we conduct experiments on a benchmark crowdsourcing ner dataset @xcite to evaluate our methods. we take a standard bilstm-crf @xcite model with bert @xcite word representations as the baseline, and adapt it to our representation learning model. experimental results show that our method is able to model crowdsourced annotations effectively. under the unsupervised setting, our model can give a strong performance, outperforming previous work significantly. in addition, the model performance can be greatly boosted by feeding with small-scale expert annotations, which can be a prospective direction for low-resource scenarios. ## the basic idea here we describe the concepts of the domain adaptation and crowdsourcing learning in detail, and show how they are connected together. ## a case study on ner in this section, we take ner as a case study, which has been investigated most frequently in nlp @xcite , and propose a representation learning model mainly inspired by the domain adaptation model of @xcite to perform crowdsourcing learning. in addition, we introduce the unsupervised and supervised settings for crowdsourcing learning which are directly borrowed from the domain adaptation. ## conclusion and future work we studied the connection between crowdsourcing learning and domain adaptation, and then proposed to treat crowdsourcing learning as a domain adaptation problem. following, we took ner as a case study, suggesting a representation learning model from recent advances of domain adaptation for crowdsourcing learning. by this case study, we introduced unsupervised and supervised crowdsourcing learning, where the former is a widelystudied setting while the latter has been seldom investigated. finally, we conducted experiments on a widely-adopted benchmark dataset for crowdsourcing ner, and the results show that our representation learning model is highly effective in unsupervised learning, achieving the best performance in the literature. in addition, the supervised learning with a very small scale of expert annotations can boost the performance significantly. our work sheds light on the application of effective domain adaptation models on crowdsourcing learning. there are still many other sophisticated cross-domain models, such as adversarial learning @xcite and self-training @xcite . future work may include how to apply these advances to crowdsourcing learning properly. model text and entities unsupervised mv pace, a junior, helped [ohio state]loc to a 10-1 record and a berth in the rose bowl against [arizona]org state. lc-cat pace, a junior, helped [ohio state]org to a 10-1 record and a berth in the [rose bowl]misc against [arizona]org state. this work pace, a junior, helped [ohio state]org to a 10-1 record and a berth in the [rose bowl]misc against [arizona state]org. supervised (25%) mv pace, a junior, helped [ohio state]loc to a 10-1 record and a berth in the [rose bowl]misc against [arizona state]loc . gold [pace]p er, a junior, helped [ohio state]org to a 10-1 record and a berth in the [rose bowl]misc against [arizona]org state. lc-cat pace, a junior, helped [ohio state]org to a 10-1 record and a berth in the [rose bowl]misc against [arizona state]loc . this work [pace]p er, a junior, helped [ohio state]org to a 10-1 record and a berth in the [rose bowl]misc against [arizona state]org. ground-truth [pace]p er, a junior, helped [ohio state]org to a 10-1 record and a berth in the [rose bowl]misc against [arizona state]org. we exploit the stochastic gradient-based online learning, with a batch size of 64, to optimize model parameters. we apply the time-step dropout, which randomly sets several representations in the sequence to zeros with a probability of 0.2, on the word representations to avoid overfitting. we use the adam algorithm to update the parameters with a constant learning rate 1 × 10 -3 , and apply the gradient clipping by a maximum value of 5.0 to avoid gradient explosion.