Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labeling (2020)

Abstract: Sequence labeling is a fundamental task for a range of natural language processing problems. When used in practice, its performance is largely influenced by the annotation quality and quantity, and meanwhile, obtaining ground truth labels is often costly. In many cases, ground truth labels do not exist, but noisy annotations or annotations from different domains are accessible. In this paper, we propose a novel framework Consensus Network (ConNet) that can be trained on annotations from multiple sources (e.g., crowd annotation, cross-domain data). It learns individual representation for every source and dynamically aggregates source-specific knowledge by a context-aware attention module. Finally, it leads to a model reflecting the agreement (consensus) among multiple sources. We evaluate the proposed framework in two practical settings of multi-source learning: learning with crowd annotations and unsupervised cross-domain model adaptation. Extensive experimental results show that our model achieves significant improvements over existing methods in both settings. We also demonstrate that the method can apply to various tasks and cope with different encoders.

## introduction sequence labeling is a general approach encompassing various natural language processing (nlp) tasks including part-of-speech (pos) tagging @xcite , word segmentation @xcite , and named entity recognition (ner) @xcite . typically, existing methods follow the supervised learning paradigm, and require high-quality annotations. while gold standard annotation is expensive and time-consuming, imperfect annotations are relatively easier to obtain from crowdsourcing (noisy labels) or other domains (out-of-domain). despite their low cost, such supervision usually can be obtained from different sources, and it has been shown that multi-source weak supervision has the potential to perform similar to gold annotations @xcite . specifically, we are interested in two scenarios: 1) learning with crowd annotations and 2) unsupervised cross-domain model adaptation. both situations suffer from imperfect annotations, and benefit from multiple sources. therefore, the key challenge here is to aggregate multi-source imperfect annotations for learning a model without knowing the underlying ground truth label sequences in the target domain. our intuition mainly comes from the phenomenon that different sources of supervision have different strengths and are more proficient with distinct situations. therefore they may not keep consistent importance during aggregating supervisions, and aggregating multiple sources for a specific input should be a dynamic process that depends on the sentence context. to better model this nature, we need to (1) explicitly model the unique traits of different sources when training and (2) find best suitable sources for generalizing the learned model on unseen sentences. in this paper, we propose a novel framework, named consensus network (connet), for sequence labeling with multi-source supervisions. we represent the annotation patterns as different biases of annotators over a shared behavior pattern. both annotator-invariant patterns and annotator-specific biases are modeled in a decoupled way. the first term comes through sharing part of low-level model parameters in a multi-task learning schema. for learning the biases, we decouple them from the model as the transformations on top-level tagging model parameters, such that they can capture the unique strength of each annotator. with such decoupled source representations, we further learn an attention network for dynamically assigning the best sources for every unseen sentence through composing a transformation that represents the agreement among sources (consensus). extensive experimental results in two scenarios show that our model outperforms strong baseline methods, on various tasks and with different encoders. connet achieves state-of-the-art performance on real-world crowdsourcing datasets and improves significantly in unsupervised crossdomain adaptation tasks over existing works. ## related work there exists three threads of related work with this paper, which are sequence labeling, crowdsourcing and unsupervised domain adaptation. neural sequence labeling. traditional approaches for sequence labeling usually need significant efforts in feature engineering for graphical models like conditional random fields (crfs) @xcite . recent research efforts in neural network models have shown that end-to-end learning like convolutional neural networks (cnns) @xcite or bidirectional long short-term memory (blstms) @xcite can largely eliminate humancrafted features. blstm-crf models have achieved promising performance @xcite and are used as our base sequence tagging model in this paper. crowd-sourced annotation. crowd-sourcing has been demonstrated to be an effective way of fulfilling the label consumption of neural models @xcite . it collects annotations with lower costs and a higher speed from non-expert contributors but suffers from some degradation in quality. @xcite proposes the pioneering work to aggregate crowd annotations to estimate true labels, and @xcite shows its effectiveness with amazon's mechanical turk system. later works @xcite @xcite focus on expectation-maximization (em) algorithms to jointly learn the model and annotator behavior on classification. recent research shows the strength of multitask framework in semi-supervised learning @xcite , cross-type learning @xcite , and learning with entity triggers @xcite . @xcite ; @xcite ; @xcite regards crowd annotations as noisy gold labels and constructs crowd components to model annotator-specific bias which were discarded during the inference process. it is worth mentioning that, it has been found even for human curated annotations, there exists certain label noise that hinders the model performance @xcite . ## multi-source supervised learning we formulate the multi-source sequence labeling problem as follows. given k sources of supervision, we regard each source as an imperfect annotator (non-expert human tagger or models trained in related domains). for the k-th source data set the tag sequence of the sentence is marked as y (k) @xmath0,j }. we define the sentence set of each annotators as application i: learning with crowd annotations. when learning with crowd-sourced data, we regard each worker as an imperfect annotator (s (k) ), who may make mistakes or skip sentences in its annotations. note that in this setting, different annotators tag subsets of the same given dataset (x ), and thus we assume there are no input distribution shifts among x (k) . also, we only test sentences in the same domain such that the distribution in target corpus t is the same as well. that is, the marginal distribution of target corpus p t (x) is the same with that for each individual source dataset, i.e. p t (x) = p k (x). however, due to imperfectness of the annotations in each source, p k (y|x) is shifted from the underlying truth p (y|x) (illustrated in the top-left part of fig. 1 ). the multi-source learning objective here is to learn a model p t (y|x) for supporting infer-ence on any new sentences in the same domain. application ii: unsupervised cross-domain model adaptation. we assume there are available annotations in several source domains, but not in an unseen target domain. we assume that the input distributions p (x) in different source domains x (k) vary a lot, and such annotations can hardly be adapted for training a target domain model. that is, the prediction distribution of each domain model (p k (y|x)) is close to the underlying truth distribution (p (y|x)) only when @xmath1. for target corpus sentences @xmath2, such a source model p k (y|x) again differs from underlying ground truth for the target domain p t (y|x) and can be seen as an imperfect annotators. our objective in this setting is also to jointly model p t (y, x) while noticing that there are significant domain shifts between t and any other x (k) . ## consensus network in this section, we present our two-phase framework connet for multi-source sequence labeling. as shown in figure 2 , our proposed framework first uses a multi-task learning schema with a special objective to decouple annotator representations as different parameters of a transformation around crf layers. this decoupling phase (section 4.2) is for decoupling the model parameters into a set of annotator-invariant model parameters and a set of annotator-specific representations. secondly, the dynamic aggregation phase (section 4.3) learns to contextually utilize the annotator representations with a lightweight attention mechanism to find the best suitable transformation for each sentence, so that the model can achieve a context-aware consensus among all sources. the inference process is described in section 4.4. ## experiments we evaluate connet in the two aforementioned settings of multi-source learning: learning with crowd annotations and unsupervised cross-domain model adaptation. additionally, to demonstrate the generalization of our framework, we also test our method on sequence labeling with transformer encoder in appendix b and text classification with mlp encoder in section 5.5. ## conclusion in this paper, we present connet for learning a sequence tagger from multi-source supervision. it could be applied in two practical scenarios: learning with crowd annotations and cross-domain adaptation. in contrast to prior works, connet learns fine-grained representations of each source which are further dynamically aggregated for every unseen sentence in the target data. experiments show that our model is superior to previous crowd-sourcing and unsupervised domain adaptation sequence labeling models. the proposed learning framework also shows promising results on other nlp tasks like text classification.