L EARNING TO C ONTEXTUALLY A GGREGATE M ULTIS OURCE S UPERVISION FOR S EQUENCE L ABELING
Anonymous authors
Paper under double-blind review
A BSTRACT
Sequence labeling is a fundamental framework for various natural language processing problems including part-of-speech tagging and named entity recognition.
Its performance is largely inﬂuenced by the annotation quality and quantity in supervised learning scenarios.
In many cases, ground truth labels are costly and time-consuming to collect or even non-existent, while imperfect ones could be easily accessed or transferred from different domains.
A typical example is crowdsourced datasets which have multiple annotations for each sentence which may be noisy or incomplete.
Additionally, predictions from multiple source models in transfer learning can be seen as a case of multi-source supervision.
In this paper, we propose a novel framework named Consensus Network (C ON N ET ) to conduct training with imperfect annotations from multiple sources .
It learns the representation for every weak supervision source and dynamically aggregates them by a context-aware attention mechanism.
Finally, it leads to a model reﬂecting the consensus among multiple sources.
We evaluate the proposed framework in two practical settings of multi-source learning: learning with crowd annotations and unsupervised cross-domain model adaptation.
Extensive experimental results show that our model achieves signiﬁcant improvements over existing methods in both settings.
I NTRODUCTION
Sequence labeling is a fundamental framework for various natural language processing (NLP) tasks including part-of-speech (POS) tagging (Ratnaparkhi, 1996), noun phrase chunking (Sang & Buchholz, 2000), word segmentation (Low et al., 2005), and named entity recognition (NER) (Nadeau & Sekine, 2007).
Typically, existing methods follow the supervised learning paradigm, and require high-quality annotations.
While ground truth labels are expensive and time-consuming, imperfect annotations are much easier to obtain from crowdsourcing (noisy labels) or other domains (out-ofdomain).
To alleviate or even address the problem of noise and incompleteness, it is important and beneﬁcial to learn from multiple sources.
Speciﬁcally, we are interested in two typical application scenarios: 1) learning with crowd annotations and 2) unsupervised cross-domain model adaptation , which are detailed in Sec.
and
Sec. 3.
The key challenge of learning with multi-source supervision is to aggregate annotators for learning a model without knowing the underlying ground truth label sequences in the target domain.
Many attempts have been made in generalizing multiple informative sources to an out-of-domain distribution for a range of tasks, including multi-class classiﬁcation (Sheshadri & Lease, 2013), object detection (Su et al., 2012) and information extraction (Liu et al., 2017).
However, most of the prior works (Nguyen et al., 2017; Wang et al., 2019; Peng & Dredze, 2016; Yang et al., 2017; Chen & Cardie, 2018) choose to use simple heuristic-based methods of aggregating source models for performing on a target corpus without carefully calibration.
Our intuition is mainly from the common phenomenon that each source of supervision has distinct strength in different inputs, and thus they should not keep consistent importance in aggregating supervisions when inferring tag sequences for new inputs.
Aggregating multiple sources for a speciﬁc input should be a dynamic process depending on the sentence context rather than a ﬁxed way.
To better model this nature, we need to (1) explicitly model the unique traits of different sources when training and (2) ﬁnd best suitable sources for generalizing the learned model on unseen sentences.
In this paper, we propose a novel framework, named Consensus Network (C ON N ET ), for sequence labeling with multi-source supervisions.
We represent the annotation patterns as different biases of annotators over a shared behavior pattern.
Both annotator-invariant patterns and annotator-speciﬁc biases are modeled in a decoupled way.
The ﬁrst term through sharing part of low-level model parameters in a multi-task learning schema.
For learning the biases, we decouple them from the model as the transformations on top-level tagging model parameters, such that they can capture the unique strength of each annotator.
With such decoupled source representations, we further learn an attention network for dynamically assigning the best sources for every unseen sentence through composing a transformation that represents the consensus.
Extensive experimental results in two scenarios show that our model always outperforms strong baseline methods.
C ON N ET achieves the state-of-the-art performance on real-world crowdsourcing datasets and improve signiﬁcantly in most unsupervised cross-domain adaptation tasks over existing works.
In addition to sequence labeling, it also shows its effectiveness on text classiﬁcation tasks.
R ELATED W ORK
Neural Sequence Labeling.
Traditional approaches for sequence labeling usually need signiﬁcant efforts in feature engineering for graphical models like hidden markov models (HMMs) (Rabiner, 1989) and conditional random ﬁelds (CRFs) (Lafferty, 2001).
Recent research efforts in neural network models have shown that end-to-end learning like convolutional neural networks (CNNs) (Ma &
Hovy, 2016b) or bidirectional long short-term memory (BLSTMs) (Lample et al., 2016) can largely eliminate human-crafted features.
Together with a ﬁnal CRF layer, these BLSTM-CRF models have achieved promising performance and are used as our base sequence tagging model in this paper.
Crowd-sourced Annotation.
Crowd-sourcing has been demonstrated to be an effective way of fulﬁlling the label consumption of neural models (Guan et al., 2017).
It collects annotations with lower costs and a higher speed by non-expert contributors but suffers from some degradation in quality.
Dawid & Skene (1979) proposes the pioneering work to aggregate crowd annotations to estimate true labels, and Snow et al.
(2008) shows its effectiveness with Amazon’s Mechanical
Turk system.
Later works (Dempster et al., 1977; Dredze et al., 2009; Raykar et al., 2010) focus on Expectation-Maximization (EM) algorithms to jointly learn the model and annotator behavior on classiﬁcation problems.
Recent research shows the strength of multi-task framework in semisupervised learning (Lan et al., 2018; Clark et al., 2018), and cross-type learning (Wang et al., 2018).
Nguyen et al.
(2017) and
Rodrigues & Pereira (2018) regards crowd annotations as noisy versions of glod labels and constructs crowd components to model annotator-speciﬁc bias which were discarded during the inference process.
Unsupervised Domain Adaptation.
Unsupervised cross-domain adaptation aims to transfer knowledge learned from high-resource domains (source domains) to boost performance on low-resource domains (target domains).
Different from supervised adaptation (Lin & Lu, 2018), we assume there is no labels at all for target corpora.
Saito et al.
(2017) and Ruder & Plank (2018)
explored bootstrapping with multi-task tri-training approach.
The method is developed for one-to-one domain adaptation and does not model the differences among multiple source domains.
Yang & Eisenstein (2015) represents each domain with a vector of metadata domain attributes and uses domain vectors to train the model to deal with domain shifting.
Ghifary et al.
(2016) uses an auto-encoder method by jointly training a predictor for source labels, and a decoder to reproduce target input with a shared encoder.
The decoder acts as a normalizer to force the model to learn shared knowledge between source and target domains.
Adversarial penalty can be applied to the loss function to make models learn domain-invariant feature only (Fernando et al., 2015; Ming Harry Hsu et al., 2015).
L EARNING WITH M ULTI - SOURCE S UPERVISION
We consider the multi-source sequence labeling problem as follows.
There are N sources of supervision, each source can be regarded as an imperfect annotator (non-expert human tagger or models trained in related domains).
For the k -th source data set S ( k ) =
{ ( x ( k ) i , y ( k ) i ) } m k
i =1 , we denote its i -th sentence as x ( k ) i which is a sequence of tokens: x ( k ) i =
( x ( k ) i, 1 , · · · , x ( k ) i,N ) .
The tag sequence of the sentence is thus to be y ( k )
i = { y ( k ) i,j } .
We deﬁne the sentence set of each annotators as X ( k ) =
{ x ( k ) i } m k
i =1 , and the whole training domain as the union of all sentence sets: X = (cid:83) ( K ) k
=1 X ( k ) .
The goal of the multi-source learning task is to use such imperfect annotations to train a model for predicting the tag sequence y for any sentence x in a target corpus T .
Note that the target corpus T can either share the same distribution with X (Application I) or be signiﬁcantly different (Application II).
In the following two subsections, we formulate two typical tasks in this problem as shown in Fig.
Application I: Learning with Crowd Annotations.
When learning with crowd-sourced sequence labeling data, we regard each worker as an imperfect annotator ( S ( k ) ), who may make mistakes or skip sentences in its annotations.
Note that for crowd-sourcing data, different annotators tag subsets of the same given dataset ( X ), and thus we assume there are no input distribution shifts among X ( k ) .
Also, we only test sentences in the same domain such that the distribution in target corpus T is the same as well.
That is, the marginal distribution of target corpus P T ( x ) is the same with that for each individual source dataset, i.e. P T ( x ) =
P k ( x ) .
However, due to imperfectness of the annotations in each source, the P k ( y | x ) has obvious shift from the underlying truth P ( y | x ) (illustrated in the top-left part of Fig. 1).
The multi-source learning objective here is to learn a model P T ( y | x ) for supporting inference on any new sentences in the same domain.
Application II: Unsupervised Cross-Domain Model Adaptation.
We assume that we have wellannotated data in some source domains while having no labels in the target domain at all.
Following the formulation, we claim that the input distributions P ( x ) in different source domains X ( k ) vary a lot, and ﬁtting those annotations can only generalize to in-domain samples.
That is, P k
( y | x ) ≈
P ( y | x ) only for x ∈ X ( k ) .
For target corpus sentences x ∈ T , such a source model P k
( y | x ) again differs from underlying ground truth for the target domain P T ( y | x ) and can be seen as an imperfect annotators.
Our objective in this setting is also to jointly model P T ( y , x ) while noticing that there are signiﬁcant domain shifts between T and any other X ( k ) .
P ROPOSED A PPROACH : C ONSENSUS N ETWORK
In this section, we present our two-phase framework C ON N ET for multi-source sequence labeling.
As shown in Figure 2, our proposed framework ﬁrst uses a multi-task learning schema with a special objective to decouple annotator representations as different parameters of a transformation around CRF layers.
This decoupling phase (Section 4.2) is for decoupling the model parameters into a set of annotator-invariant model parameters and a set of annotator-speciﬁc representations.
Secondly, the dynamic aggregation phase (Section 4.3) learns to contextually utilize the annotator representations with a lightweight attention mechanism to ﬁnd the best suitable transformation for each sentence, so that the model can achieve a context-aware consensus among all sources.
The inference process is described in Section 4.4.
4. T HE B ASE M ODEL FOR S EQUENCE L ABELING : BLSTM-CRF
Many recent sequence labeling frameworks (Ma & Hovy, 2016a; Misawa et al., 2017) share a very basic structure: a bidirectional LSTM network followed by a CRF tagging layer (i.e. BLSTMCRF).
The BLSTM encodes an input sequence x = { x 1 , x 2 , . . .
, x n } into a sequence of hidden state vectors h 1: n .
The CRF takes as input the hidden state vectors and computes an emission score
[Figure]
matrix U ∈ R n × L where L is the size of tag set.
It also maintains a trainable transition matrix
M ∈ R L × L .
We can consider U i,j is the score of labeling the tag with id j ∈ { 1 , 2 , ..., L } for i
th word in the input sequence x , and M i,j means the transition score from i th tag to j th .
The CRF further computes the score s for a predicted tag sequence y =
{ y 1 , y 2 , ..., y k } as
[Equation]
then tag sequence y follows the conditional distribution
[Equation]
4. T HE D ECOUPLING P HASE : L EARNING ANNOTATOR REPRESENTATIONS
For decoupling annotator-speciﬁc biases in annotations, we represent them as a transformation on emission scores and transition scores respectively.
Speciﬁcally, we learn a matrix A ( k ) ∈
R L × L for each imperfect annotator k and apply this matrix as transformation on U and M as follows:
[Equation]
From this transformation, we can see that the original score function s in Eq.
1 becomes an annotatorspeciﬁc computation.
The original emission and transformation score matrix U and M are still shared by all the annotators, while they both are transformed by the matrix
A ( k ) for k -th annotator.
While training the model parameters in this phase, we follow a multi-task learning schema.
That is, we share the model parameters for BLSTM and CRF (including W , b , M ), while updating A ( k ) only by examples in S k
= {X ( k ) , Y ( k ) } .
The assumption on the annotation representation A ( k ) is that it can model the pattern of annotation bias.
Each annotator can be seen as a noisy version of the shared model.
For the k -th annotator, A ( k ) models noise from labeling the current word and transferring from the previous label.
Speciﬁcally, each entry A ( k ) i,j captures the probability of mistakenly labeling i -th tag to j -th tag.
In other words, the base sequence labeling model in Sec. 4.1 learns the basic consensus knowledge while annotatorspeciﬁc components add their understanding to predictions.
4. T HE A GGREGATION P HASE : D YNAMICALLY R EACHING C ONSENSUS
In the second phase, our proposed network learns a context-aware attention module for a consensus representation supervised by combined predictions on the target data.
For each sentence in target data T , we use the model obtained from the decoupling phase to make predictions, and combine predictions using weighted voting.
The weight of each source is its normalized F 1 score on the training set.
Through weighted voting on such augmented labels over all source sentences X , we can ﬁnd a good approximation of underlying truth labels.
For better generalization and higher speed, an attention module is trained to estimate the relevance of each source to the target under the supervision of generated labels.
Speciﬁcally, we compute embedding of each sentence by concatenating the last hidden states of the forward LSTM and the backward LSTM, i.e.
[Equation]
i
[Equation]
The attention module takes as input the sentence embedding and outputs a normalized weight for each source:
[Equation]
where d is the size of each hidden state h ( i ) .
Source-speciﬁc matrices { A ( k ) }
Kk =1 are then aggregated into a consensus representation A ∗ i for sentence x i ∈
X by
[Equation]
In this way, the consensus representation contains more information about sources which are more related to the current type of sentence.
It also alleviates the contradiction problem among sources, because it could consider multiple sources of different emphasis.
Since only an attention model with weight matrix Q is trained, the amount of computation is relatively small.
We assume the sequence labeling model and annotator representations are well-trained in the previous phase.
The main objective in this phase is to learn how to select most suitable annotators for the current sentence.
4. P ARAMETER L EARNING AND I NFERENCE OF C ON N ET
C ON N ET learns parameters through two phases described above.
In the ﬁrst decoupling phase, each training instance from source S k is used for training the base sequence labeling model and its speciﬁc representation A ( k ) .
In the second aggregation phase, we use aggregated predictions from the ﬁrst phase to learn a lightweight attention module.
For each instance in the target corpus x i ∈ T , we calculate its embedding h i from BLSTM hidden states.
With these sentence embeddings, the context-aware attention module assigns weight q
i to each source and dynamically aggregates source-speciﬁc representations { A ( k ) } for inferring ˆ y i .
In the inference process, only the consolidated consensus matrix A ∗ i is applied to the base sequence learning model.
In this way, more specialist knowledge helps to deal with more complex instances.
4. M ODEL A PPLICATION
The proposed model can be applied to multi-sourcing learning.
Here we describe the application in two practical settings: learning with crowd annotations and unsupervised cross-domain model adaptation.
In the crowd annotation learning setting, the training data of the same domain is annotated by multiple noisy annotators, and each annotators is treated as a source.
In the decoupling phase, the model is trained on noisy annotations, and in the aggregation phase, it is trained with combined predictions on the training set.
In the cross-domain setting, the model has access to unlabeled training data of the target domain and clean labeled data of multiple source domains.
Each domain is treated as a source.
In the decoupling phase, the model is trained on source domains, and in the aggregation phase, the model is trained on combined predictions on the training data of the target domain.
Our framework can also potentially extend to new tasks other than sequence labeling with different encoders.
For example, if we use an MLP as encoder, we can transform its hidden representation by multiplying with crowd/consensus matrices.
The transformed representation will then used by a decoder to make predictions.
We will demonstrate this ability in experiments.
E XPERIMENTS
We evaluate C ON N ET in the two aforementioned settings of multi-source learning: learning with crowd annotations and unsupervised cross-domain model adaptation.
Additionally, to demonstrate the generalization of our framework, we also test our method on sequence labeling with transformer encoder in Appendix B and text classiﬁcation with MLP encoder in Section 5.5.
5. D ATASETS
Crowd-Annotation Datasets.
We use crowd-annotation datasets based on the 2003 CoNLL shared
NER task (Sang & De Meulder, 2003).
The real-world datasets, denoted as AMT, are collected by Rodrigues et al.
(2014) using Amazon’s Mechanical Turk where F1 scores of annotators against the ground truth vary from 17.60% to 89.11%.
Since there is no development set in AMT, we also follow Nguyen et al.
(2017) to use the AMT training set and CoNLL 2003 development and test sets, denoted as AMTC.
Overlapping sentences are removed in the training set, which is ignored in that work.
Additionally, we construct two sets of simulated datasets to investigate the quality and quantity of annotators.
To simulate the behavior of a non-expert annotator, a CRF model is trained on a small subset of training data and generates predictions on the whole set.
Because of the limited size of training data, each model would have a bias to certain patterns.
Cross-Domain Datasets.
In this setting, we investigate three NLP tasks: POS tagging, NER and text classiﬁcation.
For POS tagging task, we use the GUM portion (Zeldes, 2017) of Universal
Dependencies (UD) v2.3 corpus with 17 tags and 7 domains: academic, bio, ﬁction, news, voyage, wiki, and interview.
For NER task, we select the English portion of the OntoNotes v5 corpus (Hovy et al., 2006).
The corpus is annotated with 9 named entities with data from 6 domains: broadcast conversation (bc), broadcast news (bn), magazine (mz), newswire (nw), pivot text (pt), telephone conversation (tc), and web (web).
Multi-Domain Sentiment Dataset (MDS) v2.0 (Blitzer et al., 2007) is used for text classiﬁcation, which is built on Amazon reviews from 4 domains: books, dvd, electronics, and kitchen.
Since the dataset only contains word frequencies for each review without raw texts, we follow the setting in Chen & Cardie (2018) considering 5,000 most frequent words and use the raw counts as the feature vector for each review.
5. E XPERIMENT S ETUP
For sequence labeling tasks, we follow Liu et al.
(2018) to build the BLSTM-CRF architecture as the base model.
The dimension of character-level, word-level embeddings and BLSTM hidden layer are set as 30 , 100 and 150 respectively.
For text classiﬁcation, We use an MLP with a hidden size of 100 as encoder and a linear classiﬁcation layer for predicting their labels.
The dropout with a probability of 0 . 5 is applied to the non-recurrent connections for regularization.
The network parameters are randomly initialized and updated by stochastic gradient descent (SGD).
The learning rate is initialized as 0 . 015 and decayed by 5% for each epoch.
The training process stops early if no improvements in 15 continuous epochs and selects the best model on the development set.
For the dataset without a development set, we report the performance on the 50 -th epoch.
For each experiment, we report the average performance of 3 runs with different random initialization.
5. C OMPARISON WITH BASELINE METHODS
We compare our models with multiple baselines, which can be categorized in two groups: wrapper methods and joint models.
To demonstrate the theoretical upper bound of performance, we also train the base model using ground-truth annotations in the target domain ( Gold ).
A wrapper method consists of a label aggregator and a deep learning model.
These two components could be combined in two ways: (1) aggregating labels on crowd-sourced training set then feeding the generated labels to a Sequence Labeling Model ( SLM ) (Liu et al., 2017); (2) feeding multi-source data to a Multi-Task Learning ( MTL ) (Wang et al., 2018) model then aggregating multiple predicted labels.
We investigate multiple label aggregation strategies.
CONCAT considers all crowd annotations as gold labels.
MVT does majority voting on the token level, i.e. , the majority of labels { y ki,j } is selected as the gold label for each token x i,j .
MVS is conducted on the sequence level, addressing the problem of violating Begin/In/Out (BIO) rules.
DS (Dawid & Skene, 1979), HMM (Nguyen et al., 2017) and BEA (Rahimi et al., 2019) induce consensus labels with probability models.
In contrast with wrapper methods, joint models incorporate multi-source data within the structure of sequential taggers and jointly model all of the individual annotators.
CRF-MA models CRFs with
Multiple Annotators by EM algorithm (Rodrigues et al., 2014).
Nguyen et al.
(2017) augments the
LSTM architecture with crowd vectors.
These crowd components are element-wise added to the tags scores ( Crowd-Add ) or concatenated to the output of the hidden layer ( Crowd-Cat ).
These two methods are the most similar to our extraction phase.
We implemented them and got better results
[Table]
[Figure]
than reported.
CL-MW applies a crowd layer to a CNN-based deep learning framework (Rodrigues & Pereira, 2018).
Tri-Training uses bootstrapping with multi-task Tri-Training approach for unsupervised one-to-one domain adaptation (Saito et al., 2017; Ruder & Plank, 2018).
5. P ERFORMANCE ON LEARNING WITH C ROWD A NNOTATIONS
Tab. 1 shows the performance of aforementioned methods and our C ON N ET on two real-world datasets, i.e. AMT and AMTC.
We can see that C ON N ET outperforms all other methods on both datasets signiﬁcantly on F 1 score, which shows the effectiveness of dealing with noisy annotations for higher-quality labels.
Although CONCAT-SLM achieves the highest precision, it suffers from low recall.
All existing methods have the high-precision but low-recall problem.
One possible reason is that they try to ﬁnd the latent ground truth and throw away illuminating annotator-speciﬁc information.
So only simple mentions can be classiﬁed with great certainty while difﬁcult mentions fail to be identiﬁed without sufﬁcient knowledge.
In comparison, C ON N ET pools information from all annotations and focus on matching knowledge to make predictions.
It makes the tagging model be able to identify more mentions and get a higher recall.
It is enlightening to analyze whether the model decides the importance of annotators given a sentence.
In Fig.
4 we visualize test F1 score of all annotators on each tag, and attention weights q i in Eq.
4 for 4 sampled sentences containing different entity types.
Obviously, the 2 nd sample sentence with ORG has higher attention weights on 1 st, 5 th and 33 rd annotator who are best at labeling
ORG .
More details and cases are shown in Appendix A.1.
We also investigate multiple variants and conduct ablation study in Appendix A.2.
To analyze the impact of annotator quality, we split the origin train set into { 5 , 10 , 15 , 30 , 50 } folds and train a CRF model on each fold whose reliability could be represented as r = { 1 / 5 , 1 / 10 , 1 / 15 , 1 / 30 , 1 / 50 } because a model with less training data would have stronger bias and less generalization.
For each setting, we randomly select 5 trained models as the simulated annotators to annotate the whole training set.
When the reliability level of all annotators is too low, i.e. 1 / 50 , only the base model is used for prediction without annotator representations.
The models are then trained with simulated annotations.
Shown in Fig. 3(a), C ON N ET achieves signiﬁcant [Figure] improvements over MVT-SLM and competitive performance as Crowd-Cat .
Our model shows its effectiveness when annotators are less reliable.
Regarding the inﬂuence of annotator quantity, we split the train set into 50 subsets to train 50 models ( r = 1 / 50 ) respectively and randomly select { 5 , 10 , 15 , 30 , 50 } models as simulated annotators to annotate the whole training set.
The models are then trained with simulated annotations.
Fig.
3(b) shows C ON N ET is superior to baselines and able to well deal with many annotators while there is no obvious relationship between the performance and the number of annotators in baselines.
We can see the performance of our model increases as the number of annotators and, regardless of the number of annotators, our method consistently outperforms than other baselines.
5. P ERFORMANCE ON C ROSS -D OMAIN A DAPTATION
[Table]
The average performance of each method on each task is shown in Tab. 2.
More detailed results on each target domain can be found in Appendix A.3.
We report the accuracy for POS tagging and text classiﬁcation, and report the chunk-level F1 score for NER.
We can see that C ON N ET achieves the highest average score in all tasks.
MTL-MVT is similar to our decoupling phase without the attention module, which performs much worse.
It shows that naively doing unweighted voting does not work well.
The attention can be viewed as implicitly doing weighted voting on the feature level.
This demonstrates the importance of having such a module to assign weights to all domains based on the input sentence.
The Tri-Training model trained on the concatenated training data from all sources performs worse than C ON N ET , which suggests that it is important to have a multi-task structure to model the difference among domains.
We analyze the OntoNotes dataset to show that the attention scores generated by the model are meaningful.
Details can be found in Appendix A.4.
C ONCLUSION
In this paper, we present C ON N ET for learning a sequence tagger from multi-source supervision.
It could be applied in two practical scenarios: learning with crowd annotations and cross-domain adaptation.
In contrast to prior works, C ON N ET learns ﬁne-grained representations of each source which are further dynamically aggregated for every unseen sentence in the target data.
Experiments show that our model is superior to previous crowd-sourcing and unsupervised domain adaptation sequence labeling models.
The proposed learning framework also shows promising results on other
NLP tasks like text classiﬁcation.
A A NALYSIS OF C ON N ET WITH BLSTM E NCODER A. C ASE STUDY ON LEARNING WITH CROWD ANNOTATIONS
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure]  Defender [ PER Hassan Abbas] rose to intercept a long ball into the area in the 84th minute but only managed to divert it into the top corner of [ PER Bitar] ’s goal .
[ ORG Plymouth] 4 [ ORG Exeter] Hosts [ LOC UAE] play [ LOC Kuwait] and [ LOC South Korea] take on [ LOC Indonesia] on Saturday in Group A matches .
The former [ MISC Soviet] republic was playing in an [ MISC Asian Cup] ﬁnals tie for the ﬁrst time .
[ PER Bitar] pulled off ﬁne saves whenever they did .
[ PER Coste] said he had approached the player two months ago about a comeback .
[ ORG Goias] 1 [ ORG Gremio] [ ORG Portuguesa] 1 [ ORG Atletico Mineiro] [ LOC Melbourne] 1996-12-   On Friday for their friendly against [
LOC Scotland
] at [ LOC Murrayﬁeld] more than a year after the 30-year-old wing announced he was retiring following differences over selection .
Scoreboard in the [ MISC World Series] Cricket - [ MISC Shefﬁeld Shield] score .
“ He ended the [ MISC World Cup] on the wrong note , ” [ PER Coste] said .
Soccer - [ ORG Leeds] ’ [ PER Bowyer] ﬁned for part in fast-food fracas .
[ ORG Rugby Union] - [ PER Cuttitta] back for [ LOC Italy] after a year .
[ LOC Australia] gave [ PER Brian Lara] another reason to be miserable when they beat [ LOC West Indies] by ﬁve wickets in the opening [ MISC World Series] limited overs match on Friday .
To better understand the effect and beneﬁt of C ON N ET , we do some case study on AMTC real-world dataset with 47 annotators.
We look into some more instances to investigate the ability of attention module to ﬁnd right annotators in Fig.
5 and Tab. 3.
Sentence 1-12 contains a speciﬁc entity type respectively while 13-16 contains multiple different entities.
Compared with expertise of annotators, we can see that the attention module would give more weight on annotators who have competitive performance and preference on the included entity type.
Although top selected annotators for ORG has relatively lower expertise on ORG than PER and LOC , they are actually the top three annotators with highest expertise on ORG .
A. A BLATION S TUDY ON C ON N ET
To investigate the effectiveness of our proposed method, multiple variants of the decoupling phase and aggregation phase are evaluated on the AMT dataset, shown in Fig. 6(a).
Decoupling Phase We tried three different approaches to incorporate source-speciﬁc representation in the decoupling phase (DP).
CRF means the traditional approach without annotator matrices as
Eq.
1 while DP(1+2) is described in Eq.
Similarly, DP(1) only applies source representations
A ( k ) to the emission score U while DP(2) only transfers the transition matrix M .
They directly use latent representations U and M in the decoupling phase (not augmented by A ( k ) ) for prediction.
We can observe from the result that both variants that model the annotator-speciﬁc information can improve the result.
The underlying model keeps more consensus knowledge if we extract annotatorspeciﬁc bias on sentence encoding and label transition.
Aggregation Phase We compare four methods of generating supervision targets in the aggregation phase (AP).
OMV uses majority voting of original annotations as targets for training, while PMV substitutes them with model prediction learned from DP.
AMV extends the model by using all prediction, while AWV uses majority voting weighted by each annotator’s training F 1 score.
The results show the effectiveness of AWV , which could well approximate the ground truth and supervise the attention module to estimate the expertise of annotator on the current sentence.
We can also infer labels on the test set by conducting AWV on predictions of the underlying model with each annotator-speciﬁc components.
But it leads to heavy computation-consuming and unsatisfying performance, whose test F 1 score is 77 .
35( ± 0 . 08) .
To investigate the importance of extracted source-speciﬁc components, we trained a traditional BLSTM-CRF model by the supervision of the same AMV labels.
Its result is 78 .
93( ± 0 . 13) , which is lower than our C ON N ET .
These experiments convey four messages: (1) Using learned model prediction is better than directly using original annotations as its target; (2) It is desirable to use all model predictions instead of just conﬁne each annotator on their data set.
This means that the learned representation of individual annotator can generalize on data it didn’t annotate.
(3) Weighting aggregating models by their performance is beneﬁcial to overall accuracy.
The attention model could learn to estimate annotators’ ability based on the context.
[Figure]
A. D ETAILED RESULTS FOR C ROSS -D OMAIN A DAPTATION
The results of each task on each domain are shown in Tab. 4.
We can see that C ON N ET performs the best on most of the domains and achieves the highest average score for all tasks.
A. M ULTI - DOMAIN : A NALYSIS
We analyzed the attention scores generated by the attention module on the OntoNotes dataset.
For each sentence in the target domain we collected the attention score of each source domain, and ﬁnally the attention scores are averaged for each source-target pair.
Fig.
6(b) shows all the source-to-target average attention scores.
We can see that some domains can contribute to other related domains.
For example, bn (broadcast news) and nw (newswire) are both about news and they contribute to each other; bn and bc (broadcast conversation) are both broadcast and bn contributes to bc; bn and nw both contributes to mz (magzine) probably because they are all about news; wb (web) and tc (telephone conversation) almost make no positive contribution to any other, which is reasonable because they are informal texts compared to others and they are not necessarily related to the other.
Overall the attention scores can make some sense to human, and it suggests that the attention is meaningful and contributing to the model
[Table]
B R ESULT OF C ON N ET WITH T RANSFORMER E NCODER
[Table]
To demonstrate the generalization of our framework, we re-implement C ON N ET and some baselines ( MTV-SLM , Crowd-Add , Gold ) with Transformer-CRF as the base model.
Speciﬁcally, the base model takes Transformer as the encoder for CRF, which has shown its effectiveness in many NLP tasks (Vaswani et al., 2017; Devlin et al., 2018).
Transformer models sequences with self-attention and eliminates all recurrence.
Following the experimental settings from Vaswani et al.
(2017), we set the number of heads for multi-head attention as 8 , the dimension of keys and values as 64 , and the hidden size of the feed-forward layers as 1024 .
We conduct experiments with crowd-annotation dataset AMTC on NER task and cross-domain dataset UD on POS task, which are described in
Section 5.1.
Results are shown in Table 5.
We can see our model outperforms over other baselines in both tasks and applications.