DATN ET : D UAL A DVERSARIAL T RANSFER FOR L OWRESOURCE N AMED E NTITY R ECOGNITION
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a new architecture termed Dual Adversarial Transfer Network (DATNet) for addressing low-resource Named Entity Recognition (NER).
Speciﬁcally, two variants of DATNet, i e., DATNet-F and DATNet-P, are proposed to explore effective feature fusion between high and low resource.
To address the noisy and imbalanced training data, we propose a novel Generalized ResourceAdversarial Discriminator (GRAD).
Additionally, adversarial training is adopted to boost model generalization.
We examine the effects of different components in DATNet across domains and languages, and show that signiﬁcant improvement can be obtained especially for low-resource data.
Without augmenting any additional hand-crafted features, we achieve new state-of-the-art performances on
CoNLL and Twitter NER—88.16% F1 for Spanish, 53.43% F1 for WNUT-2016, and 42.83% F1 for WNUT-2017 1 .
I NTRODUCTION
Named entity recognition (NER) is an important step in most natural language processing (NLP) applications.
It detects not only the type of named entity, but also the entity boundaries, which requires deep understanding of the contextual semantics to disambiguate the different entity types of same tokens.
To tackle this challenging problem, most early studies were based on hand-crafted rules, which suffered from limited performance in practice.
Current methods are devoted to developing learning based algorithms, especially neural network based methods, and have been advancing the state-of-the-art consecutively (Collobert et al., 2011; Huang et al., 2015; Lample et al., 2016; Chiu & Nichols, 2016; Ma & Hovy, 2016).
These end-to-end models generalize well on new entities based on features automatically learned from the data.
However, when the annotated corpora is small, especially in the low resource scenario (Zhang et al., 2016), the performance of these methods degrades signiﬁcantly since the hidden feature representations cannot be learned adequately.
Recently, more and more approaches have been proposed to address low-resource NER.
Early works (Chen et al., 2010; Li et al., 2012) primarily assumed a large parallel corpus and focused on exploiting them to project information from high- to low-resource.
Unfortunately, such a large parallel corpus may not be available for many low-resource languages.
More recently, cross-resource word embedding (Fang & Cohn, 2017; Adams et al., 2017; Yang et al., 2017) was proposed to bridge the low and high resources and enable knowledge transfer.
Although the aforementioned transferbased methods show promising performance in low-resource NER, there are two issues deserved to be further investigated on: 1) Representation Difference - they did not consider the representation difference across resources and enforced the feature representation to be shared across languages/domains; 2) Resource Data Imbalance - the training size of high-resource is usually much larger than that of low-resource.
The existing methods neglect such difference in their models, resulting in poor generalization.
In this work, we present an approach termed Dual Adversarial Transfer Network (DATNet) to address the above issues in a uniﬁed framework for low-resource NER.
Speciﬁcally, to handle the representation difference, we ﬁrst investigate on two architectures of hidden layers (we use bidirectional long-short term memory (BiLSTM) model as hidden layer) for transfer.
The ﬁrst one is that all the units in hidden layers are common units shared across languages/domains.
The second one is composed of both private and common units, where the private part preserves the independent language/domain information.
Extensive experiments are conducted to show their advantages over each other in different situations.
On top of common units, the adversarial discriminator (AD) loss is introduced to encourage the resource-agnostic representation so that the knowledge from high resource can be more compatible with low resource.
To handle the resource data imbalance issue, we further propose a variant of the AD loss, termed Generalized Resource-Adversarial Discriminator (GRAD) , to impose the resource weight during training so that low-resource and hard samples can be paid more attention to.
In addition, we create adversarial samples to conduct the Adversarial Training (AT) , further improving the generalization and alleviating over-ﬁtting problem.
We unify two kinds of adversarial learning, i.e., GRAD and AT, into one transfer learning model, termed Dual Adversarial Transfer Network (DATNet), to achieve end-to-end training and obtain the state-of-the-art performance on a series of NER tasks–88.16% F1 for CoNLL-2002 Spanish, 53.43% and 42.83%
F1 for WNUT-2016 and 2017.
Different from prior works, we do not use additional hand-crafted features and do not use cross-lingual word embeddings while addressing the cross-language tasks.
R ELATED W ORK Named Entity Recognition
NER is typically framed as a sequence labeling task which aims at automatic detection of named entities (e.g., person, organization, location and etc.) from free text (Marrero et al., 2013).
The early works applied CRF, SVM, and perception models with handcrafted features (Ratinov & Roth, 2009; Passos et al., 2014; Luo et al., 2015).
With the advent of deep learning, research focus has been shifting towards deep neural networks (DNN), which requires little feature engineering and domain knowledge (Lample et al., 2016; Zukov Gregoric et al., 2018).
Collobert et al.
(2011) proposed a feed-forward neural network with a ﬁxed sized window for each word, which failed in considering useful relations between long-distance words.
To overcome this limitation, Chiu & Nichols (2016) presented a bidirectional LSTM-CNNs architecture that automatically detects word- and character-level features.
Ma & Hovy (2016) further extended it into bidirectional LSTM-CNNs-CRF architecture, where the CRF module was added to optimize the output label sequence.
Liu et al.
(2018) proposed task-aware neural language model termed LMLSTM-CRF, where character-aware neural language models were incorporated to extract characterlevel embedding under a multi-task framework.
Transfer Learning for NER
Transfer learning can be a powerful tool to low resource NER tasks.
To bridge high and low resource, transfer learning methods for NER can be divided into two types: the parallel corpora based transfer and the shared representation based transfer.
Early works mainly focused on exploiting parallel corpora to project information between the high- and low-resource language (Yarowsky et al., 2001;
Chen et al., 2010; Li et al., 2012;
Feng et al., 2018).
For example, Chen et al.
(2010) and Feng et al.
(2018) proposed to jointly identify and align bilingual named entities.
On the other hand, the shared representation methods do not require the parallel correspondence (Rei & Søgaard, 2018).
For instance, Fang & Cohn (2017) proposed cross-lingual word embeddings to transfer knowledge across resources.
Yang et al.
(2017) presented a transfer learning approach based on a deep hierarchical recurrent neural network (RNN), where full/partial hidden features between source and target tasks are shared.
Ni et al.
(Ni & Florian, 2016; Ni et al., 2017) utilized the Wikipedia entity type mappings to improve low-resource NER. Al-Rfou’ et al.
(2015) built massive multilingual annotators with minimal human expertise by using language agnostic techniques.
Mayhew et al.
(2017) created a cross-language NER system, which works well for very minimal resources by translate annotated data of high-resource into low-resource.
Cotterell & Duh (2017) proposed character-level neural CRFs to jointly train and predict low- and high-resource languages.
Pan et al. (2017) proposes a large-scale cross-lingual named entity dataset which contains 282 languages for evaluation. In addition, multi-task learning (Yang et al., 2016; Luong et al., 2016; Rei, 2017; Aguilar et al., 2017; Hashimoto et al., 2017; Lin et al., 2018) shows that jointly training on multiple tasks/languages helps improve performance.
Different from transfer learning methods, multi-task learning aims at improving the performance of all the resources instead of low resource only.
Adversarial Learning
Adversarial learning originates from Generative Adversarial Nets (GAN) (Goodfellow et al., 2014), which shows impressing results in computer vision.
Recently, many papers have tried to apply adversarial learning to NLP tasks.
Liu et al.
(2017) presented an adversarial multi-task learning framework for text classiﬁcation.
Gui et al.
(2017) applied the adversarial discriminator to POS tagging for Twitter.
Kim et al.
(2017) proposed a language discriminator to enable language-adversarial training for cross-language POS tagging.
Apart from adversarial discriminator, adversarial training is another concept originally introduced by (Szegedy et al., 2014;
Goodfellow et al., 2015) to improve the robustness of image classiﬁcation model by injecting malicious perturbations into input images.
Recently, Miyato et al.
(2017) proposed a semi-supervised text classiﬁcation method by applying adversarial training, where for the ﬁrst time adversarial perturbations were added onto word embeddings.
Yasunaga et al.
(2018) applied adversarial training to POS tagging.
Different from all these adversarial learning methods, our method integrates both the adversarial discriminator and adversarial training in an uniﬁed framework to enable end-to-end training.
D UAL A DVERSARIAL T RANSFER N ETWORK (DATN ET )
In this section, we introduce DATNet in more details.
We ﬁrst describe a base model for NER, and then discuss two proposed transfer architectures for DATNet.
[Figure]
3. B ASIC A RCHITECTURE
We follow state-of-the-art models for NER task (Huang et al., 2015; Lample et al., 2016; Chiu &
Nichols, 2016; Ma & Hovy, 2016), i.e., LSTM-CNNs-CRF based structure, to build the base model.
It consists of the following pieces: character-level embedding, word-level embedding, BiLSTM for feature representation, and CRF as the decoder.
The character-level embedding takes a sequence of characters in the word as atomic units input to derive the word representation that encodes the morphological information, such as root, preﬁx, and sufﬁx.
These character features are usually encoded by character-level CNN or BiLSTM, then concatenated with word-level embedding to form the ﬁnal word vectors.
On top of them, the network further incorporates the contextual information using BiLSTM to output new feature representations, which is subsequently fed into CRF layer to predict label sequence.
Although both of the word-level layer and the character-level layer can be implemented using CNNs or RNNs, we use CNNs for extracting character-level and RNNs for extracting word-level representation.
Fig.
1(a) shows the the architecture of the base model.
3. D UAL A DVERSARIAL T RANSFER A RCHITECTURE 3.2. C HARACTER - LEVEL E NCODER
Previous works have shown that character features can boost sequence labeling performance by capturing morphological and semantic information (Lin et al., 2018).
For low-resource dataset to obtain high-quality word features, character features learned from other language/domain may provide crucial information for labeling, especially for rare and out-of-vocabulary words.
Character-level encoder usually contains BiLSTM (Lample et al., 2016) and CNN (Chiu & Nichols, 2016; Ma & Hovy, 2016) approaches.
In practice, Reimers & Gurevych (2017) observed that the difference between the two approaches is statistically insigniﬁcant in sequence labeling tasks, but character-level CNN is more efﬁcient and has less parameters.
Thus, we use character-level CNN and share character features between high- and low-resource tasks to enhance the representations of low-resource.
3.2. W ORD - LEVEL E NCODER
To learn a better word-level representation, we concatenate character-level features of each word with a latent word embedding as w i =
[ w chari , w embi ] , where the latent word embedding w embi is initialized with pre-trained embeddings and ﬁxed during training.
One unique characteristic of NER is that the historical and future input for a given time step could be useful for label inference.
To exploit such a characteristic, we use a bidirectional LSTM architecture (Hochreiter & Schmidhuber, 1997) to extract contextualized word-level features.
In this way, we can gather the information from the past and future for a particular time frame t as follows, −→ h t = lstm ( −→ h t − 1 , w t ) , ←− h t = lstm ( ←− h t +1 , w t ) .
After the LSTM layer, the representation of a word is obtained by concatenating its left and right context representation as follows, h t =
[ −→ h t , ←− h t ] .
To consider the resource representation difference on word-level features, we introduce two kinds of transferable word-level encoder in our model, namely DATNet-Full Share (DATNet-F) and
DATNetPart Share
(DATNet-P).
In DATNet-F, all the BiLSTM units are shared by both resources while word embeddings for different resources are disparate.
The illustrative ﬁgure is depicted in the Fig.
1(c).
Different from DATNet-F, the DATNet-P decomposes the BiLSTM units into the shared component and the resource-related one, which is shown in the Fig. 1(b).
3.2. G ENERALIZED R ESOURCE -A DVERSARIAL D ISCRIMINATOR
In order to make the feature representation extracted from the source domain more compatible with those from the target domain, we encourage the outputs of the shared BiLSTM part to be resourceagnostic by constructing a resource-adversarial discriminator, which is inspired by the LanguageAdversarial Discriminator proposed by Kim et al.
Unfortunately, previous works did not consider the imbalance of training size for two resources.
Speciﬁcally, the target domain consists of very limited labeled training data, e.g., 10 sentences.
In contrast, labeled training data in the source domain are much richer, e.g., 10k sentences.
If such imbalance was not considered during training, the stochastic gradient descent (SGD) optimization would make the model more biased to high resource (Lin et al., 2017b).
To address this imbalance problem, we impose a weight α on two resources to balance their inﬂuences.
However, in the experiment we also observe that the easily classiﬁed samples from high resource comprise the majority of the loss and dominate the gradient.
To overcome this issue, we further propose Generalized Resource-Adversarial Discriminator (GRAD) to enable adaptive weights for each sample (note that the sample here means each sentence of resource), which focuses the model training on hard samples.
To compute the loss of GRAD, the output sequence of the shared BiLSTM is ﬁrstly encoded into a single vector via a self-attention module (Bahdanau et al., 2015), and then projected into a scalar r via a linear transformation.
The loss function of the resource classiﬁer is formulated as:
[Equation]
where I i ∈D S , I i ∈D T are the identity functions to denote whether a sentence is from high resource (source) and low resource (target), respectively; α is a weighting factor to balance the loss contribution from high and low resource; the parameter (1 − r i ) γ (or r γi ) controls the loss contribution from individual samples by measuring the discrepancy between prediction and true label (easy samples have smaller contribution); and γ scales the contrast of loss contribution from hard and easy samples.
In practice, the value of γ does not need to be tuned much and usually set as 2 in our experiment.
Intuitively, the weighting factors α and (1 − r i ) γ reduce the loss contribution from high resource and easy samples, respectively.
Note that though the resource classiﬁer is optimized to minimize the resource classiﬁcation error, when the gradients originated from the resource classiﬁcation loss are back-propagated to the other model parts than the resource classiﬁer, they are negated for parameter updates so that these bottom layers are trained to be resource-agnostic.
3.2. L ABEL D ECODER
The label decoder induces a probability distribution over sequences of labels, conditioned on the word-level encoder features.
In this paper, we use a linear chain model based on the ﬁrst-order
Markov chain structure, termed the chain conditional random ﬁeld (CRF) Lafferty et al.
(2001), as the decoder.
In this decoder, there are two kinds of cliques: local cliques and transition cliques.
Speciﬁcally, local cliques correspond to the individual elements in the sequence.
And transition cliques, on the other hand, reﬂect the evolution of states between two neighboring elements at time t − 1 and t and we deﬁne the transition distribution as θ .
Formally, a linear-chain CRF can be written as p ( y | [Equation] [Equation] =
[Equation]
, where Z ( h 1: T ) is a normalization term and y is the sequence of predicted labels as follows:
y = y 1: T .
Model parameters are optimized to maximize this conditional log likelihood, which acts as the objective function of the model.
We deﬁne the loss function for source and target resources as follows,
(cid:96) S = − (cid:80) i log p ( y | h 1: T ) , (cid:96) T = − (cid:80) i log p ( y | h 1: T ) .
3.2. A DVERSARIAL T RAINING
So far our model can be trained end-to-end with standard back-propagation by minimizing the following loss:
[Equation]
Recent works have demonstrated that deep learning models are fragile to adversarial examples
Goodfellow et al. (2015).
In computer vision, those adversarial examples can be constructed by changing a very small number of pixels, which are virtually indistinguishable to human perception (Pin-Yu et al., 2018).
Recently, adversarial samples are widely incorporated into training to improve the generalization and robustness of the model, which is so-called adversarial training (AT) (Miyato et al., 2017).
It emerges as a powerful regularization tool to stabilize training and prevent the model from being stuck in local minimum.
In this paper, we explore AT in context of NER.
To be speciﬁc, we prepare an adversarial sample by adding the original sample with a perturbation bounded by a small norm (cid:15) to maximize the loss function as follows:
[Equation]
where Θ is the current model parameters set.
However, we cannot calculate the value of η exactly in general, because the exact optimization with respect to η is intractable in neural networks.
Following the strategy in Goodfellow et al.
, this value can be approximated by linearizing it as follows,
[Equation]
where (cid:15) can be determined on the validation set.
In this way, adversarial examples are generated by adding small perturbations to the inputs in the direction that most signiﬁcantly increases the loss function of the model.
We ﬁnd such η against the current model parameterized by Θ , at each training step, and construct an adversarial example by x adv = x
+ η x .
Noted that we generate this adversarial example on the word and character embedding layer, respectively, as shown in the Fig. 1(b) and 1(c).
Then, the classiﬁer is trained on the mixture of original and adversarial examples to improve the generalization.
To this end, we augment the loss in Eqn. 2 and deﬁne the loss function for adversarial training as:
[Equation]
where (cid:96) (Θ; x ) , (cid:96) (Θ; x adv ) represents the loss from an original example and its adversarial counterpart, respectively.
Note that we present the AT in a general form for the convenience of presentation.
For different samples, the loss and parameters should correspond to their counterparts.
For example, for the source data with word embedding w S , the loss for AT can be deﬁned as follows, (cid:96) AT
= (cid:96) (Θ; w S )
+ (cid:96) (Θ; w S,adv ) with w S,adv = w S + η w S
and (cid:96) =
(cid:96) GRAD + (cid:96) S .
Similarly, we can compute the perturbations η c for char-embedding and η w T for target word embedding.
E XPERIMENTS 4. D ATASETS
In order to evaluate the performance of DATNet, we conduct the experiments on following widely used NER datasets: CoNLL-2003 English NER (Kim & De, 2003), CoNLL-2002 Spanish & Dutch
NER (Kim, 2002), WNUT-2016 & 2017 English Twitter NER (Zeman, 2017).
The statistics of these datasets are described in Table 1.
We use the ofﬁcial split of training/validation/test sets.
Since our goal is to study the effects of transferring knowledge from high-resource dataset to low-resource dataset, unlike previous works (Collobert et al., 2011; Chiu & Nichols, 2016; Yang et al., 2017) to append one-hot gazetteer features to the input of the CRF layer, and the works (Partalas et al., 2016; Limsopatham & Collier, 2016; Aguilar et al., 2017) to introduce orthographic feature as additional input for learning social media NER in tweets, we do not experiment with hand-crafted features and only consider words and characters embeddings as the inputs of our model.
To be noted, we used only train set for model training for all datasets except the WNUT-2016 NER dataset.
Since in this dataset, all the previous studies merged the training and validation sets together for training, we followed the same way for fair comparison.
Speciﬁcally, we use CoNLL-2003 English NER dataset as high-resource (i.e., source) for all the experiments on CoNLL and WNUT datasets, while CoNLL-2002 Spanish & Dutch NER datasets and WNUT-2016 & 2017 Twitter NER datasets as low-resource (i.e., target) in cross-language and cross-domain NER settings, respectively. [Table]
In addition to the CoNLL and WNUT datasets, we also experiment on the cross-language named entity dataset described in Pan et al.
(2017), which contains datasets for 282 languages, to evaluate our methods and investigate the transferability of different linguistic families and branches in both low- and high-resource scenarios.
We choose 9 languages in our experiment, where Galician (gl) ,
West Frisian (fy) , Ukrainian (uk) and Marathi (mr) are target languages, the corresponding source languages are Spanish (es) , Dutch (nl) , Russian (ru) and Hindi (hi) , and Arabic (ar) is also a source language, which is from different linguistic family. Following the setting in Cotterell & Duh (2017), we also simulate the low- and high-resource scenarios by creating 100 and 10,000 sentences split for training target language datasets, respectively. Then we create 1,000 sentences split for validation and test, respectively. For source languages, we create 10,000 sentence split for training only.
For high-resource scenario, we only conduct experiments on Galician (gl-high) and Ukrainian (uk-high) .
The list of selected datasets are described in Table 2.
[Table]
4. E XPERIMENTAL S ETUP
We use 50-dimensional publicly available pre-trained word embeddings for English, Spanish and
Dutch languages of CoNLL and WNUT datasets in our experiments, which are trained by word2vec package 2 on the corresponding Wikipedia articles (2017-12-20 dumps) (Lin et al., 2018).
For the named entity datasets selected from Pan et al. , we use 300-dimensional pre-trained word embeddings trained by fastText package 3 on Wikipedia (Bojanowski et al., 2017), and the 30dimensional randomly initialized character embeddings are used for all the datasets.
We set the ﬁlter number as 20 for char-level CNN and the dimension of hidden states of the word-level LSTM as 200 for both base model and DATNet-F. For DATNet-P, we set 100 for source, share, and target
LSTMs dimension, respectively.
Parameters optimization is performed by Adam optimizer (Kingma & Ba, 2014) with gradient clipping of 5.0 and learning rate decay strategy.
We set the initial learning rate of β 0 = 0 .
001 for all experiments.
At each epoch t , learning rate β t is updated using β t = β 0 / (1 + ρ × t ) , where ρ is decay rate with 0.05.
To reduce over-ﬁtting, we also apply Dropout (Srivastava et al., 2014) to the embedding layer and the output of the LSTM layer, respectively.
4. C OMPARISON WITH S TATE - OF -T HE -A RT R ESULTS
In this section, we compare our approach with state-of-the-art (SOTA) methods on CoNLL and WNUT benchmark datasets.
In the experiment, we exploit all the source data (i.e., CoNLLEnglish NER) and target data to improve performance of target tasks.
The averaged results with standard deviation over 10 repetitive runs are summarized in Table 3, and we also report the best results on each task for fair comparison with other SOTA methods. From results, we observe that incorporating the additional resource is helpful to improve performance.
DATNet-P model achieves the highest F1 score on CoNLL-2002 Spanish and second F1 score on CoNLL-2002 Dutch dataset while DATNet-F model beats others on WNUT-2016 and WNUT-2017 English Twitter datasets.
Different from other state-of-the-art models, DATNets do not use any addition features 4 .
[Table]
Table 4 summarizes the results of our methods under different cross-language transfer settings as well as the comparison with Cotterell & Duh (2017).
In this experiment, we study the transferability between languages not only from same linguistic family and branch, but also from different linguistic families or branches.
According to the results, DATNets outperform the transfer method of
Cotterell & Duh (2017) for both low- and high-resource scenarios within the same linguistic family and branch (i.e., in-family in-branch) transfer case.
We also observe that: 1)
For the low-resource scenario, transfer learning is signiﬁcantly helpful for improving the performance of target datasets
https://github.com/tmikolov/
https://github.com/facebookresearch/
Although our model performance on CONLL-2002 Dutch NER dataset is only comparable to the SOTA result, on the one hand, we do not use any addition features while the SOTA method did use; on the other, we are not sure if the SOTA method has incorporated the validation set into training.
And if we merge training and validation sets, we can push the F1 score to 88.71, which outperforms the SOTA method.
within both same and different linguistic family or branch (i.e., in/cross-family in/cross-branch) transfer cases, while the improvements are more prominent under the in-family in-branch case. 2)
For the high-resource scenario, say, when the target language data is sufﬁcient, the improvements of transfer learning are not very distinct compared with that for low-resource scenario under in-family in-branch case.
We also ﬁnd that there is no effect by transferring knowledge from Arabic to Galician and Ukrainian .
We suspect that it is caused by the great linguistic differences between source and target languages, since, for example, Arabic and Galician are from totally different linguistic families.
[Table]
[Figure]
[Figure]
[Figure]
gl
[Figure]
Cross
[Figure]
Family Cross
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
uk
In-Family In-Branch
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
uk
In-Family Cross-Branch - -
[Figure]
[Figure]
[Figure]
Cross-Family Cross-Branch - -
[Figure]
[Figure]
[Figure]
In-Family In-
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] Family [Figure]
[Figure] [Figure] [Figure] [Figure]
[Table]
[Figure] [Figure] 4. T RANSFER L EARNING P ERFORMANCE In this section, we investigate on improvements with transfer learning under multiple low-resource settings with partial target data.
To simulate a low-resource setting, we randomly select subsets of target data with varying data ratio at 0.05, 0.1, 0.2, 0.4, 0.6, and 1.0.
For example, 20, training tokens are sampled from the training set under a data ratio of r = 0 .
1 for the dataset CoNLL-2002 Spanish NER (Cf.
Table 1).
The results for cross-language and cross-domain transfer are shown in Fig. 2(a) and 2(b), respectively, where we compare the results with each part of DATNet under various data ratios.
From those ﬁgures, we have the following observations: 1) both adversarial training and adversarial discriminator in DATNet consistently contribute to the performance improvement; 2) the transfer learning component in the DATNet consistently improve over the base model results and the improvement margin is more substantial when the target data [Table]
ratio is lower. For example, when the data ratio is 0.05, DATNet-P model outperforms the base model by more than 4% absolutely in F1-score on Spanish NER and DATNet-F model improves around 13% absolutely in F1-score compared to base model on WNUT-2016 NER.
In the second experiment
[Table]
we further investigate DATNet on the extremely low resource cases, e.g., the number of training target sentences is 10, 50, 100, 200, 500 and 1,000.
The setting is quite challenging and fewer previous works have studied before.
The results are summarized in Table 5.
We have two interesting observations 5 : 1) DATNet-F outperforms DATNet-P on cross-language transfer when the target resource is extremely low, however, this situation is reversed when the target dataset size is large enough (here for this speciﬁc dataset, the threshold is 100 sentences); 2) DATNet-F is always superior to DATNet-P on cross-domain transfer.
For the ﬁrst observation, it is because
DATNet-F with more shared hidden units is more efﬁcient to transfer knowledge than DATNet-P when data size is extremely small.
For the second observation, because cross-domain transfer are in the same language, more knowledge is common between the source and target domains, requiring more shared hidden features to carry with these knowledge compared to cross-language transfer.
Therefore, for cross-language transfer with an extremely low resource and cross-domain transfer, we suggest using DATNet-F model to achieve better performance.
As for cross-language transfer with relatively more training data, DATNet-P model is preferred.
[Figure]
4. A BLATION S TUDY OF DATN ET
In the proposed DATNet, both GRAD and AT play important roles in low resource NER.
In this experiment, we further investigate how GRAD and AT help transfer knowledge across language/domain.
In the ﬁrst experiment 6 , we used t-SNE (Maaten & Hinton, 2008) to visualize the feature distribution of BiLSTM outputs without AD, with normal AD (GRAD without considering data imbalance), and with the proposed GRAD in Figure 3. From this ﬁgure, we can see that
the GRAD in DATNet makes the distribution of extracted features from the source and target datasets much more similar by considering the data imbalance, which indicates that the outputs of BiLSTM are resource-invariant. [Table] To better understand the working mechanism, Table 6 further reports the quantitative performance comparison between models with different components.
We observe that GRAD shows the stable superiority over the normal AD regardless of other components.
There are no always winner between DATNet-P and DATNet-F on different settings.
DATNet-P architecture is more suitable to cross-language transfer while DATNet-F is more suitable to cross-domain transfer.
[Table]
From the previous results [Table] we know that AT helps enhance the overall performance by adding perturbations to inputs with the limit of (cid:15) = 5 , i
[Table]
e., (cid:107) η (cid:107) 2 ≤ 5 . In this experiment [Table] we further investigate how target perturbation (cid:15) w T with ﬁxed source perturbation (cid:15) w S = 5 in AT affects knowledge transfer and the results on Spanish NER are summarized in Table 7.
The results generally indicate that less training data require a larger (cid:15) to prevent over-ﬁtting, which further validates the necessity of
AT in the case of low resource data.
[Table]
Finally, we analyze the discriminator weight α in GRAD and results are summarized in Table 8.
From the results, it is interesting to ﬁnd that α is directly proportional to the data ratio ρ , basically, which means that more target training data requires larger α (i.e., smaller 1 − α to reduce training emphasis on the target domain) to achieve better performance.
C ONCLUSION
In this paper we develop a transfer learning model DATNet for low-resource NER, which aims at addressing two problems remained in existing work, namely representation difference and resource data imbalance.
We introduce two variants of DATNet, DATNet-F and DATNet-P, which can be chosen for use according to the cross-language/domain user case and the target dataset size.
To improve model generalization, we propose dual adversarial learning strategies, i.e., AT and GRAD.
Extensive experiments show the superiority of DATNet over existing models and it achieves new state-of-the-art performance on CoNLL NER and WNUT NER benchmark datasets.