Paper:
Recent papers have shown that neural networks obtain state-of-the-art performance on several different sequence tagging tasks. One appealing property of such systems is their generality, as excellent performance can be achieved with a unified architecture and without task-specific feature engineering. However, it is unclear if such systems can be used for tasks without large amounts of training data. In this paper we explore the problem of transfer learning for neural sequence taggers, where a source task with plentiful annotations (e.g., POS tagging on Penn Treebank) is used to improve performance on a target task with fewer available annotations (e.g., POS tagging for microblogs). We examine the effects of transfer learning for deep hierarchical recurrent networks across domains, applications, and languages, and show that significant improvement can often be obtained. These improvements lead to improvements over the current state-ofthe-art on several well-studied tasks.1
1 INTRODUCTION
Sequence tagging is an important problem in natural language processing, which has wide applications including part-of-speech (POS) tagging, text chunking, and named entity recognition (NER). Given a sequence of words, sequence tagging aims to predict a linguistic tag for each word such as the POS tag.
An important challenge for sequence tagging is how to transfer knowledge from one task to another, which is often referred to as transfer learning (Pan & Yang, 2010). Transfer learning can be used in several settings, notably for low-resource languages (Zirikly & Hagiwara, 2015; Wang & Manning, 2014) and low-resource domains such as biomedical corpora (Kim et al., 2003) and Twitter corpora (Ritter et al., 2011)). In these cases, transfer learning can improve performance by taking advantage of more plentiful labels from related tasks. Even on datasets with relatively abundant labels, multi-task transfer can sometimes achieve improvement over state-of-the-art results (Collobert et al., 2011).
Recently, a number of approaches based on deep neural networks have addressed the problem of sequence tagging in an end-to-end manner (Collobert et al., 2011; Lample et al., 2016; Ling et al., 2015; Ma & Hovy, 2016). These neural networks consist of multiple layers of neurons organized in a hierarchy and can transform the input tokens to the output labels without explicit hand-engineered feature extraction. The aforementioned neural networks require minimal assumptions about the task at hand and thus demonstrate significant generality—one single model can be applied to multiple applications in multiple languages without changing the architecture. A natural question is whether the representation learned from one task can be useful for another task. In other words, is there a way we can exploit the generality of neural networks to improve task performance by sharing model parameters and feature representations with another task?
To address the above question, we study the transfer learning setting, which aims to improve the performance on a target task by joint training with a source task. We present a transfer learning approach based on a deep hierarchical recurrent neural network, which shares the hidden feature repre-
1Code is available at https://github.com/kimiyoung/transfer
sentation and part of the model parameters between the source task and the target task. Our approach combines the objectives of the two tasks and uses gradient-based methods for efficient training. We study cross-domain, cross-application, and cross-lingual transfer, and present a parameter-sharing architecture for each case. Experimental results show that our approach can significantly improve the performance of the target task when the the target task has few labels and is more related to the source task. Furthermore, we show that transfer learning can improve performance over state-ofthe-art results even if the amount of labels is relatively abundant.
We have novel contributions in two folds. First, our work is, to the best of our knowledge, the first that focuses on studying the transferability of different layers of representations for hierarchical RNNs. Second, different from previous transfer learning methods that usually focus on one specific transfer setting, our framework exploits different levels of representation sharing and provides a unified framework to handle cross-application, cross-lingual, and cross-domain transfer.
2 RELATED WORK
There are two common paradigms for transfer learning for natural language processing (NLP) tasks, resource-based transfer and model-based transfer. Resource-based transfer utilizes additional linguistic annotations as weak supervision for transfer learning, such as cross-lingual dictionaries (Zirikly & Hagiwara, 2015), corpora (Wang & Manning, 2014), and word alignments (Yarowsky et al., 2001). Resource-based methods demonstrate considerable success in cross-lingual transfer, but are quite sensitive to the scale and quality of the additional resources. Resource-based transfer is mostly limited to cross-lingual transfer in previous works, and there is not extensive research on extending resource-based methods to cross-domain and cross-application settings.
Model-based transfer, on the other hand, does not require additional resources. Model-based transfer exploits the similarity and relatedness between the source task and the target task by adaptively modifying the model architectures, training algorithms, or feature representation. For example, Ando & Zhang (2005) proposed a transfer learning framework that shares structural parameters across multiple tasks, and improve the performance on various tasks including NER; Collobert et al. (2011) presented a task-independent convolutional neural network and employed joint training to transfer knowledge from NER and POS tagging to chunking; Peng & Dredze (2016) studied transfer learning between named entity recognition and word segmentation in Chinese based on recurrent neural networks. Cross-domain transfer, or domain adaptation, is also a well-studied branch of model-based transfer in NLP. Techniques in cross-domain transfer include the design of robust feature representations (Schnabel & Schütze, 2014), co-training (Chen et al., 2011), hierarchical Bayesian prior (Finkel & Manning, 2009), and canonical component analysis (Kim et al., 2015).
While our approach falls into the paradigm of model-based transfer, in contrast to the above methods, our method focuses on exploiting the generality of deep recurrent neural networks and is applicable to transfer between domains, applications, and languages.
Our work builds on previous work on sequence tagging based on deep neural networks. Collobert et al. (2011) develop end-to-end neural networks for sequence tagging without hand-engineered features. Later architectures based on different combinations of convolutional networks and recurrent networks have achieved state-of-the-art results on many tasks (Collobert et al., 2011; Huang et al., 2015; Chiu & Nichols, 2015; Lample et al., 2016; Ma & Hovy, 2016). These models demonstrate significant generality since they can be applied to multiple applications in multiple languages with a unified network architecture and without task-specific feature extraction.
3 APPROACH
In this section, we introduce our transfer learning approach. We first introduce an abstract framework for neural sequence tagging, summarizing previous work, and then discuss three different transfer learning architectures.
3.1 BASE MODEL
Though many different variants of neural networks have been proposed for the problem of sequence tagging, we find that most of the models can be described with the hierarchical framework illustrated in Figure 1(a). A character-level layer takes a sequence of characters (represented as embeddings) as input, and outputs a representation that encodes the morphological information at the character level. A word-level layer subsequently combines the character-level feature representation and a word embedding, and further incorporates the contextual information to output a new feature representation. After two levels of feature extraction (encoding), the feature representation output by the word-level layer is fed to a conditional random field (CRF) layer that outputs the label sequence.
Both of the word-level layer and the character-level layer can be implemented as convolutional neural networks (CNNs) or recurrent neural networks (RNNs) (Collobert et al., 2011; Chiu & Nichols, 2015; Lample et al., 2016; Ma & Hovy, 2016). We discuss the details of the model we use in this work in Section 3.4.
3.2 TRANSFER LEARNING ARCHITECTURES
We develop three architectures for transfer learning, T-A, T-B, and T-C, are illustrated in Figures 1(b), 1(c), and 1(d) respectively. The three architectures are all extensions of the base model discussed in the previous section with different parameter sharing schemes. We now discuss the use cases for the different architectures.
3.2.1 CROSS-DOMAIN TRANSFER
Since different domains are “sub-languages” that have domain-specific regularities, sequence taggers trained on one domain might not have optimal performance on another domain. The goal of cross-domain transfer is to learn a sequence tagger that transfers knowledge from a source domain to a target domain. We assume that few labels are available in the target domain.
There are two cases of cross-domain transfer. The two domains can have label sets that can be mapped to each other, or disparate label sets. For example, POS tags in the Genia biomedical corpus can be mapped to Penn Treebank tags (Barrett & Weber-Jahnke, 2014), while some POS tags in Twitter (e.g., “URL”) cannot be mapped to Penn Treebank tags (Ritter et al., 2011).
If the two domains have mappable label sets, we share all the model parameters and feature representation in the neural networks, including the word and character embedding, the word-level layer, the character-level layer, and the CRF layer. We perform a label mapping step on top of the CRF layer. This becomes the model T-A as shown in Figure 1(b).
If the two domains have disparate label sets, we untie the parameter sharing in the CRF layer—i.e., each task learns a separate CRF layer. This parameter sharing scheme reduces to model T-B as shown in Figure 1(c).
3.2.2 CROSS-APPLICATION TRANSFER
Sequence tagging has a couple of applications including POS tagging, chunking, and named entity recognition. Similar to the motivation in (Collobert et al., 2011), it is usually desirable to exploit the underlying similarities and regularities of different applications, and improve the performance of one application via joint training with another. Moreover, transfer between multiple applications can be helpful when the labels are limited.
In the cross-application setting, we assume that multiple applications are in the same language. Since different applications share the same alphabet, the case is similar to cross-domain transfer with disparate label sets. We adopt the architecture of model T-B for cross-application transfer learning where only the CRF layers are disjoint for different applications.
3.2.3 CROSS-LINGUAL TRANSFER
Though cross-lingual transfer is usually accomplished with additional multi-lingual resources, these methods are sensitive to the size and quality of the additional resources (Yarowsky et al., 2001; Wang & Manning, 2014). In this work, instead, we explore a complementary method that exploits the cross-lingual regularities purely on the model level.
Our approach focuses on transfer learning between languages with similar alphabets, such as English and Spanish, since it is very difficult for transfer learning between languages with disparate alphabets (e.g., English and Chinese) to work without additional resources (Zirikly & Hagiwara, 2015).
Model-level transfer learning is achieved through exploiting the morphologies shared by the two languages. For example, “Canada” in English and “Canadá” in Spanish refer to the same named entity, and the morphological similarities can be leveraged for NER and also POS tagging with nouns. Thus we share the character embeddings and the character-level layer between different languages for transfer learning, which is illustrated as the model T-C in Figure 1(d).
3.3 TRAINING
In the above sections, we introduced three neural architectures with different parameter sharing schemes, designed for different transfer learning settings. Now we describe how we train the neural networks jointly for two tasks.
Suppose we are transferring from a source task s to a target task t, with the training instances being Xs and Xt. Let Ws and Wt denote the set of model parameters for the source and target tasks respectively. The model parameters are divided into two sets, task specific parameters and shared parameters, i.e.,
Ws =Ws,spec ∪Wshared,Wt =Wt,spec ∪Wshared,
where shared parameters Wshared are jointly optimized by the two tasks, while task specific parameters Ws,spec and Wt,spec are trained for each task separately.
The training procedure is as follows. At each iteration, we sample a task (i.e., either s or t) from {s, t} based on a binomial distribution (the binomial probability is set as a hyperparameter). Given the sampled task, we sample a batch of training instances from the given task, and then perform a gradient update according to the loss function of the given task. We update both the shared parameters and the task specific parameters. We repeat the above iterations until stopping. We adopt AdaGrad (Duchi et al., 2011) to dynamically compute the learning rates for each iteration. Since the source and target tasks might have different convergence rates, we do early stopping on the target task performance.
3.4 MODEL IMPLEMENTATION
In this section, we describe our implementation of the base model. Both the character-level and word-level neural networks are implemented as RNNs. More specifically, we employ gated recurrent units (GRUs) (Cho et al., 2014). Let (x1,x2, · · · ,xT ) be a sequence of inputs that can be embeddings or hidden states of other layers. Let ht be the GRU hidden state at time step t. Formally, a GRU unit at time step t can be expressed as
rt = σ(Wrxxt +Wrhht−1)
zt = σ(Wzxxt +Wzhht−1)
h̃t = tanh(Whxxt +Whh(rt ht−1)) ht = zt ht−1 + (1− zt) h̃t,
where W ’s are model parameters of each unit, h̃t is a candidate hidden state that is used to compute ht, σ is an element-wise sigmoid logistic function defined as σ(x) = 1/(1 + e−x), and denotes element-wise multiplication of two vectors. Intuitively, the update gate zt controls how much the unit updates its hidden state, and the reset gate rt determines how much information from the previous hidden state needs to be reset. The input to the character-level GRUs is character embeddings, while the input to the word-level GRUs is the concatenation of character-level GRU hidden states and word embeddings. Both GRUs are bi-directional and have two layers.
Given an input sequence of words, the word-level GRUs and the character-level GRUs together learn a feature representation ht for the t-th word in the sequence, which forms a sequence h = (h1,h2, · · · ,hT ). Let y = (y1, y2, · · · , yT ) denote the tag sequence. Given the feature representation h and the tag sequence y for each training instance, the CRF layer defines the objective function to maximize based on a max-margin principle (Gimpel & Smith, 2010) as:
f(h,y)− log ∑
y′∈Y(h)
exp(f(h,y′) + cost(y,y′)),
where f is a function that assigns a score for each pair of h and y, and Y(h) denotes the space of tag sequences for h. The cost function cost(y,y′) is added based on the max-margin principle (Gimpel & Smith, 2010) that high-cost tags y′ should be penalized more heavily.
Our base model is similar to Lample et al. (2016), but in contrast to their model, we employ GRUs for the character-level and word-level networks instead of Long Short-Term Memory (LSTM) units, and define the objective function based on the max-margin principle. We note that our transfer learning framework does not make assumptions about specific model implementation, and could be applied to other neural architectures (Collobert et al., 2011; Chiu & Nichols, 2015; Lample et al., 2016; Ma & Hovy, 2016) as well.
4 EXPERIMENTS
4.1 DATASETS
We use the following benchmark datasets in our experiments: Penn Treebank (PTB) POS tagging, CoNLL 2000 chunking, CoNLL 2003 English NER, CoNLL 2002 Dutch NER, CoNLL 2002 Spanish NER, the Genia biomedical corpus (Kim et al., 2003), and a Twitter corpus (Ritter et al., 2011).
The statistics of the datasets are described in Table 1. We construct the POS tagging dataset with the instructions described in Toutanova et al. (2003). Note that as a standard practice, the POS tags are extracted from the parsed trees. For the CoNLL 2003 English NER dataset, we follow previous works (Collobert et al., 2011) to append one-hot gazetteer features to the input of the CRF layer for fair comparison. Since there is no standard training/dev/test data split for the Genia and Twitter corpora, we randomly sample 10% for test, 10% for development, and 80% for training. We follow previous work (Barrett & Weber-Jahnke, 2014) to map Genia POS tags to PTB POS tags.
4.2 TRANSFER LEARNING PERFORMANCE
We evaluate our transfer learning approach on the above datasets. We fix the hyperparameters for all the results reported in this section: we set the character embedding dimension at 25, the word embedding dimension at 50 for English and 64 for Spanish, the dimension of hidden states of the character-level GRUs at 80, the dimension of hidden states of the word-level GRUs at 300, and the initial learning rate at 0.01. Except for the Twitter datasets, these datasets are fairly large. To simulate a low-resource setting, we also use random subsets of the data. We vary the labeling rate of the target task at 0.001, 0.01, 0.1 and 1.0. Given a labeling rate r, we randomly sample a ratio r of the sentences from the training set and discard the rest of the training data—e.g., a labeling rate of 0.001 results in around 900 training tokens on PTB POS tagging (Cf. Table 1).
The results on transfer learning are plotted in Figure 2, where we compare the results with and without transfer learning under various labeling rates. The numbers in the y-axes are accuracies for POS tagging, and chunk-level F1 scores for chunking and NER. The numbers are shown in Table 2. We can see that our transfer learning approach consistently improved over the non-transfer results. We also observe that the improvement by transfer learning is more substantial when the labeling rate is lower. For cross-domain transfer, we obtained substantial improvement on the Genia and Twitter corpora by transferring the knowledge from PTB POS tagging and CoNLL 2003 NER. For example, as shown in Figure 2(a), we can obtain an tagging accuracy of 83%+ with zero labels and 92% with only 0.001 labels when transferring from PTB to Genia. As shown in Figures 2(d) and 2(e), our transfer learning approach can improve the performance on Twitter POS tagging and NER for all labeling rates, and the improvements with 0.1 labels are more than 8% for both datasets. Cross-application transfer also leads to substantial improvement under low-resource conditions. For
example, as shown in Figures 2(g) and 2(h), the improvements with 0.1 labels are 6% and 3% on CoNLL 2000 chunking and CoNLL 2003 NER respectively when transferring from PTB POS tagging. Figures 2(j) and 2(i) show that cross-lingual transfer can improve the performance when few labels are available.
Figure 2 further shows that the improvements by different architectures are in the following order: T-A > T-B > T-C. This phenomenon can be explained by the fact that T-A shares the most model parameters while T-C shares the least. Transfer settings like cross-lingual transfer can only use T-C because the underlying similarities between the source task and the target task are less prominent (i.e., less transferable), and in those cases the improvement by transfer learning is less substantial.
Another interesting comparison is among Figures 2(a), 2(b), and 2(c). Figure 2(a) is cross-domain transfer, Figure 2(b) is transfer across domains and applications at the same time, and Figure 2(c) combines all the three transfer settings (i.e., from Spanish NER in the general domain to English POS tagging in the biomedical domain). The results show that the improvement by transfer learning diminishes when the transfer becomes “indirect” (i.e., the source task and the target task are more loosely related).
We also study using different transfer learning models for the same task. We study the effects of using T-A, T-B, and T-C when transferring from PTB to Genia, and the results are included in the lower part of Table 2. We observe that the performance gain decreases when less parameters are shared (i.e., T-A > T-B > T-C).
4.3 COMPARISON WITH STATE-OF-THE-ART RESULTS
In the above section, we examine the effects of different transfer learning architectures. Now we compare our approach with state-of-the-art systems on these datasets.
We use publicly available pretrained word embeddings as initialization. On the English datasets, following previous works that are based on neural networks (Collobert et al., 2011; Huang et al., 2015; Chiu & Nichols, 2015; Ma & Hovy, 2016), we experiment with both the 50-dimensional SENNA embeddings (Collobert et al., 2011) and the 100-dimensional GloVe embeddings (Pennington et al., 2014) and use the development set to choose the embeddings for different tasks and settings. For Spanish and Dutch, we use the 64-dimensional Polyglot embeddings (Al-Rfou et al., 2013). We set the hidden state dimensions to be 300 for the word-level GRU. The initial learning rate for AdaGrad is fixed at 0.01. We use the development set to tune the other hyperparameters of our model.
Our results are reported in Table 3. Since there are no standard data splits on the Genia and Twitter corpora, we do not include these datasets into our comparison. The results for CoNLL 2000 chunking, CoNLL 2003 NER, and PTB POS tagging are obtained by transfer learning between the three tasks, i.e., transferring from two tasks to the other. The results for Spanish and Dutch NER are obtained with transfer learning between the NER datasets in three languages (English, Spanish, and Dutch). From Table 3, we can draw two conclusions. First, our transfer learning approach achieves new state-of-the-art results on all the considered benchmark datasets except PTB POS tagging, which indicates that transfer learning can still improve the performance even on datasets with
relatively abundant labels. Second, our base model (w/o transfer) performs competitively compared to the state-of-the-art systems, which means that the improvements shown in Section 4.2 are obtained over a strong baseline.
5 CONCLUSION
In this paper we develop a transfer learning approach for sequence tagging, which exploits the generality demonstrated by deep neural networks in previous work. We design three neural network architectures for the settings of cross-domain, cross-application, and cross-lingual transfer. Our transfer learning approach achieves significant improvement on various datasets under low-resource conditions, as well as new state-of-the-art results on some of the benchmarks. With thorough experiments, we observe that the following factors are crucial for the performance of our transfer learning approach: a) label abundance for the target task, b) relatedness between the source and target tasks, and c) the number of parameters that can be shared. In the future, it will be interesting to combine model-based transfer (as in this work) with resource-based transfer for cross-lingual transfer learning.
ACKNOWLEDGMENTS
This work was funded by NVIDIA, the Office of Naval Research grant N000141512791, the ADeLAIDE grant FA8750-16C-0130-001, the NSF grant IIS1250956, and Google Research.

Review:
Authors' response well answered my questions. Thanks!
Evaluation not changed.
###
This paper proposes a hierarchical framework of transfer learning for sequence tagging, which is expected to help the target task with the source task, by sharing as many levels of representation as possible. It is a general framework for various neural models. The paper has extensive and solid experiments, and the performance is competitive with the state of the art on multiple benchmark datasets. The framework is clear by itself, except that more details about training procedure, i.e. sec-3.3, need to be added. 
The experimental results show that for some task pairs {s,t}, this framework can help low-resource target task t, and the improvement increases with more levels of representations can be shared. Firstly, I suggest that the terms *source* and *target* should be more precisely defined in the current framework, because, due to Sec-3.3, the s and t in each pair are sort of interchangeable. That is, either of them can be the *source* or *target* task, especially when p(X=s)=p(X=t)=0.5 is used in the task sampling. The difference is: one is low-resourced and the other is not. Thus it could be thought of as multi-tasking between tasks with imbalanced resource. So one question is: does this framework simultaneously help both tasks in the pair, by learning more generalizable representations for different domains/applications/languages? Or is it mostly likely to only help the low-resourced one? Does it come with sacrifice on the high-resourced side? 
Secondly, as the paper shows that the low-resourced tasks are improved for the selected task pairs, it would also be interesting and helpful to know how often this could happen. That is, when the tasks are randomly paired (one chosen from a low-resource pool and the other from a high resource pool), how often could this framework help the low-resourced one?
Moreover, the choice of T-A/T-B/T-C lies intuitively in how many levels of representation *could* be shared as possible. This implicitly assumes share more, help more. Although I tend to believe so, it would be interesting to have some empirical comparison. For example, one could perhaps select some cross-domain pair, and see if T-A > T-B > T-C on such pairs, as mentioned in the author’s answer to the pre-review question. 
In general, I think this is a solid paper, and more exploration could be done in this direction. So I tend to accept this paper.