T RANSFER L EARNING FOR S EQUENCES VIA L EARNING TO C OLLOCATE
A BSTRACT
Transfer learning aims to solve the data sparsity for a target domain by applying information of the source domain.
Given a sequence (e.g. a natural language sentence), the transfer learning, usually enabled by recurrent neural network (RNN), represents the sequential information transfer.
RNN uses a chain of repeating cells to model the sequence data.
However, previous studies of neural network based transfer learning simply represents the whole sentence by a single vector, which is unfeasible for seq2seq and sequence labeling.
Meanwhile, such layer-wise transfer learning mechanisms lose the ﬁne-grained cell-level information from the source domain.
In this paper, we proposed the aligned recurrent transfer, ART, to achieve celllevel information transfer.
ART is in a recurrent manner.
Each cell attentively accepts transferred information from a set of positions in the source domain.
Therefore, ART learns the cross-domain word collocations in a more ﬂexible way.
We conducted extensive experiments on both sequence labeling tasks (POS tagging, NER) and sentence classiﬁcation (sentiment analysis).
ART outperforms the state-of-the-arts over all experiments.
I NTRODUCTION
Most previous NLP studies focus on open domain tasks.
But due to the variety and ambiguity of natural language (Glorot et al., 2011; Song et al., 2011), models for one domain usually incur more errors when adapting to another domain.
This is even worse for neural networks since embeddingbased neural network models usually suffer from overﬁtting (Peng et al., 2015).
While existing NLP models are usually trained by the open domain, they suffer from severe performance degeneration when adapting to speciﬁc domains.
This motivates us to train speciﬁc models for speciﬁc domains.
The key issue of training a speciﬁc domain is the insufﬁciency of labeled data.
Transfer learning is one promising way to solve the insufﬁciency (Jiang & Zhai, 2007).
Existing studies (Daum´e III, 2007; Jiang & Zhai, 2007) have shown that (1) NLP models in different domains still share many common features (e.g. common vocabularies, similar word semantics, similar sentence syntaxes), and (2) the corpus of the open domain is usually much richer than that of a speciﬁc domain.
Our transfer learning model is under the pre-training framework.
We ﬁrst pre-train the model for the source domain.
Then we ﬁne-tune the model for the target domain.
Recently, some pre-trained models (e.g. BERT (Devlin et al., 2018), ELMo (Peters et al., 2018), {+GPT-
+} (Radford et al., 2019)) successfully learns general knowledge for text.
The difference is that these models use a large scale and domain-independent corpus for pre-training.
In this paper, we use a small scale but domaindependent corpus as the source domain for pre-training.
We argue that, for the pre-training corpus, the domain relevance will overcome the disadvantage of limited scale.
Most previous transfer learning approaches (Li et al., 2018; Ganin et al., 2016) only transfer information across the whole layers.
This causes the information loss from cells in the source domain.
‘‘Layer-wise transfer learning” indicates that the approach represents the whole sentence by a single vector.
So the transfer mechanism is only applied to the vector.
We highlight the effectiveness of precisely capturing and transferring information of each cell from the source domain in two cases.
First, in seq2seq (e.g. machine translation) or sequence labeling (e.g. POS tagging) tasks, all cells directly affect the results.
So layer-wise information transfer is unfeasible for these tasks.
Second, even for the sentence classiﬁcation, cells in the source domain provide more ﬁne-grained information to understand the target domain.
For example, in ﬁgure 1, parameters for “hate” are insufﬁciently trained.
The model transfers the state of “hate” from the source domain to understand it better.
[Figure]
Besides transferring the corresponding position’s information, the transfer learning algorithm captures the cross-domain long-term dependency .
Two words that have a strong dependency on each other can have a long gap between them.
Being in the insufﬁciently trained target domain, a word needs to represent its precise meaning by incorporating the information from its collocated words.
Here “collocate” indicates that a word’s semantics can have long-term dependency on other words.
To understand a word in the target domain, we need to precisely represent its collocated words from the source domain.
We learn the collocated words via the attention mechanism (Bahdanau et al., 2015).
For example, in ﬁgure 1, “hate” is modiﬁed by the adverb “sometimes”, which implies the act of hating is not serious.
But the “sometimes” in the target domain is trained insufﬁciently.
We need to transfer the semantics of “sometimes” in the source domain to understand the implication.
Therefore, we need to carefully align word collocations between the source domain and the target domain to represent the long-term dependency.
In this paper, we proposed ART (aligned recurrent transfer), a novel transfer learning mechanism, to transfer cell-level information by learning to collocate cross-domain words.
ART allows the celllevel information transfer by directly extending each RNN cell.
ART incorporates the hidden state representation corresponding to the same position and a function of the hidden states for all words weighted by their attention scores.
Cell-Level Recurrent Transfer ART extends each recurrent cell by taking the states from the source domain as an extra input.
While traditional layer-wise transfer learning approaches discard states of the intermediate cells, ART uses cell-level information transfer, which means each cell is affected by the transferred information.
For example, in ﬁgure 1, the state of “hate” in the target domain is affected by “sometimes” and “hate” in the source domain.
Thus ART transfers more ﬁne-grained information.
Learn to Collocate and Transfer For each word in the target domain, ART learns to incorporate two types of information from the source domain: (a) the hidden state corresponding to the same word, and (b) the hidden states for all words in the sequence.
Information (b) enables ART to capture the cross-domain long-term dependency.
ART learns to incorporate information (b) based on the attention scores (Bahdanau et al., 2015) of all words from the source domain.
Before learning to transfer, we ﬁrst pre-train the neural network of the source domain.
Therefore ART is able to leverage the pre-trained information from the source domain.
A LIGNED R ECURRENT T RANSFER
In this section, we elaborate the general architecture of ART.
We will show that, ART precisely learns to collocate words from the source domain and to transfer their cell-level information for the target domain.
Architecture The source domain and the target domain share an RNN layer, from which the common information is transferred.
We pre-train the neural network of the source domain.
Therefore the shared
RNN layer represents the semantics of the source domain.
The target domain has an additional
RNN layer.
Each cell in it accepts transferred information through the shared RNN layer.
Such information consists of (1) the information of the same word in the source domain (the red edge in ﬁgure 2); and (2) the information of all its collocated words (the blue edges in ﬁgure 2).
ART uses attention (Bahdanau et al., 2015) to decide the weights of all candidate collocations.
The RNN cell controls the weights between (1) and (2) by an update gate.
Figure 2 shows the architecture of ART.
The yellow box contains the neural network for the source domain, which is a classical RNN.
The green box contains the neural network for the target domain.
S i and T i are cells for the source domain and target domain, respectively.
T i takes x i as the input, which is usually a word embedding.
The two neural networks overlap each other.
The source domain’s neural network transfers information through the overlapping modules.
We will describe the details of the architecture below.
Note that although ART is only deployed over RNNs in this paper, its attentive transfer mechanism is easy to be deploy over other structures (e.g. Transformer (Vaswani et al., 2017))
[Figure]
RNN for the Source Domain The neural network of the source domain consists of a standard
RNN.
Each cell S i captures information from the previous time step h Si − 1 , computes and passes the information h
Si to the next time step.
More formally, we deﬁne the performance of S i as:
[Equation]
(1) where θ S is the parameter (recurrent weight matrix).
Information Transfer for the Target Domain Each RNN cell in the target domain leverages the transferred information from the source domain.
Different from the source domain, the i -th hidden state in the target domain h Ti is computed by:
[Equation]
where (cid:103) h Ti − 1 contains the information passed from the previous time step in the target domain ( h Ti − 1 ), and the transferred information from the source domain ( ψ i ).
We compute it by:
[Equation]
where θ f is the parameter for f .
Note that both domains use the same RNN function with different parameters ( θ S and θ T ).
Intuitively, we always want to transfer the common information across domains.
And we think it’s easier to represent common and shareable information with an identical network structure.
Learn to Collocate and Transfer We compute ψ i by aligning its collocations in the source domain.
We consider two kinds of alignments: (1) The alignment from the corresponding position.
This makes sense since the corresponding position has the corresponding information of the source domain.
The alignments from all collocated words of the source domain.
This alignment is used to represent the long-term dependency across domains.
We use a “concentrate gate” u i to control the ratio between the corresponding position and collocated positions.
We compute ψ i by:
[Equation]
where
[Equation]
π i denotes the transferred information from collocated words.
◦ denotes the element-wise multiplication.
W u and C u are parameter matrices.
In order to compute π i , we use attention (Bahdanau et al., 2015) to incorporate information of all candidate positions in the sequence from the source domain.
We denote the strength of the collocation intensity to position j in the source domain as α ij .
We merge all information of the source domain by a weighted sum according to the collocation intensity.
More speciﬁcally, we deﬁne π i as:
[Equation]
where
[Equation]
a ( h Ti , h Sj ) denotes the collocation intensity between the i -th cell in the target domain and the j -th cell in the source domain.
The model needs to be evaluated O ( n 2 ) times for each sentence, due to the enumeration of n indexes for the source domain and n indexes for the target domain.
Here n denotes the sentence length.
By following Bahdanau et al.
, we use a single-layer perception:
[Equation]
where W a and U a are the parameter matrices.
Since U a h
Sj does not depend on i , we can pre-compute it to reduce the computation cost.
Update New State To compute f , we use an update gate z i to determine how much of the source domain’s information (cid:101) ψ
i should be transferred.
(cid:101) ψ
i is computed by merging the original input x i , the previous cell’s hidden state
h Ti − 1 and the transferred information ψ i .
We use a reset gate r i to determine how much of h Ti − 1 should be reset to zero for (cid:101)
ψ i .
More speciﬁcally, we deﬁne f as:
[Equation]
where
[Equation]
Here these W, U, C are parameter matrices.
Model Training:
We ﬁrst pre-train the parameters of the source domain by its training samples.
Then we ﬁne-tune the pre-trained model with additional layers of the target domain.
The ﬁne-tuning uses the training samples of the target domain.
All parameters are jointly ﬁne-tuned.
ART OVER LSTM
In this section, we illustrate how we deploy ART over LSTM.
LSTM speciﬁcally addresses the issue of learning long-term dependency in RNN.
Instead of using one hidden state for the sequential memory, each LSTM cell has two hidden states for the long-term and short-term memory.
So the
ART adaptation needs to separately represent information for the long-term memory and short-term memory.
The source domain of ART over LSTM uses a standard LSTM layer.
The computation of the t -th cell is precisely speciﬁed as follows:
[Equation] [Equation]
[Equation]
Here h St and c St denote the short-term memory and long-term memory, respectively.
In the target domain, we separately incorporate the short-term and long-term memory from the source domain.
More formally, we compute the t -the cell in the target domain by:
[Equation]
where f ( h Ti − 1 , ψ hi | θ fh ) and f ( c Ti − 1 , ψ ci | θ fc ) are computed by Eq (6) with parameters θ fh and θ fc , respectively.
ψ hi and ψ ci are the transferred information from the short-term memory ( h S 1 · · · h Sn in
Eq.
and the long-term memory ( c S 1 · · · c Sn in Eq.
(13)), respectively.
Bidirectional Network
We use the bidirectional architecture to reach all words’ information for each cell.
The backward neural network accepts the x i ( i = 1 . . . n ) in reverse order.
We compute the ﬁnal output of the ART over LSTM by concatenating the states from the forward neural network and the backward neural network for each cell.
E XPERIMENTS
We evaluate our proposed approach over sentence classiﬁcation (sentiment analysis) and sequence labeling task (POS tagging and NER).
4. S ETUP
All the experiments run over a computer with Intel Core i7 4.0GHz CPU, 32GB RAM, and a GeForce
GTX 1080 Ti GPU.
Network Structure: We use a very simple network structure.
The neural network consists of an embedding layer, an ART layer as described in section 3, and a task-speciﬁc output layer for the prediction.
We will elaborate the output layer and the loss function in each of the tasks below.
We use 100d GloVe vectors (Pennington et al., 2014) as the initialization for ART and all its ablations.
Competitor Models We compare ART with the following ablations
• LSTM (no transfer learning):
It uses a standard LSTM without transfer learning.
It is only trained by the data of the target domain.
• LSTM-u
It uses a standard LSTM and is trained by the union data of the source domain and the target domain.
• LSTM-s
It uses a standard LSTM and is trained only by the data of the source domain.
Then parameters are used to predicting outputs of samples in the target domain.
• Layer-Wise Transfer (LWT) (no cell-level information transfer): It consists of a layerwise transfer learning neural network.
More speciﬁcally, only the last cell of the RNN layer transfers information.
This cell works as in ART.
LWT only works for sentence classiﬁcation.
We use LWT to verify the effectiveness of the cell-level information transfer.
• Corresponding Cell Transfer (CCT) (no collocation information transfer): It only transfers information from the corresponding position of each cell.
We use CCT to verify the effectiveness of collocating and transferring from the source domain.
We also compare ART with state-of-the-art transfer learning algorithms.
For sequence labeling, we compare with hierarchical recurrent networks (HRN) (Yang et al., 2017) and FLORS (Schnabel & Sch¨utze, 2014; Yin, 2015).
For sentence classiﬁcation, we compare with DANN (Ganin et al., 2016), DAmSDA (Ganin et al., 2016), AMN (Li et al., 2017), and HATN (Li et al., 2018).
Note that FLORS, DANN, DAmSDA, AMN and HATN use labeled samples of the source domain and unlabeled samples of both the source domain and the target domain for training.
Instead, ART and
HRN
4. S ENTENCE C LASSIFICATION : S ENTIMENT A NALYSIS
Datasets: We use the Amazon review dataset (Blitzer et al., 2007), which has been widely used for cross-domain sentence classiﬁcation.
It contains reviews for four domains: books, dvd, electronics, kitchen .
Each review is either positive or negative.
We list the detailed statistics of the dataset in
Table 1.
We use the training data and development data from both domains for training and validating.
And we use the testing data of the target domain for testing.
Avg.
L denotes the average length of review.
Vocab.
denotes the vocabulary size
[Table]
Model Details: To adapt ART to sentence classiﬁcation, we use a max pooling layer to merge the states of different words.
Then we use a perception and a sigmoid function to score the probability of the given sentence being positive.
We use binary cross entropy as the loss function.
The dimension of each LSTM is set to 100.
We use the Adam (Kingma & Ba, 2015) optimizer.
We use a dropout probability of 0.5 on the max pooling layer.
[Table] [Table]
[Table] [Table]
[Table]
Results: We report the classiﬁcation accuracy of different models in Table 2.
The no-transfer LSTM only performs accuracy of 76 .
3% on average.
ART outperforms it by 9 . 5% .
ART also outperforms its ablations and other competitors.
This overall veriﬁes the effectiveness of ART.
Effectiveness of Cell-Level Transfer LWT only transfers layer-wise information and performs accuracy of 77 .
4% on average.
But ART and CCT transfer more ﬁne-grained cell-level information.
CCT outperforms LWT by 2 .
ART outperforms LWT by 8 . 4% .
This veriﬁes the effectiveness of the cell-level transfer.
Effectiveness of Collocation and Transfer CCT only transfers the corresponding position’s information from the source domain.
It achieves accuracy of 80 .
3% on average.
ART outperforms
CCT by 5 . 5% on average.
ART provides a more ﬂexible way to transfer a set of positions in the source domain and represent the long-term dependency.
This veriﬁes the effectiveness of ART in representing long-term dependency by learning to collocate and transfer.
Minimally Supervised Domain Adaptation We also evaluate ART when the number of training samples for the target domain is much fewer than that of the source domain.
For each target domain in the Amazon review dataset, we combine the training/development data of rest three domains as the source domain.
We show the results in Table 3.
ART outperforms all the competitors by a large margin.
This veriﬁes its effectiveness in the setting of minimally supervised domain adaptation.
[Table]
4. S EQUENCE L ABELING
We evaluate the effectiveness of ART w.r.t.
sequence labeling.
We use two typical tasks: POS tagging and NER (named entity recognition).
The goal of POS tagging is to assign part-of-speech tags to each word of the given sentence.
The goal of NER is to extract and classify the named entity in the given sentence.
Model Settings: POS tagging and NER are multi-class labeling tasks.
To adapt ART to them, we follow HRN (Yang et al., 2017) and use a CRF layer to compute the tag distribution of each word.
We predict the tag with maximized probability for each word.
We use categorical cross entropy as the loss function.
The dimension of each LSTM cell is set to 300.
By following HRN, we use the concatenation of 50d word embeddings and 50d character embeddings as the input of the ART layer.
We use 50 1d ﬁlters for CNN char embedding, each with a width of 3.
We use the Adagrad (Duchi et al., 2011) optimizer.
We use a dropout probability of 0.5 on the max pooling layer.
Dataset: We use the dataset settings as in Yang et al.
For POS Tagging, we use Penn
Treebank (PTB) POS tagging, and a Twitter corpus (Ritter et al., 2011) as different domains.
For
NER, we use CoNLL 2003 (Tjong Kim Sang & De Meulder, 2003) and Twitter (Ritter et al., 2011) as different domains.
Their statistics are shown in Table 4.
By following Ritter et al.
(2011), we use 10% training samples of Twitter (Twitter/0.1), 1% training samples of Twitter (Twitter/0.01), and 1% training samples of CoNLL (CoNLL/0.01) as the training data for the target domains to simulate a low-resource setting.
Note that the label space of these tasks are different.
So some baselines (e.g.
LSTM-u, LSTM-s) cannot be applied.
[Table]
[Table]
Table 5 shows the per-word accuracy (for POS tagging) and f1 (for NER) of different models.
From the table, we see that the performances of all tasks are improved by ART.
For example, when transferring from PTB to Twitter/0.1, ART outperforms HRN by 2.2%.
ART performs the best among all competitors in almost all cases.
This veriﬁes the effectiveness of ART w.r.t.
sequence labeling.
Note that FLORS is independent of the target domain.
If the training corpus of the target domain is quite rare (Twitter/0.01), FLORS performs better.
But with richer training data of the target domain (Twitter/0.1), ART outperforms FLORS.
4. V ISUALIZATION OF THE A LIGNED T RANSFER
ART aligns and transfers information from different positions in the source domain.
Intuitively, we use the alignment and attention matrices to represent cross-domain word dependencies.
So positions with stronger dependencies will be highlighted during the transfer.
We visualize the attention matrix for sentiment analysis to verify this.
We show the attention matrices for the short-term memory h and for the long-term memory c in ﬁgure 3.
[Figure]
Effectiveness of the Cell-Level Transfer From ﬁgure 3, words with stronger emotions have more attentions.
For example, the word “pleased” for the long-term memory and “easy to move” for the short-term memory have strong attention for almost all words, which sits well with the intuition of cell-level transfer.
The target domain surely wants to accept more information from the meaningful words in the source domain, not from the whole sentence.
Notice that c
and h have different attentions.
Thus the two attention matrices represent discriminative features.
Effectiveness in Representing the Long-Term Dependency We found that the attention reﬂects the long-term dependency.
For example, in ﬁgure 3 (b), although all words in the target domain are affected by the word “easy”, the word “very” has highest attention.
This makes sense because “very” is actually the adverb for “easy”, although they are not adjacent.
ART highlights cross-domain word dependencies and therefore gives a more precise understanding of each word.
R ELATED WORK
Neural network-based transfer learning The layer-wise transfer learning approaches (Glorot et al., 2011; Ajakan et al., 2014; Zhou et al., 2016) represent the input sequence by a non-sequential vector.
These approaches cannot be applied to seq2seq or sequence labeling tasks.
To tackle this problem, algorithms must transfer cell-level information in the neural network (Yang et al., 2017).
Some approaches use RNN to represent cell-level information.
Ying et al.
(2017) trains the RNN layer by domain-independent auxiliary labels.
Ziser & Reichart (2018) trains the RNN layer with pivots.
However, the semantics of a word can depend on its collocated words.
These approaches cannot represent the collocated words.
In contrast, ART successfully represents the collocations by attention.
Pre-trained models ART uses a pre-trained model from the source domain, and ﬁne-tunes the model with additional layers for the target domain.
Recently, pre-trained models with additional layers are shown to be effectiveness for many downstream models (e.g. BERT (Devlin et al., 2018),
ELMo (Peters et al., 2018)).
As a pre-trained model, ELMo uses bidirectional LSTMs to generate contextual features.
Instead, ART uses attention mechanism in RNN that each cell in the target domain directly access information of all cells in the source domain.
ART and these pre-trained models have different goals.
ART aims at transfer learning for one task in different domains, while
BERT and ELMo focus on learning general word representations or sentence representations.
C ONCLUSION
In this paper, we study the problem of transfer learning for sequences.
We proposed the ART model to collocate and transfer cell-level information.
ART has three advantages: (1) it transfers more ﬁne-grained cell-level information, and thus can be adapted to seq2seq or sequence labeling tasks; (2) it aligns and transfers a set of collocated words in the source sentence to represent the cross domain long-term dependency; (3) it is general and can be applied to different tasks.
Besides, ART veriﬁed the effectiveness of pre-training models with the limited but relevant training corpus.
A CKNOWLEDGMENTS
Guangyu Zheng and Wei Wang were supported by Shanghai Software and Integrated Circuit Industry
Development Project(170512).