L IFELONG W ORD E MBEDDING VIA M ETA -L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning high-quality word embeddings is of signiﬁcant importance in achieving better performance in many down-stream learning tasks.
On one hand, traditional word embeddings are trained on a large scale corpus for general-purpose tasks, which are often sub-optimal for many domain-speciﬁc tasks.
On the other hand, many domain-speciﬁc tasks do not have a large enough domain corpus to obtain high-quality embeddings.
We observe that domains are not isolated and a small domain corpus can leverage the learned knowledge from many past domains to augment that corpus in order to generate high-quality embeddings.
In this paper, we formulate the learning of word embeddings as a lifelong learning process.
Given knowledge learned from many previous domains and a small new domain corpus, the proposed method can effectively generate new domain embeddings by leveraging a simple but effective algorithm and a meta-learner, where the metalearner is able to provide word context similarity information at the domain-level.
Experimental results demonstrate that the proposed method can effectively learn new domain embeddings from a small corpus and past domain knowledges 1 .
We also demonstrate that general-purpose embeddings trained from a large scale corpus are sub-optimal in domain-speciﬁc tasks.
I NTRODUCTION
Learning word embeddings (Mnih & Hinton (2007); Turian et al.
Mikolov et al.
(2013a;b;c);
Pennington et al.
(2014)) has received a signiﬁcant amount of attention due to its high performance on many down-stream learning tasks.
Word embeddings have been shown effective in NLP tasks such as named entity recognition (Sien ˇ cnik (2015)), sentiment analysis (Maas et al.
and syntactic parsing (Durrett & Klein (2015)).
Such embeddings are shown to effectively capture syntactic and semantic level information associated with a given word (Mikolov et al.
(2013a)).
The “secret sauce” of training word embedding is to turn a large scale in-domain corpus into billions of training examples.
There are two common assumptions for training word embeddings: 1) the training corpus is largely available and bigger than the training data of the potential downstream learning tasks; and 2) the topic of the training corpus is closely related to the topic of the down-stream learning tasks.
However, real-world learning tasks often do not meet one of these assumptions.
For example, a domain-speciﬁc corpus that is closely related to a down-stream learning task may often be of limited size.
If we lump different domain corpora together and train general-purpose embeddings over a large scale corpus (e.g., GloVe embeddings (Pennington et al.
(2014)) are trained from the corpus Common Crawl, which covers almost any topic on the web), the performance of such embeddings on many domain-speciﬁc tasks is sub-optimal (we show this in Section 6).
A possible explanation is that although many domain words share similar meanings with the same out-of-domain words, with no in-domain awareness, dumping many out-of-domain co-occurrences as training examples may bias in-domain embeddings.
(e.g., if the domain is about food, then an out-of-domain “python” as a programming language can bias “java”, while the indomain word “chocolate” is more likely to help).
To solve the problem of the limited domain corpus, one possible solution is to use transfer learning (Pan & Yang (2010)) for training domain-speciﬁc embeddings (Bollegala et al.
; Yang et al.
However, these methods just manage to leverage out-of-domain embeddings trained from a large scale corpus to help limited in-domain corpus.
The very in-domain corpus is never expanded. Also, one common assumption of these works is that a pair of similar source domain and target domain is manually identiﬁed in advance.
In reality, given many domains, manually catching useful information in so many domains are very hard.
In contrast, we humans learn the meaning of a word more smartly.
We accumulate different domain contexts for the same word.
When a new learning task comes, we may quickly identify the new domain contexts and borrow the word meanings from existing domain contexts.
This is where lifelong learning comes to the rescue.
Lifelong machine learning (LML) is a continual learning paradigm that retains the knowledge learned in past tasks 1 , . . . , n , and uses it to help learning the new task n + 1 (Thrun (1996);
Silver et al.
; Chen & Liu (2016)).
In the setting of word embedding: we assume that the learning system has seen n domain corpora: ( D 1 , . . .
, D n ) , when a new domain corpus D n +1 comes by demands from that domain’s potential down-stream learning tasks, the learning system can automatically generate word embeddings for the n + 1 -th domain by effectively leveraging useful past domain knowledge.
The main challenges of this task are 2 fold.
1) How to identify useful past domain knowledge to train the embeddings for the new domain.
2) How to automatically identify such kind of information, without help from human beings.
To tackle these challenges, the system has to learn how to identify similar words in other domains for a given word in a new domain.
This, in general, belongs to metalearning (Vilalta & Drissi (2002); Peng et al.
Here we do not focus on speciﬁc embedding learning but focus on learning how to characterize corpora of different domains for embedding purpose.
The main contributions of this paper can be summarized as follows: 1) we propose the problem of lifelong word embedding, which may beneﬁt many down-stream learning tasks.
We are not aware of any existing work on word embedding using lifelong learning 2)
we propose a lifelong embedding learning method, which leverages meta-learning to aggregate useful knowledge from past domain corpora to generate embeddings for the new domain.
R ELATED W ORKS
Learning word embeddings has been studied for a long time (Mnih & Hinton (2007)).
Many earlier methods employ complex neural network architectures (Collobert & Weston (2008); Mikolov et al.
(2013c)).
Recently, a simple and effective unsupervised model called skip-gram (Mikolov et al.
(2013b;c)) was proposed to turn plain text corpus into large-scale training examples without any human annotation.
It uses the current word to predict the surrounding words in a context window by maximizing the likelihood of the predictions.
The learned parameters for each word are then the embeddings of that word.
Although such embeddings can be trained in large scale and easily obtained online (Pennington et al.
; Bojanowski et al.
, they are sub-optimal for many domain-speciﬁc tasks (Bollegala et al.
; Yang et al.
Domain corpus also suffers from limited size to train high-quality embeddings.
Our work is most related to Lifelong Machine Learning (LML))
(or lifelong learning).
Much of the work on LML focused on supervised learning (Thrun (1996); Silver et al.
; Ruvolo &
Eaton (2013); Chen & Liu (2016))
Recent years, several works have also been done in the unsupervised setting, mainly on topic modeling (Chen & Liu (2014)), information extraction (Mitchell et al.
and graph labeling (Shu et al.
However, we are not aware of any existing research that has been done on using lifelong learning for word embedding.
LML is related to transfer learning and multi-task learning (Pan & Yang (2010)), which have been leveraged in word embeddings (Bollegala et al.
; Yang et al.
However, LML is different from transfer learning (see the survey book from Chen & Liu (2016)).
Given many domains with uncertain noise for the new domain, the lack of guidance on which kind of information is worth learning from the past domains is a problem.
And there’s no good measure of similarity of two words in different domains.
The proposed method leverages meta-learning (Vilalta & Drissi (2002)), which is to perform machine learning on learning tasks.
Recently, meta-learning (or learning to learn) has been used to learn parameters of an optimizer (Andrychowicz et al.
to learn neural architectures (Fernando et al.
We leverage a meta-learner to accumulate knowledge during lifelong learning.
M ODEL O VERVIEW
The overall lifelong learning process is depicted in Figure 1.
Given a series of domain corpora D n = d 1 , d 2 , . . .
, d n , the lifelong learning system ﬁrst learns a meta-model (learner) on domainlevel word context similarity from the ﬁrst m domain corpus.
As more domains arrive, the system accumulates the knowledge of domain corpora.
When a new domain D n +1 comes, the system uses the meta-learner to catch past domain knowledge that is useful and related to the new domain D n + as augmented knowledge.
With the augmented knowledge, the word embedding learning process is performed and the resulting embeddings are used for further down-stream learning tasks.
The metalearner here plays a central role in automatically identifying useful knowledge from past domains to help the new domain.
M ETA -L EARNER
In this subsection, we describe how a meta-learner can help to identify similar words from many past domain corpora.
When it comes to borrowing knowledge from past domains, the ﬁrst problem is what to borrow.
Although binary cross-domain embeddings are studied in Bollegala et al.
Yang et al.
, they mostly assume that a relevant domain is already identiﬁed and shared words between two domains have similar meanings.
In reality, given a wide spectrum of domains, borrowing knowledge from a non-relevant domain may not helpful or even harmful to word embeddings (we show this Section 6).
The meaning of one word in one domain may be quite different from the same word in another.
For example, the word “java” in the programming context is different from the restaurant context.
Borrowing the knowledge from a restaurant corpus can be harmful to the representations of “java” in a programming context.
4. T RAINING E XAMPLES
On top of learning embeddings for speciﬁc domains, we build a meta-learner to learn a general word context similarity from the ﬁrst m domains.
We ﬁrst characterize words in a domain corpus as training examples.
Given a speciﬁc word in a domain, we choose its co-occurrence with words in the vocabulary as features.
We choose co-occurrence because it is close to the training examples of the skip-gram model we use later.
Due to the large size of a vocabulary, we only select the top f frequent words throughout all m domains as word features V wf .
Then given a domain corpus D i , we sample l sub-corpora D i,j ∼ P ( D i ) by selecting a ﬁxed amount of chunks in D i .
A chunk can be a sentence or a document in D i .
We randomly select a ﬁxed amount of chunks because the word features built from the sub-corpus are expected to be on the same scale.
Then we randomly select a subset of words from top f words as training example words V meta .
These training example words are the same in all domains D 1: m .
We use these words in V meta as co-occurrence features and build features u w i,j,k for the word w k ∈ V meta on the j -th sub-corpus of the i -th domain.
We build word features for all m domain sub-corpora D 1: m, 1: l .
Finally, a pairwise meta-learner is trained on pairs of word features drawn from different domain sub-corpora for the same word.
Given a word w k ∈ V meta , a pair of word features ( u w i,j,k , u w i,j (cid:48) ,k ) , where j (cid:54) =
j (cid:48) , forms a postive example; whereas ( u w i,j,k , u w i (cid:48) ,j (cid:48) ,k ) with i (cid:54)
=
i (cid:48)
( j and j (cid:48) can be equal or not) forms a negative example.
The m domains are split into disjoint m t training domains, validation domains, and testing domains.
So both the validation and testing examples are unseen examples during training.
We enforce such isolation and wish the trained meta-learner can be more generally applied to the rest n − m domains.
4. P AIRWISE N ETWORK
We train a simple but efﬁcient neural network to learn pairwise domain-level word context similarity.
The idea of making such a network small but high-throughput is crucial in lifelong settings.
This is because the meta-learner is heavily used in the later lifelong learning process.
Given so many domains with so many words asking for detecting similarity, a small pairwise network with fewer parameters is desirable to save more memory being used for high-throughput inference.
The proposed pairwise network contains only one shared fully-connected layer (normalized by the co-occurrence feature) to learn continuous features from co-occurrence (discrete) features, a cosine function to learn similarity and a sigmoid layer to generate predictions like linear regression.
The network is parameterized as follows:
[Equation]
where | · | 1 is the l 1 -norm, W s and b are weights and σ ( · ) is the sigmoid function.
Cosine similarity is deﬁned as Cosine ( x , y ) =
x · y || x || 2 ·|| y
|| 2 .
Most trainable weights of this simple network reside in
W 1 , which learn continuous features on the f words.
These weights can also be interpreted as an embedding matrix for the f words.
These f word embeddings serve as general word embeddings to explain domain-speciﬁc words. We train the meta-learner over a hold-out domain set as the base meta-learner M . Then we ﬁne-tune the meta-learner based on new domain corpus, as described in the next section.
L IFELONG L EARNING P ROCESS
The previous section ends up with a base meta-learner M .
In this section, we describe how the lifelong learning system works based on M , n − m domains, and the new domain corpus D n +1 .
5. W ORD C ONTEXT R ETRIEVAL
Assume the lifelong learning system has seen n domain corpora.
The system stores knowledge into a knowledge base K .
The knowledge base K contains a base meta-learner M trained over the ﬁrst m domains, ﬁne-tuned meta-learners M m +1: n and knowledge over past n − m domains.
The knowledge includes the vocabulary of word features V wf , n − m domain corpora D m : n , vocabularies on n − m domains V m : n , and word features on those vocabularies E m : n .
The word features E m : n are computed from one sample from each domain corpus.
Given a new domain corpus D n +1 , the lifelong learning system ﬁrst ﬁne-tunes the base meta-learner M . This ends with a ﬁne-tuned meta-learner M n +1 for this new domain. The tuning process makes the meta-learner more suitable for the new domain to retrieve past knowledge.
Tuning examples are sampled similarly as the training examples of base meta-learner, except that negative examples are sampled between D n +1 and D m +1: n .
Then the lifelong learning system retrieves similar in-domain word context information as augmented knowledge, which is used in embedding training in the next subsection.
The retrieval process is described in Algorithm 1.
Firstly, line 1-2 build word features for the new domain corpus.
These two operations are already done when preparing ﬁne-tuning data for the metalearner. Here we just mention them for storing knowledge purpose in line 21. Line 3 retrieves past domain knowledge, which is the reversed process similar to line 21 for the new domain.
Line 5retrieves relevant words from past domains. More importantly, the ﬁne-tuned meta-learner at lineﬁnds similar words from past domains. Then we only keep similar words with a probability higher than a threshold delta at line 11. This threshold controls the quality of the accumulated words in line 16-20.
Finally, line 16-20 retrieves co-occurrence information as augmented knowledge.
Such augmented knowledge is more close to the co-occurrence information in the new domain D n +1 .
Line 21 simply stores the knowledge of the new domain for further use.
Input : a knowledge base K containing knowledge over past ( n − m ) domains, a new domain corpus D n +1 , and a ﬁne-tuned meta-learner M n +1 .
Output : a word co-occurrence set A , where each element is a 2-tuple ( w t , w c ) , representing useful knowledge from past domains.
1 V n +1 ← BuildVocab
( D n +1 ) 2 E n +1 ← BuildWordFeature ( D n +1 , K .V wf ) {+
+} ( D m : n , V m : n , E m : n , M m : n ) ← K m : n 4 W ← {∅} 5 for ( V i , E i ) ∈
V m : n , E m : n do
O ← V i ∩ V n +
F i ← RetrieveWordFeature ( E i , O )
F n +1 ← RetrieveWordFeature ( E n +1 , O )
S ← M n +1 . inference ( F i , F n
1 ) for do then
W
W end end 15 end 16 W 17 A ←[-InvertbyDomainIndex ( W ) 
-]
18 for D O i ∈
D m : n , W m : n do
A ← A ∪ ScanCo-Occurrence D , O 20 end 21 K n +1 ← ( D n
V n
1 , E n
1 , M
) 22 return A
5. L IFELONG W ORD E MBEDDING
In this subsection, we ﬁrst describe the skip-gram model introduced by Mikolov et al.
(2013b) in the context of a new domain in the lifelong setting.
Given a new domain corpus D n +1 with a vocabulary V n +1 , the goal of the skip-gram model is to learn a vector representation for each word w ∈ V n +1 in that domain.
Assume the domain corpus is represented as a sequence of words
D n +1 = ( w 1 , . . . , w T ) , the objective of the skip-gram model is to maximize the following loglikelihood:
[Equation]
where C t is the set of indices of words surrounding word w t in a ﬁxed context window; N t is a set of indices of words (negative samples) drawn from the vocabulary V n +1 for the t -th word; u and v represent word vectors (or embeddings) we are trying to learn.
The goal of skip-gram is to independently predict the presence (or absence) of context words w c given the word w t .
When the size T of the corpus is extremely large, the skip-gram model, in fact, can be fed with billions of training examples.
So the vector of a word can be trained to have a good representation of the similarity with the word’s context words.
However, depending on the speciﬁc down-stream tasks, many domain corpora may not have a large scale corpus.
And a random sequence of words drawn from other domains may not truly reﬂect the distribution P ( w c | w t ) in domain D n +1 .
This is where the previously computed augmented word co-occurrence A come to rescue.
Assume our lifelong learning system has seen m domains to build the meta-learner M and n − m domains to build the knowledge into K .
Given a new domain corpus
D n +1 , we ﬁrst perform Algorithm 1 to obtain the augmented word co-occurrence from past domains
A .
Then this co-occurence information A is integrated into the objective function of skip-gram as following:
[Equation]
where w c (cid:48) is a random word drawn from the vocabulary.
We use the default hyperparameters of skip-gram model (Mikolov et al.
(2013b)).
Note that in the skip-gram model as we scan through the corpus w t can also be w c ’s context word.
But in the augmented information here, we do not allow such bi-directional co-occurrence happen since w t may not be a useful context word for the word w c in the ( n + 1) -th domain.
E XPERIMENTAL R ESULTS
We present extensive evaluations to assess the effectiveness of our approach.
Following the suggestions of Nayak et al.
; Faruqui et al.
(2016), we leverage the learned word embeddings as continuous features in several domain-speciﬁc down-stream tasks, including document classiﬁcation, aspect extraction, and sentiment classiﬁcation.
We do not evaluate the learned embeddings directly as in traditional word embedding papers (Mikolov et al.
(2013b); Pennington et al.
because domain-speciﬁc dictionaries of similar / non-similar words are in general not available.
6. D ATASETS
We use the Amazon Review datasets He & McAuley (2016) as a huge collection of multiple-domain corpus.
We consider each second-level category (the ﬁrst level is department) as a domain and aggregate all reviews under each category as one domain corpus.
This ends up with a rather diverse domain collection. Due to limited computing resources, we limit each domain corpus up to 60 MB. We randomly select 3 domains (“Computer Components”, “Cats Supply” and “Kitchen Storage and Organization”) as new domains for down-stream tasks on product type classiﬁcation and sentiment classiﬁcation.
Then we deliberately pick the “Laptops” domain as the new domain for aspect extraction task since the annotation is on Laptop reviews.
Each new domain corpus is cut to 10 MB and 30 MB in order to test the practical performance of a small new domain.
We randomly select( m ) domains to train and evaluate the meta-learner.
Lastly, three random collections of 50, 100 and 200 ( n − m ) domains corpora are used as past domains.
6. E VALUATION OF M ETA -L EARNER
We split the 56 domains as 39 ( m t ) domains for training, 5 domains for validation and 12 domains for testing.
So the validation and testing domain corpora have no overlapping with the training domain corpora.
This leads to a more general base meta-learner for many unseen new domains.
We sample 2 ( l ) sub-corpora over the set of reviews from each domain and limit the size of the sub-corpora to 10 MB.
We select top 5000 words as word features ( f ).
We randomly selectwords ( | V meta | = 500 ) from each domain and ignore words with zero counts on co-occurrence to obtain pairwise examples.
This ends up with 80484 training examples, 6234 validation examples, and 20740 testing examples.
The f1-score of meta-learner is 81% . [Table] We further ﬁne-tune the meta-learner for each new domain. We sample 3000 words from each new domain, which ends with slightly fewer than 6000 samples after ignoring zero co-occurrences.
We select 3500 examples for training, 500 examples for validation and 2000 examples for testing.
The testing f1-score is shown in Table 1. Finally, we empirically set delta = 0 . 7 as the threshold.
6. D OWN - STREAM T ASKS
We use 3 down-stream tasks to evaluate the effectiveness of our approach.
For each task, we leverage an embedding layer to store the pre-trained embeddings.
We choose our embedding dimensions as 300, which is the same size as many pre-trained embeddings (GloVec.800B (Pennington et al.
or fastText Wiki English (Bojanowski et al.
We freeze the embedding layers during training, so the result is less affected by the rest of the model and the training data.
To make the performance of all tasks relatively consistent, we leverage the same Bi-LSTM model (Hochreiter & Schmidhuber (1997)) on top of the embedding layer to learn task-speciﬁc features from different embeddings.
The input size of Bi-LSTM is the same as the embedding layer and the output size is 128.
All tasks leverage many-to-one Bi-LSTMs for classiﬁcation purpose except aspect extraction, which uses many-to-many Bi-LSTM for sequence labeling.
In the end, a fully-connected layer and a softmax activation are applied after Bi-LSTM, with the output size speciﬁc to each task.
6. B ASELINES
No Embedding (NE) : We randomly initialize the word vectors and train the word embedding layer during the training process of each down-stream task.
Note that only in this baseline do we allow embeddings trainable.
fastText : This is the lower-cased embeddings pre-trained from English Wikipedia using fastText (Bojanowski et al.
We lower the cases of all corpora of down-stream tasks to match the words in this embedding.
Note that although the corpus of Wikipedia contains a wide spectrum of domains covering almost everything of human knowledge, the amount of corpus for a speciﬁc domain (e.g, a product) may not be large enough.
The total amount of Wikipedia is just several billions of tokens, which is on the same scale as Amazon Review datasets (8 billion tokens). GoogleNews :
This is the pre-trained embeddings using word2vec 2 based on part of the Google
News datasets, which contains 100 billion words.
GloVe.
Twitter.27B : This embedding is pre-trained using GloVe (Pennington et al. based on Tweets, which have 27 billion words.
Note this embedding is lower-cased and has 200 dimensions.
GloVe.6B : This is the lower-cased embeddings pre-trained from Wikipedia and Gigaword 5, which has 6 billions of tokens.
GloVe.840B : This is the cased embeddings pre-trained from Common Crawl, which has 840 billions of tokens.
This embedding corpus is the largest one among all embeddings.
It contains almost all web pages available before 2015.
We show that although GloVe.840B is general enough on almost any task, its performance is sub-optimal on many domain tasks. New Domain 10M (ND 10M) :
This is a baseline embedding pre-trained only from the new domain 10 MB corpus.
We show that the embeddings trained from a small corpus alone are not good enough.
New Domain 30M (ND 30M) :
Then we increase the size of the new domain corpus to 30 MB to see the difference affected by the corpus size.
[Table]
200D
[Table]
ND 10M
[Table] 50D [Table]
200 Domains + New Domain 30M (200D + ND 30M) : Another straightforward embedding is to concatenate the corpora from all past domains and the new domain together to train a mixed embedding.
We use this baseline to show that unselected past domain corpora may reduce the performance of down-stream tasks.
How to smartly adapt the embeddings into a closer context is crucial. Lifelong Past Domains + New Domain (LL [P]D + ND
X]M) : This is different versions of our proposed method.
For example, we use LL 200D + ND 30M to donate embeddings trained from a 30MB new domain corpus and 200 past domains.
6. P RODUCT T YPE C LASSIFICATION
This task is to classify a review into a product type (leaf-level category in Amazon product category system).
There are many product types under each domain (2nd-level category).
We use the randomly selected 3 domains as the new domains to form 3 multi-class classiﬁcation sub-tasks.
These domains are: Computer Components , Kitchen Storage and Organization and Cats Supplies .
For each sub-task, we randomly draw 1200 reviews for each product type.
We drop classes with less than 1200 reviews.
This ends up with 13, 17 and 11 classes for Computer Components , Kitchen
Storage and Organization and Cats Supplies , respectively.
For each sub-task, we keep 10000 reviews as the testing data (to make the result more accurate) and split the rest as 7:1 for training and validation data, respectively.
All sub-tasks are evaluated on accuracy.
We train and evaluate each sub-task on each baseline 10 times (with different initialization) and average the results.
From Table 2, we can see that the performance of different classiﬁcation tasks varies a lot. This is mostly caused by the number of classes in each sub-task.
The lifelong embeddings LL 200D + ND 30M performs best.
The maximum train corpus of this method is in total just 80 MB (
MB new domain + 50 MB from 200 past domains).
The difference in the numbers of LL past domains indicates more past domains have better results.
Surprisingly, the GloVe.840B trained on 840 billions of tokens does not perform well enough compared to the limited new domain corpora.
But it performs very well on Computer Components, which means this domain is relatively general.
Putting all past domain corpora together with the new domain corpus (200D + ND 30M) makes the result worse than to not use the past domains at all (ND 30M).
This is because those 200 domains may not be close to the new domains. 6. A SPECT E XTRACTION
Aspect extraction is an important task in sentiment analysis (Liu (2012; 2015)). We use the dataset from SemEval-2014 Task 4: Aspect-based sentiment analysis (Pontiki et al. as a downstream new domain task. This dataset contains human annotated Laptop aspects and their polarities.
It has 3045 training examples and 800 testing examples.
We use the Laptop domain corpus from the [Table] [Table] Amazon Review Dataset as the new domain corpus to train the lifelong embedding.
We leverage the original evaluation script to report precision, recall, and F1-score.
Again, we average 10 runs of the results. From Table 3, we can see that aspect extraction is quite different from product type classiﬁcation. Again, our lifelong embedding performs best. Surprisingly, the performance of 200D + ND 30M is very good.
This indicates aspect extraction requires both good general embedding and domainspeciﬁc embeddings.
For example, good representations of general words can help to identify nearby aspects and good aspects words can also help.
6. S ENTIMENT C LASSIFICATION
We select 6000 4-rating reviews as positive reviews and 6000 2-rating reviews as negative reviews from 3 domains used in product type classiﬁcation to form 3 sentiment classiﬁcation sub-tasks.
Again, to ensure enough number of valid digits in the results, we use 10000 out of 12000 reviews for testing.
The results are averaged over 10 runs.
From Table 4, we can see the performance of most domain-speciﬁc baselines is very close (We omit the minor differences between different sizes of past domains).
Sentiment classiﬁcation, in general, requires polarity words to determine the sentiment of a document.
Pre-trained general embeddings may introduce non-polarity information into the embeddings.
When domain corpus is leveraged, the difference is small.
This is close to our previous experience.
A possible explanation is that sentiment classiﬁcation relies on sentiment words like “good” or “bad”.
However, those words have similar context words, e.g., “This phone is good.”
and “This phone is bad.”.
So the co-occurrence-based training corpus of embedding is not good for learning the embeddings of sentiment words.
[Table] 6. F USING WITH THE O UT - OF -D OMAIN W ORLD Although most cross-domain embedding papers focus on leveraging different existing pre-trained embeddings, we focus on expanding the domain-speciﬁc corpus. We believe if we can expand the domain-speciﬁc training corpus on a much larger scale (like GloVe.860B), the performance of the proposed method is much better.
However, our focus does not forbid our method from leveraging existing cross-domain transfer learning method ( ? Bollegala et al. ; Yang et al. A simple way of leveraging existing embeddings in these papers is to concatenate existing pre-trained embeddings with domain-speciﬁc embeddings.
To demonstrate our method further improves the domainspeciﬁc parts of the down-stream tasks, we further evaluate two methods: (1) GloVe.840B&ND 30M, which concatenates new domain alone embeddings with GloVe.860B; (2) GloVe.840B&LL 200D + ND 30M, which concatenates our lifelong embeddings with GloVe.860B. As shown in Table 5, concatenating embeddings improve the performance a lot. Our method further improves the domain-speciﬁc parts of the embeddings. While existing cross-domain embedding methods can only use the 30 MB corpus of the new domain, our method allows those methods to further leverage the expanded corpus.
C ONCLUSIONS
In this paper, we formulate a lifelong word embedding learning process.
Given many previous domains and a small new domain corpus, the proposed method can effectively generate new domain embeddings by leveraging a simple but effective algorithm and a meta-learner.
The meta-learner is able to provide word context similarity information on domain-level.
Such information can help to accumulate new domain-speciﬁc training corpus in order to get better embedding. Experimental results show that the proposed method is effective in learning new domain embeddings from a small corpus and past domain knowledge.