W ORD
EMBEDDING RE - EXAMINED : THEORETICAL ANALYSIS ABOUT WORD SIMILARITY AND ANALOGY STRUCTURE
Anonymous authors
Paper under double-blind review
A BSTRACT
As observed in previous works, many word embedding methods exhibit two interesting properties: (1) words having similar semantic meanings are embedded closely; (2) analogy structure exists in the embedding space, such that “ Paris is to France as Berlin is to Germany ”.
We theoretically analyze the inner mechanism leading to these nice properties.
Speciﬁcally, the embedding can be viewed as a low rank transformation from the word-context co-occurrence space to the embedding space.
Such embedding transformation will preserve the relative distances among words.
Furthermore, previous studies empirically observed that the parameter α has a strong inﬂuence on the performance of word embedding but did not provide theoretical explanation.
We provide a theoretical explanation for this behavior, and derive a method to automatically ﬁnd its optimal value.
The experiments on real datasets verify our analysis.
I NTRODUCTION
Word embedding is a fundamental task in natural language processing.
Many different approaches have been proposed, including LSA (Deerwester et al., 1990), SGNS (Mikolov et al., 2013a;b),
GloVe (Pennington et al., 2014) and others.
These methods have achieved huge success in information retrieval (Salton & Buckley, 1988), entity recognition (Lample et al., 2016), sentiment analysis (Socher et al., 2013), machine translation (Sutskever et al., 2014) and so on.
Previous studies have demonstrated three well-known facts about the word embedding: (1) Levy &
Goldberg (2014b) showes many different kinds of word embeddings (e.g. SGNS ) can be understood as the matrix factorization framework.
For example, Levy & Goldberg (2014b) proves that SGNS is implicitly factorizing the shifted pointwise mutual information (PMI) matrix.
(2) Mikolov et al.
(2013b); Pennington et al.
(2014); Levy et al.
; Tian et al.
and other works observe that word embedding exhibits two nice properties: the word similarity such that words with similar semantic meanings are embedded closely in the embedding space, and the analogy structure such that “woman is to queen as man is to king”.
(3) Previous studies, including (Caron, 2001; Turney, 2012; Bullinaria & Levy, 2012; Levy et al., 2015; Artetxe et al., 2018), empirically observe that α in word embedding E
=
U d · Σ αd 1 has an important inﬂuence on the quality of word embedding.
But these works did not provide theoretical explanation for this behavior of α or practical guide to set this parameter.
In this paper, we show the word embedding can be understood as a low rank transformation process.
Furthermore, we investigate the change of the relative distances between words which is the inner mechanism that leads to these two nice properties in the word embedding.
Particularly, we ﬁnd the parameter α has inﬂuence on the change of relative distance between words.
Based on such analysis, we provide theoretical explanation the behavior of α and derive a method to automatically ﬁnd its optimal value.
To summarize, this research is motivated by two questions:
What is the inner mechanism resulting in these two good properties of word embedding? 2.
How does the parameter α inﬂuence the word embedding?
Can we ﬁnd a method to determine its optimal value?
We make the following contributions in this paper:
• We reveal how the relative distances between words change during the embedding process which can be seen as a low rank transformation process.
The nice properties of word embeddings are inherited from the original word-context co-occurrence matrix as the result of such low rank transformation.
• We theoretically explain the inﬂuence of parameter α on the word embedding by revealing it controls the change of relative distances between words.
Besides, we derive a method to automatically ﬁnd its optimal value.
We conduct experiments on real datasets to verify our ideas.
The following sections are organized as follows: section 2 introduces the background knowledge and necessary preliminaries; section 3 shows how the relative distances between words change during this linear transformation and it is the inner reason incurring these nice properties of word embedding.
Based on such analysis, section 4 shows that symmetric factorization can be suboptimal and how to improve the word embedding model with asymmetric factorization.
Section 5 contains the experiments on real datasets and section 6 discusses the related work.
Finally, section ?? is the conclusion and describes a potential work.
B ACKGROUND AND P RELIMINARIES
The goal of word embedding is to ﬁnd a good vector representation for every word in a corpus.
As summarized in Yin & Shen (2018), most existing word embedding methods can be formulated as low rank matrix approximations.
So we can analyze the word embedding algorithms (e.g., SGNS ) within the matrix factorization framework.
Suppose M ∈ R n × n is the word-context matrix where
M i,j represents some statistics between word w i and its context word w j .
These word embedding algorithms are seeking a d -dimension word embedding matrix E ∈ R n × d accompanied with a context embedding matrix C ∈ R n × d to approximate the word-context matrix M , such that
M ≈ EC T .
The matrix M can have different forms due to the design of objective function in every embedding method.
For example, M is the shifted pointwise mutual information matrix in the SGNS .
Explicitly performing singular value decomposition.
This kind of methods can be summarized as follows.
Firstly, a word-context matrix M is constructed to capture some co-occurrence statistics of every word and its context.
For example, M can be the pointwise mutual information (PMI) matrix (Church & Hanks, 1990), positive PMI (PPMI) matrix (Bullinaria & Levy, 2012) and Shifted PPMI (SPPMI) matrix (Levy & Goldberg, 2014b).
Then, SVD is used to factorize the word-context matrix
M as M = U · Σ · V T .
Finally, only the top d singular values in Σ and the corresponding columns in U and V are kept, producing Σ d , U d and V d respectively.
The word embedding E ∈ R n × d is obtained by E
= U d or E =
U d · Σ d .
Caron (2001), Turney (2012), Bullinaria & Levy (2012), Levy et al. (2015) and Artetxe et al.
(2018) discussed a more general approach which adds a parameter α to the truncated diagonal matrix Σ d and obtain the word embedding E
=
U d · Σ αd .
They observe α has an important inﬂuence on the embedding and suggest this parameter should be tuned.
However, the reason is not theoretically clear and no clear method has been provided to tune α yet.
Implicitly performing matrix factorization.
Recently, word embedding methods based on neural networks have been proposed (e.g., Bengio et al.
Particularly, Mikolov et al.
(2013a;b) proposed a popular word embedding approach SGNS based on the skip-gram with negative-sampling and it achieves the state-of-the-art results in different tasks.
Besides, GloVe is another widely used word embedding method (Pennington et al., 2014).
While SGNS and GloVe learn word embeddings by optimizing some objective functions using stochastic gradient methods, it has been shown that these two methods are implicitly performing matrix factorizations.
Speciﬁcally, Levy &
Goldberg (2014b) showed SGNS is implicitly factorizing the shifted Pointwise Mutual Information matrix.
Levy et al.
(2015) showed the GloVe is implicitly factorizing the log-count word-context co-occurrence matrix.
So these two neural network based embedding methods can also be formulated within the matrix factorization framework, and the parameter α equals to 0 .
5 as they are doing symmetric factorization of some word-context co-occurrence matrix M .
To summarize, same as the ﬁrst assumption in Yin & Shen (2018), our analysis assumes the word embeddings can be formulated as low rank matrix approximations, either explicitly or implicitly.
Speciﬁcally, the embedding matrix E equals to U d · Σ αd where U d , Σ d and α are as deﬁned before.
A NALYSIS : THE REASON WHY THE EMBEDDING EXHIBITS NICE PROPERTIES
Our start point is the relation between word embedding and matrix factorization.
As mentioned in section 2, almost all word embedding methods can be formulated as explicit or implicit matrix factorization (e.g., SGNS is implicitly doing a symmetric factorization on the shifted pointwise mutual information matrix).
We adopt the same notation in previous papers (Levy et al., 2015; Yin & Shen, 2018) to write M = U Σ V T as the SVD of some word-context co-occurrence matrix M .
For example, M can be the pointwise mutual information matrix.
Besides, Σ d is the truncated diagonal matrix containing top d singular values.
U d and V d are the corresponding truncated U and truncated
V respectively.
The d -dimension word embedding matrix E is obtained by multiplying U d with a power of Σ d ,
[Equation]
3. W ORD EMBEDDING AS LOW RANK TRANSFORMATION
The embedding E in equation 1 is indeed a low rank transformation from the original word-context co-occurrence space M .
Let us deﬁne another diagonal matrix Σ pse = diag ( σ 1 , . . .
, σ d , 0 , . .
. , 0) whose ﬁrst d elements are the singular values in Σ and the remaining diagonal elements are zeros.
We have the following observation.
Observation 1.
E is the ﬁrst d columns of E pse
[Equation]
The proof is in Appendix A.1.
Observation 1 reveals the process to compute E :
(1) ﬁrst multiplying the matrix V to the right side of M ; (2) then multiplying the diagonal matrix Σ (1 − α ) pse on the intermediate result MV ; (3) ﬁnally removing the last n − d columns of M · V · Σ ( α − 1) pse .
The ﬁrst step in this transformation process is a rotation because the matrix V is a unitary matrix.
So the relative distances between words are not changed during the ﬁrst transformation.
In the second step, the matrix Σ pse is a diagonal matrix and it is multiplied on the right side of MV .
So the second step scales every dimension of the rotated matrix.
Because we keep the ﬁrst d dimensions in the ﬁnal step, we only care about the scaling in the ﬁrst d dimensions which are determined by the corresponding d singular values with a power α .
By explictly expressing the process which transforms the word-context co-occurrence matrix M to the embedding matrix E , we can ﬁnd a fact that the relative distances between words only changes within the scaling step.
Considering vectors of all words are scaled the same at every dimension, it seems that the relative distances among words should be preserved to some extent.
In fact, we have the following two theorems:
Theorem 1.
For any two words w i and w j , if (cid:107) M i, : − M j, : (cid:107) 2 ≤ δ ( δ ≥ 0 ), then (cid:107) E i, : − E j, : (cid:107) 2 ≤ σ ( α − 1)
d δ .
Theorem 2.
For any two words w i and w j , if (cid:107) E i, : − E j, : (cid:107) 2 ≤ δ ( δ ≥ 0 ), then (cid:107) M i, : − M j, : (cid:107) 2 ≤ σ (1 − α )δ + 2( (cid:80) nk =
d +1 σ 2 k ) 1 / 2 .
The formal proof of theorem 1 and theorem 2 is in appendix A.2 and A.3.
These two theorems describe how the relative distances (measured in Euclidean distance) among words change during the embedding transformation.
Particularly, theorem 1 shows that if the distance between two words w i and w j in M space is no more than δ , their distance in the embedding space is also within a constraint.
This bound is related with the singular value σ d and the parameter α .
So theorem 1 implies that the relative distances between words will not change dramatically during this transformation from M to E with suitable α .
Theorem 2 demonstrates that if we observe two words are embedded closely in E ( (cid:107) E i, : − E j, : (cid:107) 2 is small), we can infer these two words are also close in the original M space.
To sum up, the above two theorems indicate the neighborhood structure (relative distances to other words) for every word will be preserve to some extent during the transformation from M to E .
In other words, matrix E inherits the neighborhood structure that exists in M .
As the result, E will inherit the properties in M that are related with the relative distances among words.
Now we can answer the question raised in the introduction: What is the inner mechanism resulting in the good properties of word embedding?
In fact, we will show that these properties are the result of inheritance of neighborhood structure.
3. T HE INNER MECHANISM RESULTING IN THE NICE PROPERTIES IN THE EMBEDDING
Let us focus on two important properties in E : (1) words with similar semantic meanings are likely to be embedded closely in the embedding space; (2) the word embedding tends to exhibit the analogy structure, e.g., “woman is to queen as man is to king”.
Note that these two properties are related with the relative distances among words.
So, considering the previous analysis which shows matrix E inherits the neighborhood structure existing in M , these two properties in E also inherits from M as the result.
Words with similar semantic meanings are embedded closely.
Harris (1954) indicated that if two words have almost identical environments we say that they are synonyms. .
In other words, two words will have similar neighbors if they have similar semantic meanings.
It means the corresponding rows in the word-context co-occurrence matrix M should be close for these words.
So the words with similar semantic meanings are also close in the M space which has been empirically veriﬁed in Levy et al.
Remember that theorem 1 shows if the distance between two words in M is close (within a small range δ ), these two words in E will also be close due to this low rank transformation process from M to E .
So the low rank transformation in fact makes the embedding E inherit this property existing in the original word-context matrix M .
The analogy structure in the embedding.
Different from word similarity which is about the relative distance between two words (e.g., king and queen ), the analogy structure is related with the relative distance between two pairs (e.g., (king, queen) and (man, woman) ).
So let us ﬁrst examine how the relative distance between two pairs change during the embedding process.
Theorem 3.
Let M γ 1 , : , M γ 2 , : , M β 1 , : and M β 2 , : be four rows in M matrix.
Denote M γ =
( M γ 1 , : − M γ 2 , : ) and M β =
( M β 1 , : − M β 2 , : ) .
If (cid:107) M γ, : − M β, : (cid:107) 2 ≤ δ ( δ ≥ 0 ), then
[Equation]
, : −
[Equation]
β, : (cid:107)
[Equation]
α
[Equation] [Equation]
δ .
Theorem 4.
Let M γ 1 , : , M γ 2 , : , M β 1 , : and M β 2 , : be four rows in M matrix.
Denote M γ =
( M γ 1 , : − M γ 2 , : ) and M β =
( M β 1 , : − M β 2 , : ) .
If (cid:107) E γ, : − E β, : (cid:107) 2 ≤ δ ( δ ≥ 0 ), then (cid:107) M γ, : − M β, : (cid:107) 2 ≤ σ (1 − α )
δ + 4( (cid:80) nk =
d +1 σ 2 k ) 1 / 2 .
The proof of theorem 3 and theorem 4 is in appendix A.4 and A.5, which is quite similar to the proof of theorem 1 and theorem 2.
Theorems 3 and 4 show that the relative distance between two pairs will not change dramatically during the embedding process either.
In other words, the relative distance between pairs of words will be preserved during the embedding transformation from M to E .
Furthermore, Levy & Goldberg (2014a) revealed a fact that the original word-context co-occurrence M (e.g., M can be the PPMI matrix) 2 also exhibits the analogy structure, such that M king − M queen ≈ M man − M woman .
So theorems 3 and 4 together with this fact indicate the analogy structure in E also inherits from M as the result of distance preserving in such low rank transformation process.
summer winter autumn rainy spring summers winters monsoonnights
warmest coldest snowfall precipitation humidity temperatures seasons climateshumid year windy freezing (a) α = 0 .summer
winter autumn rainy spring summers nights winters monsoon snowfall warmest coldest seasons precipitation year days temperatures weekend windy sunny humidity (b) α
=
.summer winterautumnrainy spring summers nights winters monsoon snowfall warmest coldest seasons precipitationyear days temperatures weekend windy sunny humidity (c) α = 1 .
To summarize, we prove the bound about the change of relative distances of words during the embedding process.
With appropriate α , the neighborhood structure (relative distance) of every word will not change much during the embedding process.
So the embedding E inherites the word similarity and the analogy structure from M .
We also theoretically explain how the parameter α inﬂuences the word embedding by controling the change of relative distances among words (the bounds in theorem 1-4).
So a consideration is that we can optimize these bounds by choosing suitable α .
We show the derivation of ﬁnding the suitable α with respect to the bounds in appendix A.7.
It shows the suitable α depends on σ 1 and σ d , together with how to weight the two bounds in theorem 1 (3) and theorem 2 (4).
However, the optimization over the bound only gives a rough estimation of the possible range of optimal α .
In the next section, we optimize α by directly minimizing the change of neighborhood of words instead of optimizing over the bound.
I MPROVEMENT : W ORD EMBEDDING PRESERVING THE DISTANCE STRUCTURE BETTER
In previous section, we reveal the inner mechanism why the embedding exhibits such nice properties.
Namely, the embedding is in fact a low rank transformation from M to E and the relative distances between words are preserved during such process.
As a result, the embedding matrix E inherits these nice properties existing in the original word-context co-occurrence matrix M (e.g., the PMI matrix).
Particularly, theorem 1, 2, 3 and 4 show that the parameter α has an important inﬂuence on how the relative distances between words change.
For example, we arbitrarily select a word summer in the corpus 3 and plot its nearest 20 neighbors with α equal to 0 . 0 , 0 .
5 and 1 .respectively.
Figure 1 shows that the neighborhood of summer changes with different α .
This observation gives us an inspiration that we can learn the embedding E =
U d · Σ αd by selecting a good α , thus preserving the relative distances among words during the embedding transformation process.
Preserving neighborhood structure better by choosing optimal α for embedding E =
U d · Σ αd .
Let g X ( w ) represent the neighbors of word w in X space, e.g., X can be the word-context cooccurrence matrix M or the embedding matrix E .
Let l ( · , · ) be a correlation function measuring the similarity of two different neighborhoods.
This measurement captures how much the neighborhood structure changes incurred by the embedding process.
For example, if we consider the top knn nearest neighbors as g X ( · ) and ignore the order of these nearest neighbors, g X ( w ) can be viewed as a set and l can be a similarity measurement deﬁned on sets (e.g., Jaccard Similarity).
Otherwise, if we consider the order of neighbors, g X ( w ) can be a ranking list such that the neighbors are ranked by their relative distance to w , and l now can be a measurement about the correlation between two rankings.
We use L to denote the aggregated similarity score over all words
[Equation]
where W is the word vocabulary to consider and its size is n .
Based on previous analysis, we argue we can select an optimal α for embedding E
=
U d · Σ αd to preserve the relative distances among words during the embedding transformation, such that
[Equation]
Leveraging the effect of α into the original SGNS network structure.
The above analysis is based on the view that we compute the embedding matrix E by SVD such that E
= U d · Σ αd .
We can also think about to incorporate the effect of α into the original SGNS neural network structure.
The original SGNS implicitly performing a symmetric factorization, thus implying the α equal to 0 .
Here, we consider to leverage the inﬂuence of α directly into the SGNS structure.
Consider the embedding
E =
U d · Σ αd and the context C = Σ (1 − α )
d · V Td learnt by SVD, we have
[Equation]
where e i and c i are the word vector w i and the context vector of w i respectively.
We put it as a regularization term in the original SGNS objective function to add the inﬂuence of different α to the embedding learnt by SGNS .
The detailed derivation is in appendix A.6.
To summarize, the α parameter in the embedding matrix E =
U d · Σ αd inﬂuences how the relative distances between words change during the embedding process, thus inﬂuencing the quality of the learnt embedding matrix.
We propose a method to choose the optimal α by preserving the relative distance among words during the embedding transformation.
Besides, we come up with an idea to incorporate the effect of α into the original SGNS architecture.
In the next section, we will test these ideas on real experiments.
E XPERIMENTS
In this section, we conduct experiments to verify our previous analysis.
The ﬁrst test is to verify that a good embedding does maximize the correlation deﬁned in equation 3 which measures to which extent the neighborhood structure is preserved during the embedding transformation.
The second is to test if the proposed method to incorporate α into original SGNS architecture really has improvements.
5. E XPERIMENT SETTINGS
Corpus We use the Text9 corpus (Mahoney, 2011) which is a standard benchmark used for various natural language tasks.
We follow the same pre-processing steps in Levy et al.
Word similarity and the analogy structure task
The purpose of the experiments is to verify whether the method proposed in section 4 to ﬁnd the optimal α matches the real case.
We adapt the same experiment as in Levy et al. (2015) to test the word embedding on the word similarity and analogy structure.
There are six datasets acting as the ground truth for the word similarity: WordSim353 (Finkelstein et al., 2002) and its two subsets WS Similarity and WS Relatedness (Zesch et al., 2008; Agirre et al., 2009), Bruni MEN (Bruni et al., 2012), Radinsky Mechanical Turk (Radinsky et al., 2011) and Luong Rare Words (Luong et al., 2013).
These datasets 5 contain word pairs together with human-assigned similarity scores.
Same as Levy et al.
(2015), word embedding is evaluated by the correlation between the similarity computed by the embedding vector and the human-assigned similarity score.
The function to compute correlation is the Spearman’s correlation (same as Levy et al.
Note that we discard the pairs in the test ﬁle if it contains a word that does not appear in the text9 corpus.
We use the word similarity task as an example for verifying the proposed method (equation 3 and equation 4) to learn the optimal α for word embedding E =
U d · Σ αd , and the proposed method to improve SGNS .
The codes used for the SVD embedding is adapted from https://bitbucket.org/ omerlevy/hyperwords and the codes used to learn the SGNS embedding is adapted from https://github.com/theeluwin/pytorch-sgns .
We store the codes in a shared dropbox folder 6 for the double-blind review.
[Figure]
[Table]
5. R ESULTS
Firstly, let us verify that if the embedding E
=
U d · Σ αd is good (measured by the performance on the word similarity testsets), the embedding E should preserve the relative distances between words better (measured by equation 3).
We use the “vanilla” setting in Levy et al. (2015) to construct the positive PMI (PPMI) matrix as M : window size is 2 , no dynamic context window and no subsampling, the number of negative samples is 1 and the context distribution smoothing parameter is 1 . 0 .
For equation 3, We only consider the nearest neighbor for g X and use Jaccard Similarity as the l ( · , · ) .
Note that each testset only contains a subset of words in text9 , and we also compute the overall correlation (the L in equation 3) considering the words in the corresponding testset.
The result is in ﬁgure 1.
Figure 1 shows that the performance (measured by these six word similarity groundtruth) and the proposed measurement are highly correlated, such that, for every testset, if α achieves high score word similarity test, the corresponding measurement in equation 3 is also high.
More details can be found in appendix A.8.
Besides we test whether the α (cid:63) in equationreally corresponds to α that achieves the best performance on the word similarity task.
We plot the performance of E = U · Σ with different α in ﬁgure 2.
The black vertical line corresponds to α that achieves best accuracy and the blue vertical line corresponds to the α (cid:63) learnt by equation 4.
We can see that α (cid:63) is very near to the optimal point in almost all datasets.
The only exception is the Luong testset, which contains many rare words.
We infer the reason is that the corpus cannot effectively capture this testset because more than two thirds words in Luong testsets are not in the text9 corpus.
[Table]
Secondly, we test the effect that incorporating α into SGNS .
Table 2 is the performance of the original
SGNS with α equals to 0 .
3 and 0 .
5 .According to the result (the gold line) in ﬁgure 2, if we push the α from 0 .
5 to 0 .
3 , the performance in the Luong testset should become better and the performance in all other testsets should drop.
The results in table 2 is same with this inference.
The only exception is the bruni men testset.
We infer the reason is that the performances with α = 0 .
3 and α = 0 .
5 are very close (see gold line in ﬁgure 2 (a)).
R ELATED W ORK
Explanation about the analogy structure.
Mikolov et al. (2013c) observed the analogy structure existing in the embedding created by a neural network.
Levy & Goldberg (2014a) afterwards found the same property also exists in the original word-context co-occurrence matrix.
Levy &
Goldberg (2014a) infers neural embedding process is not discovering novel patterns, but rather is doing a remarkable job at preserving the patterns inherent in the word-context co-occurrence matrix .
However, it is not clear how the neural embedding process preserves such property.
Later, several explanations were been proposed to explain the reason (Arora et al., 2016; Gittens et al., 2017; Allen & Hospedales, 2019; Ethayarajh et al., 2018).
For example, Gittens et al.
(2017) and
Allen & Hospedales (2019) referred to the paraphrasing to explain the analogy structure.
Different from these previous studies, we analyze word embedding from the low rank transformation perspective and we reveal the inner mechanism how this property is inherited during the embedding transformation process.
Explanation about the α .
The inﬂuence of α on the quality of word embedding has been observed in (Caron, 2001; Turney, 2012; Bullinaria & Levy, 2012; Levy et al., 2015; Artetxe et al., 2018).
These works empirically ﬁnd that α has an important inﬂuence to the embedding and suggest this parameter should be tuned.
However, these works do not provide theoretical explanation and no clear method has been provided to tune the α .
Yin & Shen (2018) is the only work we know which discusses the meaning of α from the pairwise inner product (PIP) loss perspective.
But their ﬁndings have some contradiction to the real experiments in Caron (2001),Turney (2012), Bullinaria &
Levy (2012), Levy et al. (2015) and Artetxe et al.
Speciﬁcally, the analysis in Yin & Shen (2018) suggests small α is easier to result in over-ﬁtting and lead to the performace drop, which implies that larger α is better.
However, the real experiments in other papers show the performance also drops with very large α .
To summarize, we explain the meaning of α in the word embedding transformation which once was not theoretically clear.
Futhermore, we propose a method to ﬁnd the best α for word embedding.
A A PPENDIX A. P ROOF OF OBSERVATION
Proof.
Firstly, we have
E =
M
[Equation]
[Equation] [Equation]
, U :
[Equation]
d ] ·   σ... σ
[Equation]


[Equation] [Equation]
σ
[Equation]
(cid:105)
Besides,
E pse =
M · V · Σ ( α − 1) pse =
U · Σ · V T · V · Σ ( α − 1) pse =
U · Σ · I · Σ ( α − 1) pse ( V is unitary matrix, so V T · V = I ) =
U · Σ · Σ ( α − 1) pse =[
U : , 1 , U : , 2 , . . .
, U : ,n ] ·   σ. . .
σ d σ d + . . .
σ n 
 ·   σ ( α − 1). . .
σ ( α − 1) d . . .

 =[ U : , 1 , U : , 2 , . . .
, U : ,n ] ·   σ ( α ). . .
σ ( α ) d . . .

 =[ σ ( α ) 1 U : , 1 , σ ( α ) 2 U : , 2 , . . .
, σ ( α ) d U : ,d , 0 , . . .
So E corresponds to the ﬁrst d columns of E pse .
A. T HE P ROOF OF T HEOREM
We need the following lemma for the proof of theorem 1.
Lemma 1.
Unitary matrix does not change the norm of a vector, such that
[Equation]
where U is a unitary matrix and a ∈ R n is a n -dimension column vector.
Proof.
The norm can be expressed by using the Hermitian product, ( · , · )
[Equation]
⇒
[Equation]
Now we can prove the theorem 1.
For any two words w i and w j , if (cid:107) M i, : − M j, : (cid:107) 2 ≤ δ , then (cid:107) E i, : − E j, : (cid:107) 2 ≤ (cid:16) σ ( α − 1)(cid:17) × δ .
Proof.
Let ˜ E denote the E pse deﬁned in section 3.
It is obvious that (cid:107) E i, : − E j, : (cid:107)
= (cid:13)(cid:13)(cid:13) ˜
E i, : − ˜ E j, : (cid:13)(cid:13)(cid:13) 2 since E is the ﬁrst d columns of ˜ E and the last ( n − d ) dimensions in ˜ E are zero.
Let h
= ( h 1 , h 2 , . .
, h n ) denote the ( M i, : − M j, : ) V .
We have (cid:107) h (cid:107)
= (cid:107) ( M i, : − M j, : )
V (cid:107) 2 .
According to lemma 1, (cid:107) ( M i, : − M j, : ) V (cid:107)
= (cid:107) ( M i, : − M j, : ) (cid:107) 2 .
So we have (cid:107) h (cid:107)
= (cid:107) ( M i, : − M j, : )
V (cid:107)
= (cid:107) ( M i, : − M j, : ) (cid:107) 2 ≤ δ .
(cid:107) E i, : − E j, : (cid:107)=
[Equation]
i, : − ˜ E j,
[Equation] [Equation]
+
[Equation]
d )
[Equation]
(note σ d is the smallest singular value)
[Equation]
A. T HE P ROOF OF T HEOREM
Proof.
Let M pse be U Σ pse V T .
It is easy to show that M pse equals to U d · Σ d · V Td .
Let ˜ M denote M pse .
We ﬁrst show that (cid:13)(cid:13)(cid:13) M i, : − ˜ M
i, : (cid:13)(cid:13)(cid:13) 2 2 (for every i ) is constrained by the singular values of M .
Firstly, we have
[Equation]
So we have
[Equation]
M − ˜ M (cid:13)(cid:13)(cid:13)= (cid:13)(cid:13) U Σ V T − U Σ pse V T
(cid:13)(cid:13)= (cid:13)(cid:13)
U ( Σ − Σ pse ) V T (cid:13)(cid:13)= (cid:107) ( Σ − Σ pse ) (cid:107)
(lemma 1) =(
σ 2 d +1 + σ 2 d +2 + . . .
+ σ 2 n ) 1 /
Then we show if (cid:107) E i, : − E j, : (cid:107) 2 ≤ δ , then (cid:13)(cid:13)(cid:13)
˜ M i, : − ˜ M j, : (cid:13)(cid:13)(cid:13) 2 ≤ δσ (1 − α ).
(cid:13)(cid:13)(cid:13) ˜
M i, : − ˜ M j, : (cid:13)(cid:13)(cid:13)= (cid:13)(cid:13)(cid:13) ( E i, : − E j, : ) · Σ (1 − α ) d · V Td (cid:13)(cid:13)(cid:13)= (cid:13)(cid:13)(cid:13) ( E i, : − E j, : ) · Σ (1 − α ) d (cid:13)(cid:13)(cid:13) 2 (lemma 1)
Let h
= ( h 1 , h 2 . .
, h n ) denote E i, : − E j, : here.
We have (cid:107) h (cid:107)
= (cid:107) E i, : − E j, : (cid:107) 2 ≤ δ .
So the above equation becomes
[Equation]
≤
[Equation]
With above preparations, we can now prove theorem 2.
[Equation]
A. T HE P ROOF OF T HEOREM
Proof.
Similar to the proof in section
A
[Equation]
2, let ˜ E denote the E pse and h
=
( h 1 , h , 2 . . .
, h ) denote the (( M γ 1 , : − M γ 2 , : ) − ( M β 1 , : − M β 2 , : )) V .
We have (cid:107) h (cid:107)
= (cid:107) (( M γ 1 , : − M γ 2 , : ) − ( M β 1 , : − M β 2 , : ))
V (cid:107)
= (cid:107) (( M γ 1 , : − M γ 2 , : ) − ( M β 1 , : − M β 2 , : ))
(cid:107) 2 ≤ δ .
Besides, (cid:107) ( E γ 1 , : − E γ 2 , : ) − ( E β 1 , : − E β 2 , : ) (cid:107)
= (cid:13)(cid:13)(cid:13) ( ˜ E γ 1 , : − ˜ E
γ 2 , : ) − ( ˜ E β 1 , : − ˜ E β 2 , : ) (cid:13)(cid:13)(cid:13) 2 since E is the ﬁrst d columns of ˜ E and the last ( n − d ) dimensions in ˜ E are zero.
(cid:107) ( E γ 1 , : − E γ 2 , : ) − ( E β 1 , : − E β 2 , : )
(cid:107)= (cid:13)(cid:13)(cid:13) ( ˜ E γ 1 , : − ˜ E
γ 2 , : ) − ( ˜ E β 1 , : − ˜ E β 2 , : ) (cid:13)(cid:13)(cid:13)= (cid:13)(cid:13)(cid:13) ( M γ 1 , : − M γ 2 , : ) · V · Σ ( α − 1) pse − ( M β 1 , : − M β 2 , : ) · V · Σ ( α − 1) pse (cid:13)(cid:13)(cid:13)= (cid:13)(cid:13)(cid:13) (( M γ 1 , : − M γ 2 , : ) − ( M β 1 , : − M β 2 , : )) · V · Σ ( α − 1) pse (cid:13)(cid:13)(cid:13)= (cid:13)(cid:13)(cid:13) h · Σ ( α − 1) pse (cid:13)(cid:13)(cid:13)=(( σ ( α − 1)h 1 ) 2 + ( σ ( α − 1)h 2 ) 2 + . . .
+ ( σ ( α − 1) d h d )
+
+ . . .
+ 0) 1 /≤ σ ( α − 1) d · ( h 21 + h 22 + . . .
+ h 2 d ) 1 / 2 (note σ d is the smallest singular value) ≤ σ ( α − 1) d · (cid:107) h (cid:107) 2 ≤ σ ( α − 1) d · δ
A. T HE P ROOF OF T HEOREM
The proof is quite similar to the proof in section A.3.
Let M pse be U Σ pse V T .
According to section
A.3, we know M pse equals to U d · Σ d · V td .
Let ˜ M denote M pse .
According to section A.3, we have (cid:13)(cid:13)(cid:13) M i, : − ˜ M
i, : (cid:13)(cid:13)(cid:13) 2 ≤ ( σ 2 d +1 + σ 2 d +2 + . . .
+ σ 2 n ) 1 / 2 (for every i ).
Then with the same process in section A.3, we show if (cid:107) ( E γ 1 , : − E γ 2 , : ) − ( E β 1 , : − E β 2 , : ) (cid:107) 2 ≤ δ , then (cid:13)(cid:13)(cid:13) (( ˜ M γ 1 , : − ˜ M γ 2 , : ) − ( ˜ M β 1 , : − ˜ M β 2 , : ))
(cid:13)(cid:13)(cid:13) 2 ≤ δσ (1 − α ).
[Equation]
(lemma 1)
Let h
= ( h 1 , h 2 . .
, h n ) denote (( E γ 1 , : − E γ 2 , : ) − ( E β 1 , : − E β 2 , : )) here.
We have (cid:107) h (cid:107)
= (cid:107) (( E γ 1 , : − E γ 2 , : ) − ( E β 1 , : − E β 2 , : ))
(cid:107) 2 ≤ δ .
So the above equation becomes
[Equation]
With above preparations, we can now prove theorem 4.
Proof.
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
= δσ (1 − α )
+ 4( n (cid:88) k
= d + σ 2 k ) 1 /
A. T HE PROOF OF ALPHA REGULARIZATION
Proof.
Firstly, we have
[Equation]
Td =  
[Equation]
v Tλ
[Equation]
v T... λ
1 −
[Equation]
d v Td  
Since U and V is orthogonal, we have
[Equation]
Then, if ( (cid:107) e i (cid:107) 22 ·
[Equation]
c i (cid:107) 22 ) (cid:54) = 1 , we have
[Equation]
A. T HE OPTIMAL α IN TERMS OF MINIMIZING THE WEIGHTED SUMMATION OF THE BOUNDS IN T HEOREM 1 AND
We assume that the optimal α should minimize the outcome bounds in both Theorem 1 ( B 1 ) and 2 ( B 2 ) 7 .
The reason is that if two words are close in the original M space, these two words should also be close in the E space because we do not want the relative distances of words change dramatically, and vice versa.
Thus we derive the optimal α by minimizing the weighted summation of the bounds
[Equation]
where λ is the weight coefﬁcient in the summation.
We derive the optimal α in three cases, i.e. Case 1: σ 1 ≥ σ d ≥ 1 , Case 2: σ 1 ≥ 1 ≥ σ d , Case 3: σ d ≤ σ 1 ≤ 1 .
For Case 1, when α ≥ 1 , we have
[Equation]
Easy to prove that B is a convex function, so we derive α by solving the equation ∂B∂α = 0 , for simplicity, we use k = 1 − λλ ,
[Equation]
Similarly, when α ≤ 1 , we have
[Equation]
Then we have the optimal α in terms of k as
[Equation]
For Case 2, from the formulation of B , we can easily derive the optimal α is 1.
For Case 3, similar with the derivation of Case 1, we have the optimal α in terms of k as,
[Equation]
A.
P ERFORMANCE ON WORD SIMILARITY TASK AND THE CORRELATION SCORE DEFINED IN EQUATION 3 WITH DIFFERENT
[Table]
− 0 .0.740 / 0.
[Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
A. T HE NEIGHBORHOOD OF summer WITH DIFFERENT α
[Table]
precipitation
[Table]
seasons
[Table]
months
[Table]
months
[Table]
months
[Table]
, monsoon, winters
[Table]
snowfall, precipitation, humidity, temperatures, seasons, year, climates, freezing, humid, windy
[Table]
spring
[Table]
snowfall, precipitation, humidity, temperatures, seasons, year, climates, humid, freezing,
windy
[Table]
, rainy, spring, summers, winters, monsoon, nights, warmest, coldest snowfall, precipitation, humidity, temperatures, seasons, climates, year, humid, freezing, windy 0. winter, autumn, rainy, spring, summers, winters, monsoon, nights, warmest, coldest snowfall, precipitation, humidity, temperatures, seasons, climates, humid, year, windy, freezing 0.
winter, autumn, rainy, spring, summers, winters, monsoon, nights, warmest, coldest snowfall, precipitation, temperatures, humidity, seasons, year, humid, climates, windy, freezing 0. winter, autumn, rainy, spring, summers, winters, monsoon, nights, warmest, coldest snowfall, precipitation, temperatures, humidity, seasons, year, humid, climates, windy, sunny 0.
winter, autumn, rainy, spring, summers, winters, monsoon, nights, warmest, snowfall coldest, precipitation, temperatures, seasons, humidity, year, humid, windy, climates,
sunny
[Table]
winter, autumn, rainy, spring, summers, winters, nights, monsoon, snowfall, warmest coldest, precipitation, seasons, temperatures, year, humidity, windy, humid, climates, days 0.
winter, autumn, rainy, spring, summers, nights, winters, monsoon, snowfall, warmest coldest, seasons, precipitation, year, days, temperatures, weekend , windy, sunny , humidity
[Table]
winter, autumn, rainy, spring, nights, summers, winters, monsoon, snowfall, warmest coldest, seasons, year, days, weekend, precipitation, annual , sunny, temperatures, month
[Table]
winter, autumn, rainy, spring, nights, summers, winters, monsoon, snowfall, warmest days, coldest, year, seasons, weekend, annual, week , sunny, month, snow 0.
winter, autumn, spring, rainy, nights, winters, summers, days, monsoon, year weekend, seasons, snowfall, annual, week, night , warmest, coldest, snow, month
[Table]
winter, autumn, spring, nights, rainy, days, winters, summers, night, year weekend, annual, week, seasons, monsoon, snow, holiday , rain , month, day 1.
winter, autumn, spring, nights, rainy
[Table]