A N E MPIRICAL S TUDY ON P OST - PROCESSING M ETHODS FOR W ORD E MBEDDINGS
Anonymous authors
Paper under double-blind review
A BSTRACT
Word embeddings learnt from large corpora have been adopted in various applications in natural language processing and served as the general input representations to learning systems.
Recently, a series of post-processing methods have been proposed to boost the performance of word embeddings on similarity comparison and analogy retrieval tasks, and some have been adapted to compose sentence representations.
The general hypothesis behind these methods is that by enforcing the embedding space to be more isotropic, the similarity between words can be better expressed.
We view these methods as an approach to shrink the covariance/gram matrix, which is estimated by learning word vectors, towards a scaled identity matrix.
By optimising an objective in the semi-Riemannian manifold with Centralised Kernel Alignment (CKA), we are able to search for the optimal shrinkage parameter, and provide a post-processing method to smooth the spectrum of learnt word vectors which yields improved performance on downstream tasks.
I NTRODUCTION
Distributed representations of words have been widely dominating the Natural Language Processing research area and other related research areas where discrete tokens in text are part of the learning systems.
Fast learning algorithms are being proposed, criticised, and improved.
Despite the fact that there exist various learning algorithms (Mikolov et al., 2013a; Pennington et al., 2014; Bojanowski et al., 2017) for producing high-quality distributed representations of words, the main objective is roughly the same, which is drawn from the Distributional Hypothesis (Harris, 1954).
The algorithms assume that there is a smooth transition of meaning at the word-level, thus they learn to assign higher similarity for adjacent word vectors than those that are not adjacent.
The overwhelming success of distributed word vectors leads to subsequent questions and analyses on the information encoded in the learnt space.
As the learning algorithms directly and only utilise the co-occurrence provided by large corpora, it is easy to hypothesise that the learnt vectors are correlated with the frequency of each word, which may not be relevant to the meaning of a word (Turney & Pantel, 2010) and might hurt the expressiveness of the learnt vectors.
One can theorise the frequency-related components in the learnt vector space, and remove them (Arora et al., 2017; Mu et al., 2018; Ethayarajh, 2018).
These post-processing methods, whilst very effective and appealing, are derived from heavy assumptions on the representation geometry and the similarity measure.
In this work we re-examine the problem of post-processing word vectors as a shrinkage estimation of the true/underlying oracle gram matrix of words, which is a rank-deﬁcient matrix due to the existence of synonyms and antonyms.
Constrained from the semi-Riemannian manifold (Benn &
Tucker, 1987; Abraham et al., 1983) where positive semi-deﬁnite matrices, including gram matrices, exist, and Centralised Kernel Alignment (Cortes et al., 2012), we are able to deﬁne an objective to search for the optimal shrinkage parameter, also called mixing parameter, that is used to mix the estimated gram matrix and a predeﬁned target matrix with the maximum similarity with the oracle matrix on the semi-Riemannian manifold
Our contribution can be considered as follows
We deﬁne the post-processing on word vectors as a shrinkage estimation, in which the estimated gram matrix is calculated from the pretrained word embeddings, and the target matrix is a scaled identity matrix for smoothing the spectrum of the estimated one.
The goal is to ﬁnd the optimal mixing parameters to combine the estimated gram matrix with the target in the semi-Riemannian manifold that maximises the similarity between the combined matrix and the oracle one.
We choose to work with the CKA, as it is invariant to isotropic scaling and also rotation, which is a desired property as the angular distance (cosine similarity) between two word vectors is commonly used in downstream tasks for evaluating the quality of learnt vectors.
Instead of directly deriving formulae from the CKA (Eq. 1), we start with the logarithm of CKA (Eq. 2), and then ﬁnd an informative lower bound (Eq. 5).
The remaining analysis is done on the lower bound.
A useful objective is derived to search for the optimal mixing parameter, and the performance on the evaluation tasks shows the effectiveness of our proposed method.
Compared with two other methods, our approach shows that the shrinkage estimation on a sub-Riemannian manifold works better than that on an Euclidean manifold for word embedding matrices, and it provides an alternative way to smooth the spectrum by adjusting the power of each linearly independent component instead of removing top ones.
R ELATED W ORK
Several post-processing methods have been proposed for readjusting the learnt word vectors (Mu et al., 2018; Arora et al., 2017; Ethayarajh, 2018) to make the vector cluster more isotropic w.r.t.
the origin such that cosine similarity or dot-product can better express the relationship between word pairs.
A common practice is to remove top principal components of the learnt word embedding matrix as it has been shown that those directions are highly correlated with the occurrence.
Despite the effectiveness and efﬁciency of this family of methods, the number of directions to remove and also the reason to remove are rather derived from empirical observations, and they don’t provide a principled way to estimate these two factors.
In our case, we derive a useful objective that one could take to search for the optimal mixing parameter on their own pretrained word vectors.
As described above, we view the post-processing method as a shrinkage estimation on the semiRiemannian manifold.
Recent advances in machine learning research have developed various ways to learn word vectors in hyperbolic space (Nickel & Kiela, 2017; Dhingra et al., 2018; Gulcehre et al., 2019).
Although the Riemannian manifold is a speciﬁc type of hyperbolic geometry, our data samples on the manifold are gram matrices of words instead of word vectors themselves in previous work, , which makes our work different from previous ones.
The word vectors themselves are still considered in Euclidean space for the simple computation of similarity between pairs, and the intensity of the shrinkage estimation is optimised to adjust the word vectors.
Our work is also related to previously proposed shrinkage estimation of covariance matrices (Ledoit & Wolf, 2004;
Sch ¨ afer & Strimmer, 2005).
These methods are practically useful when the number of variables to be estimated is much more than that of the samples available.
However, the mixing operation between the estimated covariance matrix and the target matrix is a weighted sum, and ldistance is used to measure the difference between the true/underlying covariance matrix and the mixed covariance matrix.
As known, covariance matrices are positive semi-deﬁnite, thus they lie on a semi-Riemannian manifold with a manifold-observing distance measure.
A reasonable choice is to conduct the shrinkage estimation on the semi-Riemannian manifold, and CKA is applied to measure the distance as rotations and istropic scaling should be ignored with measuring two sets of word vectors.
The following sections will discuss our approach in details.
S HRINKAGE OF G RAM M ATRIX
Learning word vectors can be viewed as estimating the oracle gram matrix where the relationship of words is expressed, and the post-processing method we propose here aims to best recover the oracle gram matrix.
Suppose there exists an ideal gram matrix K ∈ R n × n where each entry represents the similarity of a word pair deﬁned by a kernel function k ( w i , w j ) .
Given the assumption of a gram matrix and the existence of polysemies, the oracle gram matrix K is positive semi-deﬁnite and its rank is between 0 and n and denoted as k .
A set of pretrained word vectors is provided by a previously proposed algorithm, and for simplicity, an embedding matrix E ∈ R n × d is constructed in which each row is a word vector v i .
The estimated gram matrix K (cid:48)
=
EE (cid:62) ∈ R n × n has rank d and d ≤ k .
It means that overall, the previously proposed algorithms give us a low-rank approximation of the oracle gram matrix K .
The goal of shrinkage is to, after the estimated gram matrix K (cid:48) is available, ﬁnd an optimal postprocessing method to maximise the similarity between the oracle gram matrix K and the estimated one K (cid:48) given simple assumptions about the ideal one without directly ﬁnding it.
Therefore, a proper similarity measure that inherits and respects our assumptions is crucial.
3. C ENTRALISED K ERNEL A LIGNMENT (CKA)
The similarity measure between two sets of word vectors should be invariant to any rotations and also to isotropic scaling .
A reasonable choice is the Centralised Kernel Alignment (CKA) (Cortes et al., 2012) deﬁned below:
[Equation]
where K c = (cid:2) I − 1 n 11 (cid:62) (cid:3) K (cid:2)
I − 1 n 11 (cid:62) (cid:3) , and (cid:104) K c , K (cid:48)
c (cid:105) F =
Tr( K c K (cid:48) c ) .
For simplicity of the derivations below, we assume that the ideal gram matrix K is centralised, and the estimated one K (cid:48) can be centralised easily by removing the mean of word vectors.
In the following section, K and K (cid:48) are used to denote centralised gram matrices.
As shown in Eq.
1, it is clear that ρ ( K , K (cid:48) ) is invariant to any rotations and isotropic scaling, and doesn’t suffer from the issues of K and K (cid:48) being low-rank.
The centralised kernel alignment has been recommended recently as a proper measure for comparing the similarity of features learnt by neural networks (Kornblith et al., 2019).
As learning word vectors can be thought of as a one-layer neural network with linear activation function, ρ ( K , K (cid:48) ) is a reasonable similarity measure between the ideal gram matrix and our shrunken one provided by learning algorithms.
Given the CauchySchwartz inequality and also the non-negativity of the Frobenius norm, ρ ( K , K (cid:48) ) ∈ [0 , 1] .
Since both K and K (cid:48) are positive semi-deﬁnite, their eigenvalues are denoted as { λ σ 1 , λ σ 2 , ..., λ σ k } and { λ ν 1 , λ ν 2 , ..., λ ν d } respectively, and the eigenvalues of KK (cid:48) are denoted as { λ 1 , λ 2 , ..., λ d } as the rank is determined by the matrix with lower rank.
The log-transformation is conducted to simplify the derivation.
[Equation]
where the lower bound is given by the AM − GM inequality, and the equality holds when all eigenvalues of KK (cid:48) are the same λ
1 = λ 2 = ...
=
λ d .
3. S HRINKAGE OF G RAM M ATRIX ON SEMI -R IEMANNIAN M ANIFOLD
As the goal is to ﬁnd a post-processing method that maximises the similarity ρ ( K , K (cid:48) ) , a widely adopted approach is to shrink the estimated gram matrix K (cid:48) towards the target matrix T with a predeﬁned structure.
The target matrix T is usually positive semi-deﬁnite and has same or higher rank than the estimated gram matrix.
In our case, we assume T is full rank as it simpliﬁes equations.
Previous methods rely on a linear combination of K (cid:48) and T in Euclidean space (Ledoit & Wolf, 2004; Sch¨afer & Strimmer, 2005).
However, as gram matrices are positive semi-deﬁnite, they nat- urally lie on the semi-Riemannian manifold.
Therefore, a suitable option for shrinkage is to move the estimated gram matrix K (cid:48) towards the target matrix T on a semi-Riemannian manifold (Brenier, 1987), and the resulting matrix Y is given by
[Equation]
where β ∈ [0 , 1] is the mixing parameter that indicates the strength of the shrinkage, and T 1 / 2 is the square root of the matrix T .
The objective is to ﬁnd the optimal β that maximises log ρ ( K , K (cid:48) ) .
[Equation]
The objective deﬁned in Eq.
4 is hard to work with, instead, we maximise the lower bound in Eq.
2 to ﬁnd optimal mixing parameter β (cid:63) .
By plugging Y into Eq. 2 and denoting λ y i as the i -th eigenvalue of Y , we have:
[Equation]
3. S CALED I DENTITY M ATRIX AS T ARGET FOR S HRINKAGE
Recent progress in analysing pretrained word vectors (Mu et al., 2018; Arora et al., 2017) recom- mended to make them isotropic by removing top principal components as they highly correlate with the frequency information of words, and it results in a more compressed eigenspectrum.
In Eu- clidean space-based shrinkage methods (Anderson, 2004; Sch¨afer & Strimmer, 2005), the spectrum is also smoothed by mixing the eigenvalues with ones to balance the overestimated large eigenvalues and underestimated small eigenvalues.
In both cases, the goal is to smooth the spectrum to make it more isotropic, thus an easy target matrix to start with is the scaled identity matrix, which is noted as T
= α
I , where α is the scaling parameter
and I is the identity matrix.
Thus, the resulting matrix
Y and the lower bound in Eq.
2 become
[Equation]
It is easy to show that, when β = 0 , the resulting matrix Y becomes the target matrix α I , and when β = 1 , no shrinkage is performed as Y = K (cid:48) .
Eq.
7 indicates that the lower bound is only a function of β with no involvement from the scaling parameter α brought by the target matrix as the CKA is invariant to isotropic scaling.
3. N OISELESS E STIMATION
Starting from the simplest case, we assume that the estimation process from learning word vectors E to constructing the estimated gram matrix K (cid:48) is noiseless, thus the ﬁrst order and the second order derivative of L ( β ) with respect to β are crucial for ﬁnding the optimal β (cid:63) :
[Equation]
Since L ( β ) is invariant to isotropic scaling on { λ ν i | i ∈ 1 , 2 , ..., d } , we are able to scale them by the inverse of their sum, then p ν i =
λ
ν
i / (cid:80) di =1 λ ν i deﬁnes a distribution.
To avoid losing information of the estimated gram matrix K (cid:48) , we redeﬁne the plausible range for β as (0 , 1] .
Eq.
8 can be interpreted as
[Equation]
where D KL ( q || p ) is the Kullback-Leibler divergence and it is non-negative, H ( q ) is the entropy of the uniform distribution q , and H ( r ( β ) , p ) is the cross-entropy for r and p .
The inequality derives from the fact that the uniform distribution has the maximum entropy.
Thus, the ﬁrst order derivative L (cid:48) ( β ) is always less than 0 .
With the same deﬁnition of q i and r ( β ) i in Eq.
10, the second order derivative L (cid:48)(cid:48) ( β ) can be rewritten as
[Equation]
As shown, L (cid:48) ( β ) < 0 means that L ( β ) monotonically decreases in the range of (0 , 1] for β , and
the resulting matrix Y =
α I completely ignores the estimated gram matrix K (cid:48) .
L (cid:48) ( β ) also monotonically decreases in the range of (0 , 1] for β because L (cid:48)(cid:48) ( β ) < 0 .
The observation is similar to the ﬁnding reported in previous work (Sch ¨ afer & Strimmer, 2005) on shrinkage estimation of covariance matrices that any target will increase the similarity between the oracle covariance matrix and the mixed covariance matrix.
In our case, simply reducing β from 1 to 0 increases Eq. 2, and consequently, β → 0 + loses all information estimated which is not ideal.
However, one can rewrite Eq.
11 as L (cid:48)(cid:48) ( β ) =
H ( r ( β ) , p ) 2 − H ( q ) H ( r ( β ) , p ) and q is the uniform distribution which has maximum entropy, the derivative of L (cid:48)(cid:48)
( β ) with respect to H ( r ( β ) , p ) gives
2 H ( r ( β ) , p ) − H ( q ) , and by setting the derivative to 0 , it tells us that there exists a β which results in a speciﬁc r that gives H ( r ( β ) , p ) =
12 H ( q ) .
Since H ( r ( β ) , p ) monotonically increases when β moves from 0 + to 1 , and lim β → 0 + H ( r ( β ) , p )
=
H ( q, p ) <
12 H ( q ) and H ( r (1) , p ) > 12 H ( q ) , there exists a single β ∈ (0 , 1] that gives the smallest value of L (cid:48)(cid:48) ( β ) .
This indicates that there exists a ˆ β that leads to the slowest change in the function value L ( β ) .
Then, we simply set β (cid:63) =
ˆ β .
In practice, β should be larger than 0 .
5 in order not to make the resulting matrix overpowered by the target matrix, so one could run a binary search on β ∈ [0 . 5 , 1] that gives smallest value of L (cid:48)(cid:48)
( β ) .
Intuitively, this method is similar to using an elbow plot to ﬁnd the optimal number of components to keep when running Principal Component Analysis (PCA) on observed data.
To translate in our case, a larger value of β leads to ﬂatter spectrum and higher noise level as the scaled identity matrix represents isotropic noise.
The optimal β (cid:63) gives us a point where the increase of L ( β ) is the slowest.
Once optimal mixing parameters β (cid:63) are found, the resulting matrix Y = K (cid:48) β (cid:63)
=
E β (cid:63) ( E β (cid:63) ) (cid:62) , and α is omitted here.
As a post-processing method for word vectors, we propose to transform the word vectors in the following fashion:
E =
E − 1 n (cid:80) ni
=1 E i · 2.
Singular Value Decomposition E = USV (cid:62) 3.
Optimal Mixing Parameter β
(cid:63) =
arg min β L (cid:48)(cid:48) 4.
Reconstruct the matrix E (cid:63) =
U ( S ) β (cid:63) V (cid:62)
E XPERIMENTS
Three learning algorithms are applied to derive word vectors, including skipgram (Mikolov et al., 2013b), CBOW (Mikolov et al., 2013a) and GloVe (Pennington et al., 2014).
2 EE (cid:62) serves as the estimated gram matrix K (cid:48) .
Hyperparameters of each algorithm are set to the recommended values.
The training corpus is collected from Wikipedia and it contains around 4600 million tokens.
The unique tokens include ones that occur at least 100 times in the corpus and the resulting dictionary size is 366990.
After learning, the word embedding matrix is centralised and then the optimal mixing parameter β (cid:63) is directly estimated from the singular values of the centralised word vectors.
Afterwards, the postprocessed word vectors are evaluated on a series of word-level tasks to demonstrate the effectiveness of our method.
Although the range of β is set to [0 . 5 , 1 . 0] and the optimal value is found by minimising L (cid:48)(cid:48) ( β ) , it seldomly hits 0 .
5 , which means that searching is still necessary.
4. C OMPARISON P ARTNERS
Two comparison partners are chosen to demonstrate the effectiveness of our proposed method.
One is the method (Mu et al., 2018) that removes the top 2 or 3 principal components on the zero-centred word vectors, and the number of top principal components to remove depends on the training algorithm used for learning word vectors, and possibly the training corpus.
Thus, the selection process is not automatic, while ours is.
The hypothesis behind this method is similar to ours, as the goal is to make the learnt word vectors more isotropic.
However, removing top principal components, which are correlated with the frequency of each word, could also remove relevant information that indicates the relationship between words.
The observation that our method is able to perform on par with or better than this comparison partner supports our hypothesis.
The other comparison partner is the optimal shrinkage (Ledoit & Wolf, 2004) on the estimated covariance matrices as the estimation process tends to overestimate the directions with larger power and underestimate the ones with smaller power, and the optimal shrinkage aims to best recover the true covariance matrix with minimum assumptions about it, which has the same goal as our method.
The resulting matrix Y is deﬁned to be a linear combination of the estimated covariance matrix and the target with a predeﬁned structure, and it is denoted as Y =
(1 − β )
α I + β Σ (cid:48) , where in our case, Σ (cid:48) =
E (cid:62) E ∈ R d × d .
The optimal shrinkage parameter β is found by minimising the Euclidean distance between the true covariance matrix Σ and the resulting matrix Y .
The issue here is that
Euclidean distance and linear combination may not be the optimal choice for covariance matrices, and also Euclidean distance is not invariant to isotropic scaling which is a desired property when measuring the similarity between sets of word vectors.
4. W ORD - LEVEL E VALUATION T ASKS
Three categories of word-level evaluation tasks are considered, including 8 tasks for word similarity, 3 for word analogy and 6 for concept categorisation.
The macro-averaged results for each category of tasks are presented in Table 1.
Details about tasks are included in the appendix.
[Table]
4. W ORD T RANSLATION T ASKS
Table 2 presents results on supervised and unsupervised word translation tasks (Lample et al., 2018)given pretrained word vectors from FastText (Bojanowski et al., 2017).
The supervised word translation is formalised as solving a Procrustes problem directly.
The unsupervised task ﬁrst trains a mapping through adversarial learning and then reﬁnes through
[Table]
4. S ENTENCE - LEVEL E VALUATION T ASKS
The evaluation tasks include ﬁve tasks from SemEval Semantic Textual Similarity (STS) in 20122016 (Agirre et al., 2012; 2013; 2014; 2015; 2016).
A sentence vector is constructed by simply averaging word vectors or postprocessed word vectors, and the similarity measure is cosine similarity as well.
The performance on each task is reported as the Pearson’s correlation score.
We evaluate our post-processing at two different levels, in which one applies the post-processing methods on word vectors before averaging them to form sentence vectors (Mu et al., 2018), and the other one applies methods on averaged word vectors, which are sentence vectors, directly (Arora et al., 2017).
Applying the post-processing to sentence vectors results in better performance and our method leads to the best performance on each dataset.
Results are reported in Table
[Table] [Table]
D ISCUSSION
The results presented in Table 1, 2 and 3 indicate the effectiveness of our methods.
In addition, by comparing to two most related post-processing methods, which are removing top principal components (Mu et al., 2018; Arora et al., 2017) and the optimal shrinkage estimation on covariance matrices in Euclidean space (Ledoit & Wolf, 2004; Sch ¨ afer & Strimmer, 2005), our method combines the best of the both worlds - the good performance of the ﬁrst method and the automaticity property of the second one.
Speciﬁcally, our method derives the resulting gram matrix in a semi-Riemannian manifold, and the shrinkage is done by moving the estimated gram matrix K (cid:48) to the scaled identity matrix α I on the geodesic on the semi-Riemannian manifold deﬁned in Eq.
Compared to measuring the Euclidean distance between two covariance matrices (Ledoit & Wolf, 2004), we chose a distance measure that respects the structure of the space where positive semi-deﬁnite matrices exist.
Also, since we are working with improving the quality of learnt word vectors, given that mostly the angular distance between a pair of word vectors matters, the similarity measure between two spaces needs to be invariant to rotation and isotropic scaling, and the Centralised Kernel Alignment is a good ﬁt in our case.
Compared to the post-processing method that removes top principal components, our method deﬁnes an objective function where the unique minimum point in the speciﬁc range indicates the optimal shrinkage parameter β (cid:63) , and which doesn’t require any human supervision on ﬁnding the parameters (the number of principal components to remove in that method).
The derivations in our work are mostly done on the lower bound in Eq.
2 of the original log of CKA ρ ( K , K (cid:48) ) , and the bound is tight when KK (cid:48) has a ﬂat spectrum.
In this sense, the lower bound essentially deﬁnes the similarity of the resulting matrix Y and a scaled identity matrix where the scaling factor is the average of the eigenvalues of KK (cid:48) .
As the isotropic scaling can be ignored in our case, intuitively, the lower bound (Eq. 2) gives us how to shrink an estimated gram matrix to an identity matrix.
Since the target matrix T
= α I is also a scaled identity matrix, the lower bound keeps increasing as Y travels from K (cid:48) to T , and it explains why the ﬁrst order derivative is negative and the second order derivative is more informative.
Our method is very similar to using an elbow plot to ﬁnd the number of eigenvalues to keep in PCA.
Future work should focus on ﬁnding a tighter bound that considers the interaction between the true gram matrix K and the estimated one K (cid:48) .
Figure 1 shows that our methods give solid and stable improvement against two comparison partners on learnt word vectors provided by skipgram and CBOW with different dimensions, and slightly worse than removing top PCs on GloVe.
However, skipgram and CBOW provide much better performance thus improvement on these algorithms is more meaningful, and our method doesn’t require manual speculations on the word vectors to determine the number of PCs to remove.
An interesting observation is that there is a limit on the performance improvement from increasing the dimensionality of word vectors, which was hypothesised and analysed in prior work (Yin & Shen, 2018).
Figure 2 shows that our method consistently outperforms two comparison partners when limited amount of
[Figure]
training corpus is provided and the observation is consistent across learnt word vectors with varying dimensions. the perspective of shrinkage estimation of covariance matrices, the main goal is to optimally recover the oracle covariance matrix under a certain assumption about the oracle one when limited data is provided.
The results presented in Figure 2 indicate that our method is better at recovering the gram matrix of words where relationships of word pairs are expressed as it gives better performance on word-level evaluation tasks on average
[Figure]
C ONCLUSION
We deﬁne a post-processing method in the view of shrinkage estimation of the gram matrix.
Armed with CKA and geodesic measured on the semi-Riemannian manifold, a meaningful lower bound is derived and the second order derivative of the lower bound gives the optimal shrinkage parameter β (cid:63) to smooth the spectrum of the estimated gram matrix which is directly calculated by pretrained word vectors.
Experiments on the word similarity and sentence similarity tasks demonstrate the effectiveness of our model.
Compared to the two most relevant post-processing methods (Mu et al., 2018; Ledoit & Wolf, 2004), ours is more general and automatic, and gives solid performance improvement.
As the derivation in our paper is based on the general shrinkage estimation of positive semi-deﬁnite matrices, it is potentially useful and beneﬁcial for other research ﬁelds when Euclidean measures are not suitable.
Future work could expand our work into a more general setting for shrinkage estimation as currently the target in our case is an isotropic covariance matrix.
A PPENDIX A W ORD - LEVEL E VALUATION T ASKS
8 word similarity tasks :
MEN(Bruni et al., 2014), SimLex999(Hill et al., 2015), MTurk(Halawi et al., 2012), WordSim353(Finkelstein et al., 2002), WordSim-353-REL(Agirre et al., 2009),
WordSim-353-SIM(Agirre et al., 2009), RG65(Rubenstein & Goodenough, 1965).
On these, the similarity of a pair of words is calculated by the cosine similarity between two word vectors, and the performance is reported by computing Spearman’s correlation between human annotations with the predictions from the learnt vectors.
3 word analogy tasks :
Google Analogy(Mikolov et al., 2013a), SemEval-2012(Jurgens et al., 2012), MSR(Mikolov et al., 2013c).
The task here is to answer the questions in the form of “ a is to a (cid:63) as b is to b (cid:63) ”, where a , a (cid:63) and b are given and the goal is to ﬁnd the correct answer b (cid:63) .
The ”3CosAdd” method (Levy et al., 2015) is applied to retrieve the answer, which is denoted as arg max b (cid:63) ∈ V w \{ a (cid:63) ,b,a } = cos( b (cid:63) , a (cid:63) − a + b ) .
Accuracy is reported.
6 concept categorisation tasks
BM(Baroni et al., 2010),
AP(Almuhareb, 2006),
BLESS(Jastrzebski et al., 2017), ESSLI(Baroni et al., 2008).
The task is to measure whether words with similar meanings are nicely clustered together.
An unsupervised clustering algorithm with predeﬁned number of clusters is performed on each dataset, and the performance is reported also in accuracy. B P YTHON C ODE g 5mean =
) centred emb = emb − mean u , s , vh
= np . l i n a l g . svd ( centred emb ) 9def o b j e c t i v e ( b ) : l = s ** l o g l = np .
log ( l ) l2b = l
** (2*b ) 14f
i r s t t e r m = np mean ( l o g l * l2b * l o g l ) *np mean ( l2b ) np mean l o g l *
d e i v t v e = − np mean f t t e m
Listing 1: Python code for searching for the optimal β (cid:63)