A LL - BUT - THE -T OP : S IMPLE AND E FFECTIVE P OSTPROCESSING FOR W ORD R EPRESENTATIONS
A BSTRACT
Real-valued word representations have transformed NLP applications; popular examples are word2vec and GloVe, recognized for their ability to capture linguistic regularities.
In this paper, we demonstrate a very simple , and yet counter-intuitive, postprocessing technique – eliminate the common mean vector and a few top dominating directions from the word vectors – that renders off-the-shelf representations even stronger .
The postprocessing is empirically validated on a variety of lexical-level intrinsic tasks (word similarity, concept categorization, word analogy) and sentence-level tasks (semantic textural similarity and text classiﬁcation) on multiple datasets and with a variety of representation methods and hyperparameter choices in multiple languages; in each case, the processed representations are consistently better than the original ones
I NTRODUCTION
Words and their interactions (as sentences) are the basic units of natural language.
Although words are readily modeled as discrete atomic units, this is unable to capture the relation between the words.
Recent distributional real-valued representations of words (examples: word2vec, GloVe) have transformed the landscape of NLP applications – for instance, text classiﬁcation (Socher et al., 2013b;
Maas et al., 2011; Kim, 2014), machine translation (Sutskever et al., 2014; Bahdanau et al., 2014) and knowledge base completion (Bordes et al., 2013; Socher et al., 2013a).
The success comes from the geometry of the representations that efﬁciently captures linguistic regularities: the semantic similarity of words is well captured by the similarity of the corresponding vector representations.
A variety of approaches have been proposed in recent years to learn the word representations:
Collobert et al.
(2011); Turian et al.
(2010) learn the representations via semi-supervised learning by jointly training the language model and downstream applications; Bengio et al.
Mikolov et al.
; Huang et al.
(2012) do so by ﬁtting the data into a neural network language model; Mikolov et al.
; Mnih & Hinton (2007) by log-linear models; and Dhillon et al.
; Pennington et al.
; Levy & Goldberg (2014); Stratos et al.
; Arora et al. (2016) by producing a lowdimensional representation of the cooccurrence statistics.
Despite the wide disparity of algorithms to induce word representations, the performance of several of the recent methods is roughly similar on a variety of intrinsic and extrinsic evaluation testbeds.
In this paper, we ﬁnd that a simple processing renders the off-the-shelf existing representations even stronger .
The proposed algorithm is motivated by the following observation.
Observation Every representation we tested, in many languages, has the following properties:
• The word representations have non-zero mean – indeed, word vectors share a large common vector (with norm up to a half of the average norm of word vector).
• After removing the common mean vector, the representations are far from isotropic – indeed, much of the energy of most word vectors is contained in a very low dimensional subspace (say, 8 dimensions out of 300).
Implication Since all words share the same common vector and have the same dominating directions, and such vector and directions strongly inﬂuence the word representations in the same way, we propose to eliminate them by: (a) removing the nonzero mean vector from all word vectors, effectively reducing the energy; (b) projecting the representations away from the dominating D directions, effectively reducing the dimension.
Experiments suggest that D depends on the representations (for example, the dimension of the representation, the training methods and their speciﬁc hyperparameters, the training corpus) and also depends on the downstream applications.
Nevertheless, a rule of thumb of choosing D around d/ 100 , where d is the dimension of the word representations, works uniformly well across multiple languages and multiple representations and multiple test scenarios.
We emphasize that the proposed postprocessing is counter intuitive – typically denoising by dimensionality reduction is done by eliminating the weakest directions (in a singular value decomposition of the stacked word vectors), and not the dominating ones.
Yet, such postprocessing yields a “puriﬁed” and more “isotropic” word representation as seen in our elaborate experiments.
Experiments By postprocessing the word representation by eliminating the common parts, we ﬁnd the processed word representations to capture stronger linguistic regularities.
We demonstrate this quantitatively, by comparing the performance of both the original word representations and the processed ones on three canonical lexical-level tasks:
• word similarity task tests the extent to which the representations capture the similarity between two words – the processed representations are consistently better on seven different datasets, on average by 1.7%; • concept categorization task tests the extent to which the clusters of word representations capture the word semantics – the processed representations are consistently better on three different datasets, by 2.8%, 4.5% and 4.3%; • word analogy task tests the extent to which the difference of two representations captures a latent linguistic relation – again, the performance is consistently improved (by 0.5% on semantic analogies, 0.2% on syntactic analogies and 0.4% in total).
Since part of the dominant components are inherently canceled due to the subtraction operation while solving the analogy, we posit that the performance improvement is not as pronounced as earlier.
Extrinsic evaluations provide a way to test the goodness of representations in speciﬁc downstream tasks.
We evaluate the effect of postprocessing on a standardized and important extrinsic evaluation task on sentence modeling: semantic textual similarity task – where we represent a sentence by its averaged word vectors and score the similarity between a pair of sentences by the cosine similarity between the corresponding sentence representation.
Postprocessing improves the performance consistently and signiﬁcantly over 21 different datasets (average improvement of 4%).
Word representations have been particularly successful in NLP applications involving supervisedlearning, especially in conjunction with neural network architecture.
Indeed, we see the power of postprocessing in an experiment on a standard text classiﬁcation task using a well established convoluntional neural network (CNN) classiﬁer (Kim, 2014) and three RNN classiﬁers (with vanilla
RNN, GRU (Chung et al., 2015) and LSTM Greff et al.
as recurrent units).
Across two different pre-trained word vectors, ﬁve datasets and four different architectures, the performance with processing improves on a majority of instances (34 out of 40) by a good margin (2.85% on average), and the two performances with and without processing are comparable in the remaining ones.
Related Work.
Our work is directly related to word representation algorithms, most of which have been elaborately cited.
Aspects similar to our postprocessing algorithm have appeared in speciﬁc NLP contexts very recently in (Sahlgren et al., 2016)
(centering the mean) and (Arora et al., 2017) (nulling away only the ﬁrst principal component).
Although there is a superﬁcial similarity between our work and (Arora et al.
, the nulling directions we take and the one they take are fundamentally different.
Speciﬁcally, in Arora et al.
(2017), the ﬁrst dominating vector is *dataset-speciﬁc*, i.e., they ﬁrst compute the sentence representation for the entire semantic textual similarity dataset, then extract the top direction from those sentence representations and ﬁnally project the sentence representation away from it.
By doing so, the top direction will inherently encode the common information across the entire dataset, the top direction for the "headlines" dataset may encode common information about news articles while the top direction for "Twitter’15" may encode the common information about tweets.
In contrast, our dominating vectors are over the entire vocabulary of the language.
More generally, the idea of removing the top principal components has been studied in the context of positive-valued, high-dimensional data matrix analysis (Bullinaria & Levy, 2012; Price et al., 2006).
Bullinaria & Levy (2012) posits that the highest variance components of the cooccurrence matrix are corrupted by information other than lexical semantics, thus heuristically justifying the removal of the top principal components.
A similar idea appears in the context of population matrix analysis (Price et al., 2006), where the entries are also all positive.
Our postprocessing operation is on dense low-dimensional representations (with both positive and negative entries).
We posit that the postprocessing operation makes the representations more “isotropic” with stronger self-normalization properties – discussed in detail in Section 2 and Appendix
A.
Our main point is that this isotropy condition can be explicitly enforced to come up with new embedding algorithms (of which our proposed post-processing is a simple and practical version).
P OSTPROCESSING
We test our observations on various word representations: four publicly available word representations (WORD2VEC 1 (Mikolov et al., 2013) trained using Google News, GLOVE 2 (Pennington et al., 2014) trained using Common Crawl, RAND-WALK (Arora et al., 2016) trained using Wikipedia and
TSCCA 3 trained using English Gigaword) and two self-trained word representations using CBOW and Skip-gram (Mikolov et al., 2013) on the 2010 Wikipedia corpus from (Al-Rfou et al., 2013).
The detailed statistics for all representations are listed in Table 1.
For completeness, we also consider the representations on other languages: a detailed study is provided in Appendix C.2.
[Table]
[Figure]
Let v ( w ) ∈
R
d be a word representation for a given word w in the vocabulary V .
We observe the following two phenomena in each of the word representations listed above:
• { v ( w ) : w ∈ V} are not of zero-mean: i.e., all v ( w ) share a non-zero common vector, v ( w ) =
˜ v ( w ) + µ, where µ is the average of all v ( w ) ’s, i.e., µ = 1 / |V| (cid:80) w ∈V v ( w ) .
The norm of µ is approximately 1/6 to 1/2 of the average norm of all v ( w ) (cf.
Table 1)
• { ˜ v ( w ) : w ∈ V} are not isotropic: Let u 1 , ..., u
d be the ﬁrst to the last components recovered by the principal component analysis (PCA) of { ˜ v ( w ) : w ∈ V} , and σ 1 , ..., σ d be the corresponding normalized variance ratio.
Each ˜ v ( w ) can be written as a linear combinations of u : ˜ v ( w )
= (cid:80) di
=
α i ( w ) u i .
As shown in Figure 1,
we observe that σ i decays near exponentially for small values of i and remains roughly constant over the later ones.
This suggests there exists D such that α i (cid:29) α j for all i ≤ D and j (cid:29) D ; from Figure 1 one observes that D is roughly 10 with dimension d = 300 .
Angular Asymmetry of Representations A modern understanding of word representations involves either PMI-based (including word2vec (Mikolov et al., 2010; Levy & Goldberg, 2014) and
GloVe (Pennington et al., 2014)) or CCA-based spectral factorization approaches.
While CCA-based spectral factorization methods have long been understood from a probabilistic (i.e., generative model) view point (Browne, 1979; Hotelling, 1936) and recently in the NLP context (Stratos et al., 2015), a corresponding effort for the PMI-based methods has only recently been conducted in an inspired work (Arora et al., 2016).
Arora et al.
(2016) propose a generative model (named RAND-WALK) of sentences, where every word is parameterized by a d -dimensional vector.
With a key postulate that the word vectors are angularly uniform (“isotropic"), the family of PMI-based word representations can be explained under the RAND-WALK model in terms of the maximum likelihood rule.
Our observation that word vectors learnt through PMI-based approaches are not of zero-mean and are not isotropic (c.f. Section 2) contradicts with this postulate.
The isotropy conditions are relaxed in Section 2.2 of (Arora et al., 2016), but the match with the spectral properties observed in Figure 1 is not immediate.
This contradiction is explicitly resloved by relaxing the constraints on the word vectors to directly ﬁt the observed spectral properties.
The relaxed conditions are: the word vectors should be isotropic around a point (whose distance to the origin is a small fraction of the average norm of word vectors) lying on a low dimensional subspace.
Our main result is to show that even with this enlarged parameter-space, the maximum likelihood rule continues to be close to the PMI-based spectral factorization methods.
A brief summary of RAND-WALK, and the mathematical connection between our work and theirs, are explored in detail in Appendix A.
2. A LGORITHM
Since all word representations share the same common vector µ and have the same dominating directions and such vector and directions strongly inﬂuence the word representations in the same way, we propose to eliminate them, as formally achieved as Algorithm 1.
[Table]
(cid:62) [Table]
w
[Table] (cid:1) u [Table]
Output : Processed representations v (cid:48) ( w ) .
Signiﬁcance of Nulled Vectors Consider the representation of the words as viewed in terms of the top D PCA coefﬁcients α (cid:96) ( w ) , for 1 ≤ (cid:96) ≤ D .
We ﬁnd that these few coefﬁcients encode the frequency of the word to a signiﬁcant degree; Figure 2 illustrates the relation between the ( α 1 ( w ) , α 2 ( w )) and the unigram probabilty p ( w ) , where the correlation is geometrically visible.
Discussion In our proposed processing algorithm, the number of components to be nulled, D , is the only hyperparameter that needs to be tuned.
We ﬁnd that a good rule of thumb is to choose D approximately to be d/ 100 , where d is the dimension of a word representation.
This is empirically justiﬁed in the experiments of the following section where d = 300 is standard for published word representations.
We trained word representations for higher values of d using the WORD2VEC and
GLOVE algorithms and repeated these experiments; we see corresponding consistent improvements due to postprocessing in Appendix C.
2. P OSTPROCESSING AS A “R OUNDING ” TOWARDS I SOTROPY
The idea of isotropy comes from the partition function deﬁned in (Arora et al., 2016),
[Equation]
where Z ( c ) should approximately be a constant with any unit vector c (c.f. Lemma 2.1 in (Arora et al., 2016)).
Hence, we mathematically deﬁne a measure of isotropy as follows,
[Equation]
where I ( { v ( w ) } ) ranges from 0 to 1, and I ( { v ( w ) } ) closer to 1 indicates that { v ( w ) } is more isotropic.
The intuition behind our postprocessing algorithm can also be motivated by letting
I ( { v ( w ) } )
→ 1 .
Let V be the matrix stacked by all word vectors, where the rows correspond to word vectors, and 1 |V| be the |V| -dimensional vectors with all entries equal to one, Z ( c ) can be equivalently deﬁned as follows,
[Equation]
I ( { v ( w ) } ) is, therefore, can be very coarsely approximated by,
• A ﬁrst order approximation :
[Equation]
Letting I ( { v ( w ) } )
= 1 yields (cid:107) 1 (cid:62)|V| V (cid:107) = 0 , which is equivalent to (cid:80) w ∈V v ( w ) = 0 .
The intuition behind the ﬁrst order approximation matches with the ﬁrst step of the proposed algorithm, where we enforce v ( w ) to have a zero mean.
• A second order approximation :
[Equation]
where σ min and σ max are the smallest and largest singular value of V , respectively.
Letting
I ( { v ( w ) } )
= 1 yields (cid:107) 1 (cid:62)|V| V (cid:107) = 0 and σ min = σ max .
The fact that σ min = σ max suggests the spectrum of v ( w ) ’s should be ﬂat.
The second step of the proposed algorithm removes the highest singular values, and therefore explicitly ﬂatten the spectrum of V .
Empirical Veriﬁcation
Indeed, we empirically validate the effect of postprocessing of on
I ( { v ( w ) } ) .
Since there is no closed-form solution for arg max (cid:107) c (cid:107)
=1 Z ( c ) or arg min (cid:107) c (cid:107) =
1 Z ( c ) , and it is impossible to enumerate all c ’s, we estimate the measure by,
[Equation]
where C is the set of eigenvectors of V (cid:62) V .
The value of I ( { v ( w ) } ) for the original vectors and processed ones are reported in Table 2, where we can observe that the degree of isotropy vastly increases in terms of this measure.
[Figure]
A formal way to verify the isotropy property is to directly check if the “self-normalization" property (i.e., Z ( c ) is a constant, independent of c (Andreas & Klein, 2015)) holds more strongly.
Such a validation is seen diagrammatically in Figure 3 where we randomly sampled 1,000 c ’s as (Arora et al., 2016).
[Figure]
E XPERIMENTS
Given the popularity and widespread use of WORD2VEC (Mikolov et al., 2013) and GLOVE (Pennington et al., 2014), we use their publicly available pre-trained reprepsentations in the following experiments.
We choose D = 3 for WORD2VEC and D = 2 for GLOVE.
The key underlying principle behind word representations is that similar words should have similar representations.
Following the tradition of evaluating word representations (Schnabel et al., 2015; Baroni et al., 2014), we perform three canonical lexical-level tasks: (a) word similarity; (b) concept categorization; (c) word analogy; and one sentence-level task: (d) semantic textual similarity.
The processed representations consistently improve performance on all three of them, and especially strongly on the ﬁrst two.
[Table]
Word Similarity
The word similarity task is as follows: given a pair of words, the algorithm assigns a “similarity" score – if the pair of words are highly related then the score should also be high and vice versa.
The algorithm is evaluated in terms of Spearman’s rank correlation compared to (a gold set of) human judgements.
For this experiment, we use seven standard datasets: the ﬁrst published RG65 dataset (Rubenstein & Goodenough, 1965); the widely used
WordSim-353 (WS) dataset (Finkelstein et al., 2001) which contains 353 pairs of commonly used verbs and nouns; the rare-words (RW) dataset (Luong et al., 2013) composed of rarely used words; the MEN dataset (Bruni et al., 2014) where thepairs of words are rated by crowdsourced participants; the MTurk dataset (Radinsky et al., 2011) where the 287 pairs of words are rated in terms of relatedness; the SimLex-999 (SimLex) dataset (Hill et al., 2016) where the score measures “genuine" similarity; and lastly the SimVerb-3500 (SimVerb) dataset (Gerz et al., 2016), a newly released large dataset focusing on similarity of verbs.
In our experiment, the algorithm scores the similarity between two words by the cosine similarity between the two corresponding word vectors ( CosSim( v 1 , v 2 )
= v (cid:62) 1 v 2 / (cid:107) v 1 (cid:107)(cid:107) v 2 (cid:107) ).
The detailed performance on the seven datasets is reported in Table 3, where we see a consistent and signiﬁcant performance improvement due to postprocessing, across all seven datasets.
These statistics
(average improvement of 2.3 %) suggest that by removing the common parts, the remaining word representations are able to capture stronger semantic relatedness/similarity between words.
[Table]
Concept Categorization
This task is an indirect evaluation of the similarity principle: given a set of concepts, the algorithm needs to group them into different categories (for example, “bear” and “cat” are both animals and “city” and “country” are both related to districts).
The clustering performance is then evaluated in terms of purity (Manning et al., 2008) – the fraction of the total number of the objects that were classiﬁed correctly.
We conduct this task on three different datasets: the Almuhareb-Poesio (ap) dataset (Almuhareb, 2006) contains 402 concepts which fall into 21 categories; the ESSLLI 2008 Distributional Semantic
Workshop shared-task dataset
(Baroni et al., 2008) that contains 44 concepts in 6 categories; and the
Battig test set (Baroni & Lenci, 2010) that contains 83 words in 10 categories.
Here we follow the setting and the proposed algorithm in (Baroni et al., 2014; Schnabel et al., 2015) – we cluster words (via their representations) using the classical k -Means algorithm (with ﬁxed k ).
Again, the processed vectors perform consistently better on all three datasets (with average improvement of 2.5%); the full details are in Table 4.
[Table]
Word Analogy
The analogy task tests to what extent the word representations can encode latent linguistic relations between a pair of words.
Given three words w 1 , w 2 , and w 3 , the analogy task requires the algorithm to ﬁnd the word w 4 such that w 4 is to w 3 as w 2 is to w 1 .
We use the analogy dataset introduced in (Mikolov et al., 2013).
The dataset can be divided into two parts: (a) the semantic part containing around 9k questions, focusing on the latent semantic relation between pairs of words (for example, what is to Chicago as Texas is to Houston); and (b) the syntatic one containing roughly 10.5k questions, focusing on the latent syntatic relation between pairs of words (for example, what is to “amazing” as “apprently” is to “apparent”).
In our setting, we use the original algorithm introduced in (Mikolov et al., 2013) to solve this problem, i.e., w 4 is the word that maximize the cosine similarity between v ( w 4 ) and v ( w 2 ) − v ( w 1 ) + v ( w 3 ) .
The average performance on the analogy task is provided in Table 5 (with a detailed performance provided in Table 19 in Appendix D).
It can be noticed that while postprocessing continues to improve the performance, the improvement is not as pronounced as earlier.
We hypothesize that this is because the mean and some dominant components get canceled during the subtraction of v ( w 2 ) from v ( w 1 ) , and therefore the effect of postprocessing is less relevant.
WORD2VEC
GLOVE orig.
proc.
orig.
proc.
SICK 70.
[Table]
Semantic Textual Similarity Extrinsic evaluations measure the contribution of a word representation to speciﬁc downstream tasks; below, we study the effect of postprocessing on a standard sentence modeling task – semantic textual similarity (STS) which aims at testing the degree to which the algorithm can capture the semantic equivalence between two sentences.
For each pair of sentences, the algorithm needs to measure how similar the two sentences are.
The degree to which the measure matches with human judgment (in terms of Pearson correlation) is an index of the algorithm’s performance.
We test the word representations on 20 textual similarity datasets from
2012-2015 SemEval STS tasks ( al.,
and the SemEval Semantic Related task (SICK) (Marelli et al., 2014). Representing sentences by the average of their constituent word representations is surprisingly effective in encoding the semantic information of sentences (Wieting et al., 2015; Adi et al., 2016) and close to the state-of-the-art in these datasets.
We follow this rubric and represent a sentence s based on its averaged word representation, i.e., v ( s ) =
| s | (cid:80) w ∈ s v ( w ) , and then compute the similarity between two sentences via the cosine similarity between the two representations.
The average performance of the original and processed representations is itemized in Table 6 (with a detailed performance in Table 20 in Appendix E) – we see a consistent and signiﬁcant improvement in performance because of postprocessing (on average 4 % improvement).
P OSTPROCESSING AND S UPERVISED C LASSIFICATION
Supervised downstream NLP applications have greatly improved their performances in recent years by combining the discriminative learning powers of neural networks in conjunction with the word representations.
We evaluate the performance of a variety of neural network architectures on a standard and important NLP application: text classiﬁcation , with sentiment analysis being a particularly important and popular example.
The task is deﬁned as follows: given a sentence, the algorithm needs to decide which category it falls into.
The categories can be either binary (e.g., positive/negative) or can be more ﬁne-grained (e.g. very positive, positive, neutral, negative, and very negative).
We evaluate the word representations (with and without postprocessing) using four different neural network architectures (CNN, vanilla-RNN, GRU-RNN and LSTM-RNN) on ﬁve benchmarks: (a) the movie review (MR) dataset (Pang & Lee, 2005); (b) the subjectivity (SUBJ) dataset (Pang & Lee, 2004); (c) the TREC question dataset (Li & Roth, 2002); (d) the IMDb dataset (Maas et al., 2011); (e) the stanford sentiment treebank (SST) dataset (Socher et al., 2013a).
A detailed description of these standard datasets, their training/test parameters and the cross validation methods adopted is in Appendix F. Speciﬁcally, we allow the parameter D (i.e., the number of nulled components) to vary between 0 and 4, and the best performance of the four neural network architectures with the now-standard CNN-based text classiﬁcation algorithm (Kim, 2014) (implemented using tensorﬂow 4 ) is itemized in Table 7.
The key observation is that the performance of postprocessing is better in a majority (34 out of 40) of the instances by 2.32% on average, and in the rest the instances the two performances (with and without postprocessing) are comparable.
[Table]
A further validation of the postprocessing operation in a variety of downstream applications (eg: named entity recognition, syntactic parsers, machine translation) and classiﬁcation methods (eg: random forests, neural network architectures) is of active research interest.
Of particular interest is the impact of the postprocessing on the rate of convergence and generalization capabilities of the classiﬁers.
Such a systematic study would entail a concerted and large-scale effort by the research community and is left to future research.
Discussion All neural network architectures, ranging from feedforward to recurrent (either vanilla or GRU or LSTM), implement at least linear processing of hidden/input state vectors at each of their nodes; thus the postprocessing operation suggested in this paper can in principle be automatically “learnt” by the neural network, if such internal learning is in-line with the end-to-end training examples.
Yet, in practice this is complicated due to limitations of optimization procedures (SGD) and sample noise.
We conduct a preliminary experiment in Appendix B and show that subtracting the mean (i e., the ﬁrst step of postprocessing) is “effectively learnt" by neural networks within their nodes.
C ONCLUSION
We present a simple postprocessing operation that renders word representations even stronger, by eliminating the top principal components of all words.
Such an simple operation could be used for word embeddings in downstream tasks or as intializations for training task-speciﬁc embeddings.
Due to their popularity, we have used the published representations of WORD2VEC and GLOVE in English in the main text of this paper; postprocessing continues to be successful for other representations and in multilingual settings – the detailed empirical results are tabulated in Appendix C.
Appendix: All-but-the-Top: Simple and Effective postprocessing for Word Representations A A NGULAR A SYMMETRY OF R EPRESENTATIONS
A modern understanding of word representations involves either PMI-based (including word2vec (Mikolov et al., 2010; Levy & Goldberg, 2014) and GloVe (Pennington et al., 2014)) or CCA-based spectral factorization approaches.
While CCA-based spectral factorization methods have long been understood from a probabilistic (i.e., generative model) view point (Browne, 1979; Hotelling, 1936) and recently in the NLP context (Stratos et al., 2015), a corresponding effort for the PMI-based methods has only recently been conducted in an inspired work (Arora et al., 2016).
(Arora et al., 2016) propose a generative model (named RAND-WALK) of sentences, where every word is parameterized by a d -dimensional vector.
With a key postulate that the word vectors are angularly uniform (“isotropic"), the family of PMI-based word representations can be explained under the RAND-WALK model in terms of the maximum likelihood rule.
Our observation that word vectors learnt through PMI-based approaches are not of zero-mean and are not isotropic (c.f. Section 2) contradicts with this postulate.
The isotropy conditions are relaxed in Section 2.2 of (Arora et al., 2016), but the match with the spectral properties observed in Figure 1 is not immediate.
In this section, we resolve this by explicitly relaxing the constraints on the word vectors to directly ﬁt the observed spectral properties.
The relaxed conditions are: the word vectors should be isotropic around a point (whose distance to the origin is a small fraction of the average norm of word vectors) lying on a low dimensional subspace.
Our main result is to show that even with this enlarged parameter-space, the maximum likelihood rule continues to be close to the PMI-based spectral factorization methods.
Formally, the model, the original constraints of (Arora et al., 2016) and the enlarged constraints on the word vectors are listed below:
• A generative model of sentences : the word at time t ,
denoted by w t is generated via a log-linear model with a latent discourse variable
c t (Arora et al., 2016), i.e.,
[Equation]
where v ( w ) ∈
R
d is the vector representation for a word w in the vocabulary V , c t is the latent variable which forms a “slowly moving" random walk, and the partition function:
Z ( c ) =
[Equation]
w
[Equation]
• Constraints on the word vectors : (Arora et al., 2016) suppose that there is a Bayesian priori on the word vectors:
The ensemble of word vectors consists of i.i.d.
draws generated by v = s · ˆ v , where ˆ v is from the spherical Gaussian distribution, and s is a scalar random variable.
A deterministic version of this prior is discussed in Section 2.2 of (Arora et al., but part of these (relaxed) conditions on the word vectors are speciﬁcally meant for Theorem 4.
and not the main theorem (Theorem 2.2).
The geometry of the word representations is only evaluated via the ratio of the quadratic mean of the singular values to the smallest one being small enough.
This meets the relaxed conditions
but not sufﬁcient to validate the proof approach of the main result (Theorem 2. ; what would be needed is that the ratio of the largest singular value to the smallest one be small.
• Revised conditions : We revise the Bayesian prior postulate (and in a deterministic fashion) formally as follows: there is a mean vector µ , D orthonormal vectors u 1 , . . .
, u D (that are orthogonal and of unit norm), such that every word vector v ( w ) can be represented by,
[Equation]
where µ is bounded, α i is bounded by A , D is bounded by DA
= o ( d ) , ˜ v ( w ) are statistically isotropic.
By statistical isotropy, we mean: for high-dimensional rectangles R ,
1 |V| (cid:80)
[Figure]
∈V 1 (˜ v ( w ) ∈ R ) → (cid:82) R f
(˜ v ) d ˜ v , as |V| → ∞ , where f is an angle-independent pdf, i.e., f (˜ v ) is a function of (cid:107) ˜ v (cid:107) .
The revised postulate differs from the original one in two ways: (a) it imposes a formal deterministic constraint on the word vectors; (b) the revised postulate allows the word vectors to be angularly asymmetric: as long as the energy in the direction of u 1 ,... , u D is bounded, there is no constraint on the coefﬁcients.
Indeed, note that there is no constraint on ˜ v ( w ) to be orthogonal to u 1 , . . .
u D .
Empirical Validation We can verify that the enlarged conditions are met by the existing word representations.
Speciﬁcally, the natural choice for µ is the mean of the word representations and u 1 . . .
u D are the singular vectors associated with the top D singular values of the matrix of word vectors.
We pick D = 20 for WORD2VEC and D = 10 for GLOVE, and the corresponding value of
DA 2 for WORD2VEC and GLOVE vectors are both roughly 40, respectively; both values are small compared to d = 300 .
[Figure]
This leaves us to check the statistical isotropy of the “remaining" vectors ˜ v ( w ) for words w in the vocabulary.
We do this by plotting the remaining spectrum (i.e. the ( D + 1) -th, ..., 300th singular values) for the published WORD2VEC and GLOVE vectors in Figure 4.
As a comparison, the empirical spectrum of a random Gaussian matrix is also plotted in Figure 4.
We see that both spectra are ﬂat (since the vocabulary size is much larger than the dimension d = 300 ).
Thus the postprocessing operation can also be viewed as a way of making the vectors “more isotropic”.
Mathematical Contribution Under the revised postulate, we show that the main theorem in (Arora et al., 2016) (c.f. Theorem 2.2) still holds.
Formally:
Theorem A.1 Suppose the word vectors satisfy the constraints.
Then
[Equation]
where p ( w ) is the unigram distribution induced from the model (2) , and p ( w 1 , w 2 ) is the probability that two words w 1 and w 2 occur with each other within distance q .
The proof is in Appendix G. Theorem A.1 suggests that the RAND-WALK generative model and its properties proposed by (Arora et al., 2016) can be generalized to a broader setting (with a relaxed restriction on the geometry of word representations) – relevantly, this relaxation on the geometry of word representations is empirically satisﬁed by the vectors learnt as part of the maximum likelihood rule.
B N EURAL N ETWORKS L EARN TO P OSTPROCESS
Every neural network family posseses the ability to conduct linear processing inside their nodes; this includes feedforward and recurrent and convolutional neural network models.
Thus, in principle, the postprocessing operation can be “learnt and implemented" within the parameters of the neural network.
On the other hand, due to the large number of parameters within the neural network, it is unclear how to verify such a process, even if it were learnt (only one of the layers might be implementing the postprocessing operation or via a combination of multiple effects).
To address this issue, we have adopted a comparative approach in the rest of this section.
The comparative approach involves adding an extra layer interposed in between the inputs (which are word vectors) and the rest of the neural network.
This extra layer involves only linear processing.
Next we compare the results of the ﬁnal parameters of the extra layer (trained jointly with the rest of tne neural network parameters, using the end-to-end training examples) with and without preprocessing of the word vectors.
Such a comparative approach allows us to separate the effect of the postprocessing operation on the word vectors from the complicated “semantics” of the neural network parameters.
[Figure] [Figure]
Experiment We construct a modiﬁed neural network by explicitly adding a “postprocessing unit" as the ﬁrst layer of the RNN architecture (as in Figure 5, where the appended layer is used to test the ﬁrst step (i.e., remove the mean vector)) of the postprocessing algorithm.
In the modiﬁed neural network, the input word vectors are now v ( w ) − b instead of v ( w ) .
Here b is a bias vector trained jointly with the rest of the neural network parameters.
Note that this is only a relabeling of the parameters from the perspective of the RNN architecture: the nonlinear activation function of the node is now operated on A ( v ( w ) − b ) + b (cid:48) =
Av ( w ) + ( b (cid:48) − Ab ) instead of the previous Av ( w ) + b (cid:48) .
Let b proc .
and b orig .
be the inferred biases when using the processed and original word representations, respectively.
We itemize the cosine similarity between b proc .
+ µ and b orig .
in Table 6 for the 5 different datasets and 3 different neural network architectures.
In each case, the cosine similarity is remarkably large (on average 0.66, in 300 dimensions) – in other words, trained neural networks implicitly postprocess the word vectors nearly exactly as we proposed.
This agenda is successfully implemented in the context of verifying the removal of the mean vector.
The second step of our postprocessing algorithm (i.e., nulling away from the top principal components) is equivalent to applying a projection matrix P
=
I − (cid:80) Di
=1 u i u (cid:62) i on the word vectors, where u i is the i -th principal component and D is the number of the removed directions.
A comparative analysis effort for the second step (nulling the dominant PCA directions) is the following.
Instead of applying a bias term b , we multiply by a matrix Q to simulate the projection operation.
The input word vectors are now Q orig .
v ( w ) instead of v ( w ) for the original word vectors, and Q proc .
Pv ( w ) instead of Pv ( w ) for the processed vectors.
Testing the similarity between Q orig .
P and Q proc .
, allows us to verify if the neural network learns to conduct the projection operation as proposed.
In our experiment, we found that such a result cannot be inferred.
One possibility is that there are too many parameters in both Q proc .
and Q orig . , which adds randomness to the experiment.
Alternatively, the neural network weights may not be able to learn the second step of the postprocessing operation (indeed, in our experiments postprocessing signiﬁcantly boosted end-performance of neural network architectures).
A more careful experimental setup to test whether the second step of the postprocessing operation is learnt is left as a future research direction.
C E XPERIMENTS ON V ARIOUS R EPRESENTATIONS
In the main text
[Table]
we have reported empirical results for two published word representations:
WORD2VEC and GLOVE, each in 300 dimensions.
In this section, we report the results of the same experiments in a variety of other settings to show the generalization capability of the postprocessing operation: representations trained via WORD2VEC and GLOVE algorithms in dimensions other than 300, other representations algorithms (speciﬁcally TSCCA and RAND-WALK) and in multiple languages
[Table]
C. S TATISTICS OF M ULTILINGUAL W ORD R EPRESENTATIONS
We use the publicly available TSCCA representations (Dhillon et al., 2012) on German, French,
Spanish, Italian, Dutch and Chinese.
The detailed statistics can be found in Table 8 and the decay of their singular values are plotted in Figure 7.
[Table]
[Figure]
C. M ULTILINGUAL G ENERALIZATION
In this section, we perform the word similarity task with the original and the processed TSCCA word representations in German and Spanish on three German similarity datasets (GUR65 – a German version of the RG65 dataset, GUR350, and ZG222 in terms of relatedness) (Zesch & Gurevych, 2006) and the Spanish version of RG65 dataset (Camacho-Collados et al., 2015).
The choice of D is 2 for both German and Spanish.
The experiment setup and the similarity scoring algorithm are the same as those in Section 3.
The detailed experiment results are provided in Table 9, from which we observe that the processed representations are consistently better than the original ones.
This provides evidence to the generalization capabilities of the postprocessing operation to multiple languages (similarity datasets in Spanish and
German were the only ones we could locate).
C. G ENERALIZATION TO D IFFERENT R EPRESENTATION A LGORITHMS
Given the popularity and widespread use of WORD2VEC (Mikolov et al., 2013) and GLOVE (Pennington et al., 2014), the main text has solely focused on their published publicly avalable
[Table]
300-dimension representations.
In this section, we show that the proposed postprocessing algorithm generalizes to other representation methods.
Speciﬁcally, we demonstrate this on RAND-WALK (obtained via personal communication) and TSCCA (publicly available) on all the experiments of
Section 3.
The choice of D is 2 for both RAND-WALK and TSCCA.
In summary, the performance improvements on the similarity task, the concept categorization task, the analogy task, and the semantic textual similarity dataset are on average 2.23%, 2.39%, 0.11% and 0.61%, respectively.
The detailed statistics are provided in Table 10, Table 11, Table 12 and Table 13, respectively.
These results are a testament to the generalization capabilities of the postprocessing algorithm to other representation algorithms.
[Table] [Table]
[Table]
C. R OLE OF D IMENSIONS
The main text has focused on the dimension choice of d = 300 , due to its popularity.
In this section we explore the role of the dimension in terms of both choice of D and the performance of the postprocessing operation – we do this by using skip-gram model on the 2010 snapshot of Wikipedia corpus (Al-Rfou et al., 2013) to train word representations.
We ﬁrst observe that the two phenomena of Section 2 continue to hold:
• From Table 14 we observe that the ratio between the norm of µ and the norm average of all v ( w ) spans from 1/3 to 1/4;
[Table] [Table]
• From Figure 8 we observe that the decay of the variance ratios σ i is near exponential for small values of i and remains roughly constant over the later ones.
[Figure]
A rule of thumb choice of D is around d/ 100 .
We validate this claim empirically by performing the tasks in Section 3 on word representations of higher dimensions, ranging from 300 to 1000, where we set the parameter D = d/ 100 .
In summary, the performance improvement on the four itemized tasks of Section 3 are 2.27%, 3.37%, 0.01 and 1.92% respectively; the detailed results can be found in Table 15, Table 16, Table 17, and Table 18.
Again, note that the improvement for analogy tasks is marginal.
These experimental results justify the rule-of-thumb setting of D = d/ 100 , although we emphasize that the improvements can be further accentuated by tuning the choice of D based on the speciﬁc setting.
D E XPERIMENTS ON W ORD A NALOGY T ASK
The detailed performance on the analogy task is provided in Table 19.
E E XPERIMENTS ON S EMANTIC T EXTUAL S IMILARITY T ASK
The detailed performance on the semantic textual similarity is provided in Table 20.
Dim    orig.
proc.
orig.
proc.
orig.
proc.
orig.
proc.
RG 73.
WS 70.
RW 46.
MEN 75.
Mturk 75.
SimLex 34.
SimVerb 22.
Dim    orig.
proc.
orig.
proc.
orig.
proc.
orig.
proc.
RG 77.
WS 70.
RW 45.
MEN 75.
Mturk 67.
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table]
F S TATISTICS OF T EXT C LASSIFICATION D ATASETS
We evaluate the word representations (with and without postprocessing) using four different neural network architectures (CNN, vanilla-RNN, GRU-RNN and LSTM-RNN) on ﬁve benchmarks:
• the movie review (MR) dataset (Pang & Lee, 2005) where each review is composed by only one sentence; • the subjectivity (SUBJ) dataset (Pang & Lee, 2004) where the algorithm needs to decide whether a sentence is subjective or objective;
[Table]
SICK 68.
all 58.
[Table]
Dim
[Table] [Table]
[Table] [Table]
[Table]
SICK 68.
all 59.
x100) on the semantic textual similarity tasks.
[Table]
• the TREC question dataset (Li & Roth, 2002) where all the questions in this dataset has to be partitioned into six categories; • the IMDb dataset (Maas et al., 2011) – each review consists of several sentences; • the Stanford sentiment treebank (SST) dataset
(Socher et al., 2013a), where we only use the full sentences as the training data.
In TREC, SST and IMDb, the datasets have already been split into train/test sets.
Otherwise we use 10-fold cross validation in the remaining datasets (i.e., MR and SUBJ).
Detailed statistics of various features of each of the datasets are provided in Table 21.
G P ROOF OF T HEOREM A.
Given the similarity between the setup in Theorem 2.2 in (Arora et al., 2016) and Theorem A.1, many parts of the original proof can be reused except one key aspect – the concentration of Z ( c ) .
We summarize this part in the following lemma:
WORD2VEC
GLOVE orig.
proc.
orig.
proc.
2012.MSRpar 42.
2012.MSRvid 72.
2012.OnWN 69.
2012.SMTeuroparl 53.
2012.SMTnews 49.
2013.FNWN 40.
2013.OnWN 67.
2013.headlines 61.
2014.OnWN 74.
2014.deft-forum 32.
2014.deft-news 66.
2014.headlines 58.
2014.images 73.
2014.tweet-news 71.
2015.answers-forum 46.
2015.answers-students 68.
2015.belief 59.
2015.headlines 61.
2015.images 78.
SICK 70.
[Table]
Lemma G.1 Let c be a random variable uniformly distributed over the unit sphere, we prove that with high probability, Z ( c ) / |V| converges to a constant Z :
[Equation]
where (cid:15) z
= Ω(( D + 1) / |V| ) and δ = Ω(( DA 2 + (cid:107) µ (cid:107) 2 ) /d ) .
Our proof differs from the one in (Arora et al., 2016) in two ways: (a) we treat v ( w ) as deterministic parameters instead of random variables and prove the Lemma by showing a certain concentration of measure; (b) the asymmetric parts µ and u 1 ,..., u D , (which did not exist in the original proof), need to be carefully addressed to complete the proof.
G. P ROOF OF L EMMA G.
Given the constraints on the word vectors (3), the partition function Z ( c ) can be rewritten as,
[Equation]
The equation above suggests that we can divide the proof into ﬁve parts.
Step 1: for every unit vector c , one has,
[Equation]
Proof Let M, N be a positive integer, and let A M ⊂ R d such that,
[Equation]
Since A M,N can be represented by a union of countable disjoint rectangles [Equation] we know that for every
M, N ∈ N + ,
[Equation]
Further, since A M,N are disjoint for different M ’s and R d = ∪ ∞ M =1 A M,N , one has,
[Equation]
The above statement holds for every N .
Let N → ∞ , by deﬁnition of integration, one has,
[Equation]
which yields,
[Equation]
[Equation]
Similarly, one has,
[Equation]
v (cid:1)(cid:1) , as |V| → ∞ .
Putting (6) and (7) proves (5).
Step 2: the expected value,
[Equation]
is a constant independent of c
[Equation]
Proof Let Q ∈ R
d
×
d be a orthonormal matrix such that Q (cid:62) c
= c where c 0 = (1 , 0 , ..., 0) (cid:62) and det( Q ) = 1 , then we have f (˜ v ) =
f ( Q ˜ v ) , and,
[Equation]
which proves (8).
Step 3: for any vector µ , one has the following concentration property,
[Equation]
Proof Let c 1 ,..., c d be i.i.d.
N (0 , 1) , and let C = (cid:80) di
=1 c 2 i , then c
= ( c 1 , ..., c d ) / √ C is uniform over unit sphere.
Since c is uniform, then without loss of generality we can consider µ = ( (cid:107) µ (cid:107) , 0 , ..., 0) .
Thus it sufﬁces to bound exp (cid:16) (cid:107) µ (cid:107) c 1 / √ C (cid:17) .
We divide the proof into the following steps:
• C follows chi-square distribution with the degree of freedom of d , thus C can be bounded by (Laurent & Massart, 2000),
[Equation]
• Therefore for any x > 0 , one has,
[Equation]
Let x = 1 / 4 d
one has,
[Equation]
• Since c 1 is a Gaussian random variable with variance 1 , by Chebyshev’s inequality, one has,
[Equation]
and therefore thus,
[Equation]
• Therefore we can bound exp (cid:16) (cid:107) µ (cid:107) c 1 / √ C (cid:17) by,
[Equation]
Combining the two inequalities above, one has (9) proved.
Step 4: We are now ready to prove convergence of Z ( c ) .
With (9), let C ⊂ R d such that,
[Equation]
Then we can bound the probability on C by,
[Equation]
Next, we need to show that for every w , the corresponding C ( w ) , i
[Equation]
e.,
[Equation]
We observe that α i ( w ) is bounded by A , therefore for any c that,
[Equation]
and thus,
[Equation]
which yields,
[Equation]
Therefore we prove C ( w ) ⊃ C .
Assembling everything together, one has,
[Equation]
For every c ∈ C , one has,
[Equation]
Let Z = |V| Z 0 , one can conclude that,
[Equation]
, where (cid:15) z
= Ω(( D + 1) / |V| ) and δ = Ω( DA 2 /d ) .
G. P ROOF OF T HEOREM A.
Having Lemma G.1 ready, we can follow the same proof as in (Arora et al., 2016) that both p ( w ) and p ( w, w (cid:48) ) are correlated with (cid:107) v ( w ) (cid:107) , formally
[Equation]
Therefore, the inference presented in (Arora et al., 2016)
(i.e., (4)) is obvious by assembling (12) and (13) together:
[Equation]