Semantic Field of Words Represented as Non-Linear Potential Functions
Abstract
State-of-the-art word embeddings presume a linear vector space, but this approachdoes not easily incorporate the nonlinearity that is necessary to represent polysemy.
We thus propose a novel semantic FIeld REepresentation, called FIRE, which is a
D -dimensional ﬁeld in which every word is represented as a set of its locations anda nonlinear function covering the ﬁeld.
The strength of a word’s relation to anotherword at a certain location is measured as the function value at that location.
With
FIRE, compositionality is represented via functional additivity, whereas polysemyis represented via the set of points and the function’s multimodality.
By implementing FIRE for English and comparing it with previous representation methods viaword and sentence similarity tasks, we show that FIRE produces comparable oreven better results.
In an evaluation of polysemy to predict the number of wordsenses, FIRE greatly outperformed BERT and Word2vec, providing evidence ofhow FIRE represents polysemy.
Introduction
Word representations have become an important basic technique in NLP.
The most important semantic requirement for a word representation,above all, is that similar words are mapped to similarembeddings. Furthermore, the next two most important properties are compositionality [25], i.e.,the representation of two composed words matches the composition of the two representations andpolysemy , the representation of semantic ambiguity.
Intuitively, compositionality is a linear quality,whereas polysemy is nonlinear and thus requires a representation via mathematical multimodality,which is difﬁcult to represent within a linear space.
State-of-the-art word representations are based on a linear vector space.
The linearity of vectorsnaturally accommodates compositionality but the question of how to incorporate polysemy has beenan issue.
Indeed, recent random-variable representations [27, 21, 23, 16, 8] represent polysemy but atthe cost of losing compositionality.
The most important previous work that seems to incorporate polysemy in addition to compositionalitycould be BERT [6]; however, it is a contextual embedding that represents one vector only under agiven context.
A polysemous word has multiple meanings even without context.
Our idea was thus todevise an entirely new representation that is nonlinear but naturally represents linearity as well.
Hence, FIeld REpresentation, which we call FIRE, is a non-contextual representation of words.
It isrealized in a D -dimensional space in which every point has a quantity, called a semantic potential ,that is generated by interactions among words.
Every word is represented by a pair consisting of a setof the word’s locations and a nonlinear potential function representing the word’s context, where theﬁeld outputs the semantic potential value at a location.
Two potential ﬁelds can be linearly added,which provides the basis of compositionality.
Polysemy is represented through the set of a word’s locations and the multimodality of the ﬁeld f .
Through this representation, the similarity of twowords is obtained by adding the potential ﬁeld values at the word locations.
Our experimental results conﬁrmed that FIRE showed similar or better results in comparison withprevious Word2XX representations under settings with the same number of parameters.
Furthermore,by representing multimodality over the entire ﬁeld, FIRE has a much wider expressiveness forpolysemy.
In an evaluation to predict the number of word senses, FIRE greatly outperformed BERT.
Related Works
Vectoral word representations.
The most widely used word representations are in vectoral forms,such as Word2Vec [13].
[10] showed that Word2Vec is essentially a decomposition of the point-wisemutual information matrix.
Other vectoral representation methods extract different information fromcorpora.
Important works include GloVe [17], fastText [5], ELMo [18], and BERT [6].
It has been attempted to represent polysemy via multiple vectors per word [24], but such a “mixture”of vectors lost the quality of compositionality.
Random-variable word representations.
[27] proposed Word2Gauss , which represents each wordby a Gaussian distribution.
They exploited the covariance matrix to model the semantic ambiguityof words.
Asymmetric similarity (via the Kullback-Leibler divergence) was further introduced torepresent hypernyms and hyponyms.
[21] proposed to exploit existing word vectors by representing aword with a nonparametric probability distribution estimated using its closest word vectors.
Several recent works studied the use of the Wasserstein distance to measure the similarity betweenwords, which are represented by Gaussian distributions [23] or more general elliptical distributions[16].
For elliptical distributions, the Wasserstein distance has a closed form and can be computedefﬁciently.
[8] proposed to use the entropy-regularized Wasserstein distance as the similarity betweenwords that are represented by multinomial distributions and deﬁned on the same set of (64) discretesupports in a low-dimensional space (e.g., R 2 ).
Because the supports form a point cloud, the methodis called Word2Cloud .
The multinomial distribution in Word2Cloud can be viewed as a mixture of
Dirac delta functions, a notion that we use in this work.
Word2GM [2] also uses a mixture of standarddistributions, where the random variable follows a mixture of Gaussians.
All these random-variable word representations incorporate polysemy but not compositionality.
Furthermore, many of these methods remain word representations (e.g., capturing polysemy andhypernymy) only and have not been extended to sentence representations.
Sentence representations.
A typical sentence representation is a vector obtained as a weightedadditive composition of vectoral word representations.
Various weighting heuristics have been studied,such as the term frequency–inverse document frequency (TF-IDF) [12].
[1] proposed the smoothinverse frequency (SIF), a simple weighting scheme that works exceptionally well.
In this paper, too,we evaluate the capability of the sentence ﬁeld representation produced by applying the SIF to FIREword representation.
State-of-the-art vectoral sentence representation methods adopt an adaptive weighting scheme byusing a self-attention mechanism [26]; one implementation includes BERT [6].
Such methods canalso be applied to our word ﬁeld representation, which remains a future work.
To the best of our knowledge, [21] proposed the only existing random-variable sentence representation.
Representations of the words in a sentence are composed into a Wasserstein barycenter generating anew random variable.
Whereas that computation is not additive, FIRE is additive.
Semantic Field: FIRE3.1 Word Representation
FIRE is a semantic ﬁeld generated by words.
In physics, a ﬁeld is a space in which every point hassome physical quantity, of which a common type is the potential .
FIRE uses the semantic potentialgenerated by words.
While particles in a physical ﬁeld are often identical, the particularity of FIRE is that every word produces a different semantic potential and is thus allocated within the ﬁeld inassociation with an individual potential function.
Let W be a set of words and (cid:83) be a D -dimensional real space.
Our semantic ﬁeld is deﬁned on(cid:83) .
A word w ∈ W is represented as a 2-tuple [ µ, f ( s )] , where µ represents the word’s locations.
In a previous linear space model [13], a word was usually located at a single point but could bepolysemous, i.e., with multiple locations as in [24].
Let K denote the number of such locations.f ( s ) : (cid:83) → R is a continuous real-valued function covering (cid:83) .
f ( s ) represents the context generatedby the surrounding words of w via a neural network, as will be explained in Section 4.1.
Figure 1 (left) shows an example portion of (cid:83) computed for D = 2 with (cid:83) =
[ − 4 , 4] 2 and K = 1 ,for the term bank .
There are two clusters related to bank , one for a ﬁnancial bank and the other for ariver bank.
The circles show the locations µ for 1000 words (among the most frequent 50,000 words)that were frequent and similar to bank , with similarity as deﬁned in the following section.
3.2 Similarity of Two Words
In our approach, two words w i =
[ µ i , f i ( s )] and w j = [ µ j , f j ( s )] in a ﬁeld interact.
Given thelocation s j of w j , the interaction with w i is measured by f i ( s j ) .
As mentioned above, polysemymeans that w j has multiple locations, s ( k ) j , k = 1 , . .
, K ; thus, the f i ( s ( k ) j ) must be added, or moresimply, integrated.
The location here means that an entity is scattered in a ﬁeld: it can be multiple andnoncontinuous, and it may have different mass at different locations.
This notion is mathematicallyexpressed as a measure , and an integral with a measure is formulated via the Lebesgue integral.
The simplest measure is the Dirac delta function δ ( s ) , which indicates that a location is point-wisewithin a ﬁeld.
For K = 1 and µ j ≡ δ ( s j ) , we have (cid:82) f i d µ j = f i ( s j ) .
By using the Dirac deltafunction, we can compute the integral exactly without any approximation.
As we also have the symmetric quantity of w i ’s location in the ﬁeld of f j ( s ) , the similarity of the twowords is naturally expressed as follows:
[Equation]
Using a measure for a word’s location facilitates a simple representation.
For K ≥ 1 , µ is a Diracmixture with weights m ( k ) as follows:
[Equation]
By using this µ for each word, Formula (1) reduces to the following (because (cid:82)
f d δ ( s ) =
f ( s ) ):
[Equation]
which is simply a weighted sum of the two potential functions
f i ( s ) and f j ( s ) at all the locations ofw j and w i , respectively.
Among various measure possibilities, this paper uses the Dirac delta function, because this formulationenables an extension to deﬁne sentence representations in Section 3.3.
The similarity in this workbetween words and sentences via a measure is different from the standard notion of distance in a
Euclidean space and is related to metric learning methods [9].
FIRE incorporates polysemy via (1) the multimodality of f ( s ) and (2) the number K of wordlocations.
Note that the multimodality of f ( s ) is implemented via a neural network.
3.3 Composition of Words and Similarity of Two Sentences
A set of n words, Γ = [ w 1 , . . . , w n ] , including sentences, can also be represented in [ µ, f ( s )]notation when deﬁned as a weighted average of word representations { [ µ 1 , f 1 ( s )] , . . .
, [ µ n , f n ( s )] } ,as follows:
[Equation] [Equation]
where γ i are the weights.
In this work, we used the SIF weights [1] as the γ i .
Note that this f ( s ) ismultimodal and thus polysemous, because all the f i ( s ) are multimodal.
Figure 1 (right) shows such a superposition of the words river (blue) and mortgage (red)
The similarity of two sets of words or sentences, Γ = [ µ, f ] and Γ (cid:48) =
[ µ (cid:48) , f (cid:48) ] , is measured byapplying the similarity function (1).
Let n and n (cid:48) denote the respective numbers of words in the twosentences.
The words in Γ and Γ (cid:48) are assigned weights γ =
[ γ 1 , . . . , γ n ] T and γ (cid:48) =
[ γ (cid:48) 1 , . .
, γ (cid:48) n (cid:48) ] T ,respectively.
Then, as shown in Appendix B, Formula (1) is simply calculated as follows:
[Equation]
where Σ is the word similarity matrix, deﬁned as Σ ij = sim( w i , w j ) with w i and w j being wordsin sentences Γ and Γ (cid:48) , respectively.
In other words, the similarity between two sentences can beobtained from the pairwise word similarities between the sentences.
Implementation of FIRE
4.1 Potential Functions As Planar Transformations
The simplest way to implement the potential function f ( s ) would be to construct a neural network
NN with a D -dimensional input and 1 -dimensional output, i.e., f ( s ) =
NN ( s ; θ ) , where θ representsthe parameters of the neural network.
A usual choice for the NN is a multi-layer perceptron (MLP).
A more efﬁcient way to produce the potential function would be to use the Jacobian of such an
NN : (cid:83) → (cid:83) .
The trace of the Jacobian gives the divergence at s of the vector ﬁeld generated by the
NN, which is a scalar in R :
[Equation]
As the Jacobian ∂ NN ( s ; θ ) /∂s can be decomposed into multiplication of the Jacobians of all the
NN’s layers, it can be computed efﬁciently.
In fact, it was empirically found to work better than theprevious simplest MLP.
For our implementation, we adopt a negative multi-layer planar (MLPlanar) transformation.
Each layer is a planar transformation [20] of two vectors v and u with a scalar b , as follows:
[Equation]
where θ = { v, b, u } is the parameter set.
The Jacobian of the l -th layer with input s ( l ) is ∂ Planar
[Equation]
s ( l [Equation]
[Equation]
where A ( l ) = tanh (cid:48) ( v T( l ) s ( l ) + b ( l ) )
· u ( l ) v T( l ) is a rank-one matrix, with the derivative tanh (cid:48)
=1 − tanh 2 .
Thus,
[Equation]
where s ( l ) =
Planar ( s ( l − 1) ) and s (0)
=
s .
Formula (6) uses the negative trace, so that words arelocated at “peaks” as in Figure 1 after training, rather than on the “plain”; this enables FIRE todiscover polysemy.
Formula (6) expands to a matrix polynomial of degree L , which can represent themultimodality of f ( s ) .
For the above case with D = 2 , evaluating f ( s ) takes O ( L ) time.
4.2 FIRE Training by Skip-Gram with Negative Sampling
For training FIRE, we adopt the Skip-gram algorithm with negative sampling (SGNS) [14], anunsupervised method.
More advanced training methods (e.g., GloVe ) can also be applied, whichremains a future work.
In SGNS (one negative word per time), a three-word tuple ( w i , w p , w n ) is sampled from a corpus.
Here, w i is called the center word, w p is a positive word that co-occurs with the center wordsomewhere in the corpus, and w n is an arbitrary word, called the negative word.
We use only onenegative word per positive, following the suggestion in [10].
The idea of SGNS is to optimize the parameters in word representations such that sim( w i , w p ) is highand sim( w i , w n ) is low, following the assumption that words co-occurring in the same neighborhoodare more similar.
For example, the words bank and mortgage should have a high similarity becausethey co-occur frequently.
Following the implementation in [13], the optimization function with respect to similarity is:
[Equation]
where σ ( x ) = log(1 + exp( x )) is the softplus function.
For D = 2 , evaluation of the similarity in
Formula (3) between two words has O ( KL ) time complexity.
Note that Formula (7) is not the triplet loss in metric learning.
A triplet loss is typically used toevaluate a large neural network shared by all words, whereas in FIRE, each word is represented as anindividual small neural network deﬁned by Formula (6).
4.3 Experimental Settings for Training
We obtained FIRE representations on two datasets: the large Wacky [3] dataset and the small text8dataset, which contain about 3 billion and 17 million tokens, respectively.
For both datasets, we converted the text to lowercase and tokenized it with the NLTK toolkit [4].
Wefollowed the settings used in previous works and replaced infrequent words with <unk> .
For Wacky ,infrequent words were those occurring less than 100 times, forming a vocabulary of around 270,000words; for text8 , the threshold was 5 times, forming a vocabulary of about 70,000 words.
As mentioned in Section 3.1, we used a 2-dimensional space (cid:83) =
[ − 4 , 4] 2 .
For the SGNS hyperparameters, we followed the settings recommended by [13].
We used onenegative sample per sample, and the subsample rate for w i was 1e-5 for Wacky and 1e-4 for text8 .
Thenegative sampling probability was adjusted by a power of 0.75.
For training FIRE representations, [Table] we adopted the AdamW optimizer [11].
The OneCycle learning rate scheduler [22] was applied [Table] withthe maximum learning rate set to 0. [Table]
As noted above, the time complexity for training is O ( KL ) .
For the Wacky dataset, all methods weretrained for three epochs, which took about 10 hours for FIRE with 50 parameters on a single GPU.
For text8 , FIRE was trained for 15 epochs.
In similarity benchmarks, conventional vectoral representations use the cosine similarity function,where sim( w i , w i ) =
1 ∀ i , to provide a natural regularization.
As FIRE also requires subtractingregularization terms we used the following for word similarity in our evaluation for fair comparison:
[Equation]
Substituting Σ in Formula (5) with the regularized version of word similarity gives the regularizedsentence similarity.
Because our similarity function is point-wise, smoothing is also required.
Appendix C details this process.
Baselines
We evaluated FIRE in comparison with previous semantic representations, as summarized in Table1.
For fair comparison of such different representations, the most important factor is the number ofparameters (2nd column).
For FIRE (bottom of the table), µ requires K locations with D dimensions,plus the number of weights m ( k ) , i.e., K , giving ( D + 1) K .
As for f ( s ) , v and u (Section 4.1) are
D -dimensional, are implemented as L layers, and have L intercepts b , giving (2 D + 1) L .
The totalis thus (2 D + 1) L
+ ( D + 1) K parameters per word.
The third column shows whether the methodis effective even with small D , where the small value D = 2 facilitates visualization.
The 4-5th columns in Table 1 indicate whether a method has the qualities of additive compositionalityand polysemy; FIRE is the only non-contextual method that theoretically accommodates both.
For an ablation study, we considered a variant of FIRE, denoted as FIRE/m , that ﬁxes m ( k ) i = 1 in
Formulas (2) and (3).
FIRE/m shares the properties of FIRE, but with fewer parameters per word.
We adopted the baselines listed in Table 1 in this work, as explained in Appendix D. Brieﬂy,
Word2Gauss and Word2GM represent words as Gaussian mixtures, whereas Word2Cloud represents words as a multinomial distribution of K discrete points in a low-dimensional space. /S
for
Word2Gauss and Word2GM indicates a spherical covariance matrix rI requiring one parameter
(i.e., r ), whereas Word2Gauss/D uses a diagonal covariance matrix diag ( r 1 , · · · , r D ) requiring Dparameters.
For Word2Cloud, we refer to the results reported in the original paper, because theoriginal code is not available. As Word2Cloud was only implemented on the {+relatively small text
+}
[Table]
dataset, for comparison, we constructed FIRE models with the same number of parameters andtrained them on the text8 dataset.
Evaluation on Basic Word Semantics
First, FIRE was compared with the baselines on 12 word similarity benchmarks to test the basicword semantic representational capability.
As these benchmarks are for non-contextual methods,
BERT-large was not included.
For each benchmark, we computed the similarity of a word to otherwords and generated a ranking of those words.
The gold standard was compared via Spearman’scorrelation coefﬁcient ρ , with a higher ρ indicating a better method.
The results are listed in Table 2.
Each block groups methods with the same number of parameters.
Generally speaking, our method outperformed most of the baselines, even the high-dimensional
Word2Vec.
For either #par = 50 or 100, FIRE was the only method whose performance was comparable to that of the vectoral methods.
For example, it achieved an average ρ of 47.9% and 49.8%,respectively, for #par =
40 and 50; these ρ values were only 1.1% and 0.6% less than those of
Word2Vec.
These results shows that our potential functions and measures have sufﬁciently largecapacity even with a semantic ﬁeld as small as [ − 4 , 4] 2 .
In Appendix E, we included further experimental results about K ’s effect.
Brieﬂy, our method is computationally efﬁcient even for a larger K ,because the time complexity of FIRE is O ( K ) , rather than O ( K 2 ) as in Word2GM and Word2Cloud.
Another perspective is to speciﬁcally compare the low-dimensional representations, i.e., those withsmall D .
Word2Cloud is such a representation and is comparable to FIRE with D = 2 .
Although
Word2Cloud was computed on the smaller text8 dataset, it produced better similarity scores than the2-dimensional Word2Vec listed in the top row of Table 2.
When trained on text8 , however, FIREgreatly outperformed Word2Cloud, even with a smaller number of parameters.
Evaluation of Compositionality via Sentence Similarity Benchmark
[Table]
FIRE’s linear compositional capability was evaluated using a sentence similarity benchmark.
Asentence representation is produced by composingwords in conventional vectoral word representations, making full use of linear compositionality.
Similarly, FIRE represents a sentence by composing word representations as described in Section 3.3.
The similarity of two sentences is given by Formula(5) or (8), with respect to the SIF weights [1].
Weset α = 10 − 3 for SIF.
We compared FIRE with vectoral word representations on the Word-in-Context (WiC)[19] and STS(2012-2016) benchmarks. WiC is a binary classiﬁcation task asking whether a word’s meaning differsin two given context sentences.
The semantic similarity scores between sentence pairs were computed,with scores below a threshold indicating that the word’s meaning differs in the two sentences.
STSevaluates sentence similarity by the Pearson correlation coefﬁcient between the predicted similarityand the gold standard.
Table 3 lists the results.
Note again that Word2Vec and FIRE are non-contextual, whereas BERT-largeis contextual.
The second column gives the model’s total number of parameters.
The Word2Vec and
FIRE both had 50 parameters per word.
The third gives the accuracy scores.
BERT-large, a contextual method, has 30 times more parameters than the non-contextual Word2Vecand FIRE, and its accuracy was higher because of the context stored in this large memory.
Between thenon-contextual methods, FIRE performed better than Word2Vec.
BERT’s accuracy was naturally thebest because it has context; more importantly, this result demonstrates that FIRE’s compositionalityis comparable to Word2Vec.
Evaluation of Polysemy
The quality of FIRE lies in its capability to capture polysemy.
Different meanings (i.e., senses) of aword are automatically represented as separate modes of a nonlinear function, as seen for bank in
Figure 1 with ﬁnance- and geography-related modes.
Appendix A gives more examples.
Words areassigned colors according to the clustering results in (cid:83) =
[ − 4 , 4] 2 , by the DBSCAN [7] algorithm,which automatically detects the number of clusters via a threshold parameter eps = 0 .
We quantitatively evaluated the polysemy of FIRE via the task of word sense detection , where thenumber of word senses is predicted by each method and compared with a gold standard via WordNet[15].
From the Core WordNet dataset, 1 with 5000 frequent senses of 3739 different words, weselected words annotated with at least three senses (266 words that are strongly polysemous).
From
Core WordNet and the whole WordNet, we also selected 276 words that are strongly monosemous.
We thus used 542 words, with various numbers of word senses.
The selected words are listed in
Supplementary Materials.
FIRE was compared with four previous word representations, Word2Vec, Word2Gauss, Word2GM,and BERT.
All methods except BERT are non-contextual.
With these non-contextual methodsincluding FIRE, for each target word, the most similar 1,000 words were acquired from the mostfrequent 50,000 words, and their locations were plotted in the 2D plane. For Word2Gauss and
Word2GM, we used the mean vectors of the Gaussian distributions.
See Appendix I for the details. In Figure 2 (left), we showed the results of three words: “bank” and “rock” are polysemous, while“glacier” is monosemous.
In contrast, BERT is a contextual model requiring context to count the number of senses.
For eachtarget word, 1,000 context sentences were collected from the Wacky dataset, and corresponding
BERT vectors for the target word were generated.
For a target word that was splitted into subwordsby the BERT’s default tokenizer, we took average of the subword vectors to represent the target word.
The rightmost column of Figure 2 (left) shows BERT’s results for the four sample words.
In plotting the Word2X methods, FIRE, and BERT on the 2D plane, we tested two dimensionreduction methods of Principal Component Analysis and Spectral Embeddings to acquire 2D pointsfrom the vectors for BERT (1024-d) and the other methods.
For the details, see Appendix I.
Figure 2 clearly shows how well FIRE captured the different senses.
The term “bank” producedthree senses, with those to the right corresponding to a river bank and “a large pile” as listed inmany dictionaries.
On the other hand, such clusters for Word2Vec and BERT are difﬁcult to see.
As Word2Vec is monosemous, it is not expected to produce any clusters, whereas clear clusters arebarely visible for BERT.
Although BERT performed very well in contextual processing, this showshow context and polysemy are different.
Word2GM uses two Gaussian components for each wordand is biased to show two clusters, even for the monosemous word “glacier.”
For all 542 words, the number of clusters t was counted with DBSCAN with different threshold eps :it was applied to to PCA-reduced 2-d planes, which produced a better result for BERT than applying
DBSCAN directly to the original high-dimensional spaces.
The number of clusters is comparedwith the gold standard.
As the gold standard consisted of 49% (=266/542) polysemous and 51%monosemous words, the result was judged accurate by t = 1 for monosemous words and t > 1 forpolysemous words.
Figure 2 (right) shows the results for the ﬁve methods, with the horizontal and vertical axes indicatingeps and the accuracy, respectively.
Word2Vec, Word2Gauss, and Word2GM performed at chance levelfor any eps value.
BERT was slightly better but could not detect the number of senses.
In contrast, FIRE showed a clear capability to identify polysemy.
Appendix F gives a further experimental resulton detecting the number of word senses.
Conclusion
We proposed FIRE, a semantic ﬁeld representation that is deﬁned on a D -dimensional space, with aﬁeld of word interactions.
A word is represented by a pair that consists of a set of multiple locationsfor the word and a function representing the word’s context, where the latter is implemented as asemantic potential function covering the ﬁeld.
The interaction of words is measured via a similarityfunction, which is naturally deﬁned as the sum of the potential functions at the word locations.
Themultiple possible word locations and the potential function’s multimodality account for the quality ofpolysemy, and the functional addition formally guarantees the quality of additive compositionality.
FIRE is implemented via a planar transformation of a neural network.
Multiple images of a 2Dﬁeld were shown to demonstrate the capability of polysemy representation.
FIRE was also evaluatedthrough a large-scale analysis of word and sentence similarity benchmarks, in which it showedcomparable or better results than conventional Word2XX representations.
Moreover, FIRE greatlyoutperformed BERT and Word2Vec in evaluating polysemy by predicting the number of word senses.
These results provide evidence of FIRE’s capability to incorporate polysemy as a nonlinear qualitywith compositionality as a linear quality.
Our future work will entail extension of this framework to a more general representation, which willfurther improve the capability of FIRE.
B Derivation of Sentence Similarity
Formula (5) is derived as follows:
[Equation]
=
[Equation]
(9) =
[Equation]
γ (cid:48)
[Equation]
C Smoothing of Similarity Scores by Standardization
One way to improve our framework is to standardize the similarity function.
Formally, for a word w i ,we apply the following transformation:
[Equation]
where the mean is deﬁned as
[Equation]
the variance is deﬁned as
[Equation]
and Var (cid:2) sim( w, w j ) (cid:3) is deﬁned similarly.
Such standardization works because of our measure’s point-wise nature, such that the similarityfunction in Formula (1) is optimized only at a point.
In other words, f is optimized very locally,rather than globally in (cid:83) .
Standardization of the similarity function serves to provide regularizationagainst overﬁtting.
It is executed after word representations are obtained, and thus, it does not affectthe training process.
A typical use of this standardization is for a word similarity benchmark, as in this work.
Such abenchmark usually contains hundreds of word pairs, which form a set of words.
We thus deﬁne W asthis set, and we apply the above standardization to the word representations with respect to W .
D Summary of Baseline Methods
Word2Vec[13] 2 A word is represented by a vector, trained via the Skip-gram algorithm.
GloVe[17] 3 This is a vectoral method similar to Word2Vec, but it uses a different training objective.
Word2Gauss [27] 4 A word is represented by a Gaussian random variable, parameterized by amean vector and a covariance matrix.
There are two variants of Word2Gauss: Word2Gauss/S uses aspherical covariance matrix rI that requires one parameter (i.e., r ), whereas Word2Gauss/D uses adiagonal covariance matrix diag ( r 1 , · · · , r D ) that requires D parameters.
Word2GM [2] 5 A word is represented as a weighted mixture of K Gaussian distributions, each ofwhich is parameterized as in Word2Gauss .
Word2GM also has two variants: Word2GM/S uses aspherical covariance matrix, and Word2GM/D uses a diagonal covariance matrix.
The original paper recommended setting K = 2 and using a spherical covariance matrix.
Accordingly,we compared that approach with FIRE here.
Word2Cloud [8].
A word is represented by a multinomial distribution of K discrete points in alow-dimensional space (e.g., R 2 ).
Word2Cloud also has K components.
The similarity between twowords is evaluated via the Wasserstein distance, which has at least O ( K 2 ) time complexity even witha fast approximation algorithm.
The original paper used K = 64 .
BERT-large [6]
The large version of BERT is composed of 24 Transformer layers that are stackedon vectoral word representations.
In each Transformer layer, words interact with other words toproduce new vectors that contain contextual information.
The vectoral output for a word is called acontextualized word vector, which was tested in Section 8.
We used the code for Word2GM and limited it to K = 1 , because this produced better results than the original code for Word2Gauss
E Supplementary Evaluation Results for Word Similarity
In FIRE, K is the number of locations for each word and is an important factor in polysemy.
Incomparison, Word2GM also implements polysemy, via mixtures.
As seen in Table 1 from the rowsfor Word2GM and FIRE, the mixture-based approach performed well even with small D .
From this perspective, we compared Word2GM and FIRE/m in terms of K , under the constraint of
D
= 2 .
Here, we used FIRE/m for fair comparison, so that we could construct models with a similarnumber of parameters to Word2GM.
The number of NN layers, L , was also set so that both modelswould have roughly equal numbers of parameters, as listed in the 2nd to 5th columns of Table 4.
[Table]
As mentioned in Section 4, our method is computationally efﬁcient even for a larger K : the timecomplexities of Word2GM and Word2Cloud grow by at least O ( K 2 ) , whereas FIRE has O ( K )complexity.
For example, the Word2GM training took around 24 hours for K = 2 , but the time grewto around 9 days for K = 6 .
Accordingly, we only tested K = 2 , 3 , 4 , 5 for Word2GM, whereas for
FIRE, we tested K up to 15.
Figure 4 shows the average Spearman’s ρ across the tasks of the word similarity benchmark.
Thevalues are listed in Table 4 (rightmost column).
For Word2GM, an increase in K did not improve theperformance for K ≥ 4 .
In contrast, FIRE enabled implementation of large K (e.g., 10), for whichthe performance greatly improved.
Thus, FIRE has a larger potential to improve the performance onword similarity benchmarks, by increasing the parameter K .
F Supplementary Evaluation of Detection of Word Senses
For all 542 words, the number of clusters was counted above a threshold “eps” in the clusteringalgorithm of DBSCAN.
The output number and the Pearson correlation with the gold standard werethen computed.
Figure 5 shows the results for ﬁve methods, with the horizontal axis indicating theparameter “eps” and the vertical axis indicating the Pearson correlation.
For all the tested choices of “eps,” FIRE achieved the highest Pearson correlation coefﬁcients with thegold standard for the number of word senses, reaching 19.8% for eps = 0 . 4 .
BERT showed moderateperformance, with correlation coefﬁcients between 4% and 11%, while the Word2Vec, Word2Gauss,and Word2GM methods failed for most choices of “eps.” G Dimensionalities Other Than 2 [Table]
H Results for Each of 24 STS Datasets
The STS benchmark comprises 24 datasets released during 2012-2016. Each dataset is a list ofsentence pairs. For each sentence pair, a human-annotated, gold-standard similarity score is provided. We tested Word2Vec, BERT, and our FIRE on the 24 datasets.
Following the conventional setting, amodel’s performance was evaluated by the Pearson correlation coefﬁcients between the predictedsimilarity scores and the gold standard.
Table 6 lists the results.
Each row is a dataset, and eachcolumn is a sentence-embedding model.
The bottom row gives the average scores, which are presentedin Table 3 in the main text.
[Table]
Students
[Table]
Headlines
[Table]
[Table]
[Table]
[Table]
[Table]
Headlines
[Table]
[Table]
Postediting
[Table]
Question
[Table]
[Table]
Average
[Table]
Average of all
[Table]
I Plotting on 2D Plane
In Figure 2 (left), we showed qualitative results for FIRE and the other methods, where the 1000 mostsimilar words out of the most frequent 50000 words were shown on a 2D plane.
Here, we describethe two dimension reduction methods that we used.
I.1 PCA
For Figure 2 (a), we applied principal component analysis to the word “vectors” and reduced thedimensionality to 2 to make the vectors visualizable on a 2D plane.
As some methods such as
Word2Gauss are not vectoral, we used slightly different ways to extract representative vectors forthese methods, as follows:
FIRE ( D = 2 , L = 4 , K = 1 )
We used the word location s ∈ R 2 as the vector.
Because we used
K = 1 , there was only one such vector for each of the 1000 words.
FIRE, PCA is to
Word2Vec
The word vector in R 50 was used.
Word2Gauss/S
The mean vector of the Gaussian distribution in R 49 was used.
Word2GM/S ( K = 2 )
Each word was represented by two independent Gaussian components,which represented different senses of the word.
For a target word such as “bank,” we considered500 components that were similar to the ﬁrst component of “bank” and 500 that were similar to thesecond.
As with Word2Gauss, we used the mean vector to represent a component.
We used the
Word2GM/S model listed in Table 2, which assigns 50 parameters to each word; thus, the meanvectors were in R 23 .
BERT We collected 1000 sentences for each target word from the Wacky dataset.
A sentence wasinput to the BERT model, and a contextualized word vector of the target word was acquired, with1024 dimensions.
I.2 Spectral Embedding of Word Similarity Matrix
Another way to visualize the most similar 1000 words was to use the similarity matrix between words.
For FIRE, the similarity was computed with Formulas (1) and (8).
For BERT, we used the cosinesimilarity between the contextualized word vectors.
A spectral embedding was acquired by applying SVD decomposition to the similarity matrix andtaking only the ﬁrst two eigenvectors.
We used the implementation in the sklearn.manifold package.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] All claims are supported by experimental results.
(b) Did you describe the limitations of your work?
[Yes] The longer training time ( ∼ 10hours) could be a limitation, which we have described in Section 4.3.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
We did
not recognize any negative societal impacts of this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] I have.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code, data, andpretrained models are in the supplementary materials.(b)
Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.3.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
We did not report error bars because it would be too
computationally expensive.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.3.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] The datasets wacky
and WordNet were cited.
As for text8 , we were unable to ﬁnd a proper way to cite the
creators.
(b) Did you mention the license of the assets?
[Yes] BERT-large & Word2Vec: Apache
License 2.0; Word2Gauss: MIT; Word2GM: BSD 3; Wacky: CC Attribution-
Noncommercial-Share Alike 4.0 International; text8: unavailable; WordNet 3.0 license.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We included a subset of the Core WordNet dataset, which we used to evaluate
polysemy.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
The new dataset was a subset of the WordNet that is open.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁ-
able information or offensive content?
[N/A]
The datasets do not contain personally
identiﬁable information.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]