Semantic Field of Words Represented as Nonlinear Functions
Abstract
State-of-the-art word embeddings presume a linear vector space, but this approach does not easily incorporate the nonlinearity that is necessary to represent polysemy.
We thus propose a novel semantic FIeld REepresentation, called FIRE, which is a D -dimensional ﬁeld in which every word is represented as a set of its locations and a nonlinear function covering the ﬁeld.
The strength of a word’s relation to another word at a certain location is measured as the function value at that location.
With FIRE, compositionality is represented via functional additivity, whereas polysemy is represented via the set of points and the function’s multimodality.
By implementing FIRE for English and comparing it with previous representation methods via word and sentence similarity tasks, we show that FIRE produces comparable or even better results.
In an evaluation of polysemy to predict the number of word senses, FIRE greatly outperformed BERT and Word2vec, providing evidence of how FIRE represents polysemy.
The code is available at https://github.com/kduxin/firelang .
Introduction
Word representations have become an important basic technique in NLP.
The most important semantic requirement for a word representation, above all, is that similar words are mapped to similar embeddings.
Furthermore, the next two most important properties are compositionality [27], i e., the representation of two composed words matches the composition of the two representations, and polysemy , the representation of semantic ambiguity.
Intuitively, compositionality is a linear quality [27, 1], whereas polysemy is nonlinear and thus requires a representation via mathematical multimodality, which is difﬁcult to represent within a linear space.
State-of-the-art word representations are based on a linear vector space.
The linearity of vectors naturally accommodates compositionality but the question of how to incorporate polysemy has been an issue.
Indeed, recent random-variable representations [29, 23, 25, 18, 10] represent polysemy but at the cost of losing compositionality.
The most important previous work that seems to incorporate polysemy in addition to compositionality could be BERT [8]; however, it is a contextual embedding that represents one vector only under a given context that disambiguates the meaning.
In contrast, a polysemous word has multiple meanings even without context.
Moreover, disambiguation (e.g., with BERT) is also hard, because semantics is often continuous: the word “bank” has 18 meanings in WordNet [17], which are hard to disambiguate even for humans.
Our idea was thus to devise an entirely new representation that is nonlinear but naturally represents linearity as well.
Hence, FIeld REpresentation, which we call FIRE, is a non-contextual representation of words.
It is realized in a D -dimensional space in which every point has a quantity, called a semantic strength , that is generated by interactions among words.
Every word is represented by a pair consisting of a set of the word’s locations and a nonlinear function f representing the word’s context, where the function outputs the semantic strength at a location in the ﬁeld.
Two functions can be linearly added, which provides the basis of compositionality.
Polysemy is represented through the set of a word’s locations and the multimodality of the function f .
Through this representation, the similarity of two words is obtained by adding the function values at the word locations.
Our experimental results conﬁrmed that FIRE showed similar or better results in comparison with previous Word2XX representations under settings with the same number of parameters.
Furthermore, by representing multimodality over the entire ﬁeld, FIRE has a much wider expressiveness for polysemy.
In an evaluation to predict the number of word senses, FIRE greatly outperformed BERT.
Related Works
Vectoral word representations.
The most widely used word representations are in vectoral forms, such as Word2Vec [15].
[12] showed that Word2Vec is essentially a decomposition of the point-wise mutual information matrix.
Other vectoral representation methods extract different information from corpora.
Important works include GloVe [19], fastText [6], ELMo [20], and BERT [8].
It has been attempted to represent polysemy via multiple vectors per word [26], but such a “mixture” of vectors lost the quality of compositionality.
Gaussian word representations.
[29] proposed Word2Gauss , which represents each word by a Gaussian distribution.
They exploited the covariance matrix to model the semantic ambiguity of words.
Asymmetric similarity (via the Kullback-Leibler divergence) was further introduced to represent hypernyms and hyponyms.
Word2GM [3] uses a mixture of (two) Gaussian distributions.
It is able to represent polysemy by assigning each Gaussian component to represent a separate meaning of a word.
[25] proposed to use the Wasserstein distance between two Gaussian distributions to measure word similarity, and that work was generalized by [18] to use elliptical distributions.
Between Gaussian or elliptical distributions, the Wasserstein distance has a closed form and can be computed efﬁciently.
We compare these Word2XX methods in Table 1.
Multinoulli distributions on point clouds.
[10] proposed Word2Cloud , in which each word is represented by a (discrete) Multinoulli distribution deﬁned on a cloud of 64 points in a lowdimensional space (e.g., R 2 ).
Between Multinoulli distributions, the Wasserstein distance has no closed form, but it is approximated efﬁciently by the Sinkhorn distance [7].
In the word clouds of
Word2Cloud, each point is a Dirac delta function, and we adopt that notion here.
The context mover’s distance (CMD) [23] also uses the Sinkhorn distance.
[23] used pretrained
GloVe vectors [19] in R 300 to construct a point cloud that represents a “context.”
In this paper, we do not show quantitative results for CMD, because we were unable to reproduce the original paper’s results with the published code.
Instead, we report results for Word2Cloud and GloVe: Word2Cloud is theoretically close to CMD, and [23] reported that GloVe showed comparable performance with
CMD.
Many of these random-variable word representations incorporate polysemy. However, none except CMD [23] incorporate compositionality.
Therefore, they have not been extended to sentence representations.
CMD [23] can produce a sentence representation by computing the Wasserstein barycenter of the words, which gives a new Multinoulli distribution on the point cloud.
However, that computation is not additive, whereas FIRE is additive.
Sentence representations.
A typical sentence representation is a vector obtained as a weighted additive composition of vectoral word representations.
Various weighting heuristics have been studied, such as the term frequency–inverse document frequency (TF-IDF) [14].
[2] proposed the smooth inverse frequency (SIF), a simple weighting scheme that works exceptionally well.
In this paper, too, we evaluate the capability of the sentence ﬁeld representation produced by applying the SIF to FIRE word representation.
State-of-the-art vectoral sentence representation methods adopt an adaptive weighting scheme by using a self-attention mechanism [28]; one implementation includes the sentence representation of
BERT [8].
Such methods can also be applied to our word ﬁeld representation, which remains a future work.
Semantic Field: FIRE 3.1 Word Representation
FIRE is a semantic ﬁeld generated by words.
In physics, a ﬁeld is a space in which every point has some physical quantity, of which a common type is the strength , e.g., electric ﬁeld strength.
FIRE uses the semantic strength generated by words.
While particles in a physical ﬁeld are often identical, the particularity of FIRE is that every word produces a different semantic strength and is thus allocated within the ﬁeld in association with an individual function.
Let W be a set of words and (cid:83) be a D -dimensional real space.
Our semantic ﬁeld is deﬁned on (cid:83) .
A word w ∈ W is represented as a 2-tuple [ µ, f ( s )] , where µ represents the word’s locations.
In a previous linear space model [15], a word was usually located at a single point.
However, a word can be polysemous and in such cases, it requires a representation with multiple locations, as suggested in [26].
Let K denote the number of locations.
f ( s ) : (cid:83) → R is a continuous real-valued function covering (cid:83) , realized as a neural network.
f ( s ) represents the context generated by the surrounding words of w , as will be explained in Section 4.1.
Figure 1 (left) shows an example portion of (cid:83) computed for D = 2 with (cid:83) =
[ − 4 , 4] 2 and K = 1 , for the term bank .
The location µ for bank is shown by the orange triangle.
There are two clusters related to bank , one for a ﬁnancial bank and the other for a river bank.
The circles show the locations µ for 1000 words (among the most frequent 50,000 words) that were frequent and similar to bank , with similarity as deﬁned in the following section.
3.2 Similarity of Two Words
In our approach, two words w i =
[ µ i , f i ( s )] and w j = [ µ j , f j ( s )] in a ﬁeld interact.
Given the location s j of w j , the interaction with w i is measured by f i ( s j ) .
As mentioned above, polysemy means that w j has multiple locations, s ( k ) j , k
= 1 , . .
, K ; thus, the f i ( s ( k ) j ) must be added.
The location here means that an entity is scattered in a ﬁeld: it can be multiple and noncontinuous, and it may have different mass at different locations.
This notion is mathematically expressed as a measure , and an integral with a measure is formulated via the Lebesgue integral.
The simplest measure is the Dirac delta function δ ( s ) , which indicates that a location is point-wise within a ﬁeld.
For K = 1 and µ j ≡ δ ( s j ) , we have (cid:82) f i d µ j = f i ( s j ) .
By using the Dirac delta function, we can compute the integral exactly without any approximation.
FIRE incorporates polysemy via (1) the multimodality of f ( s ) and (2) the number K of word locations.
As mentioned above, the multimodality of f ( s ) is implemented via a neural network.
As we also have the symmetric quantity of w i ’s location in the ﬁeld of f j ( s ) and w j ’s location in the ﬁeld of f i ( s ) , the similarity of the two words is naturally expressed as follows:
[Equation]
Using a measure for a word’s location facilitates a simple representation.
For K ≥ 1 , µ is a Dirac mixture with weights m ( k ) as follows:
[Equation]
By using this µ for each word, Formula (1) reduces to the following (because (cid:82)
f d δ ( s ) =
f ( s ) ):
[Equation]
which is simply a weighted sum of the two functions
f i ( s ) and f j ( s ) at all the locations of w j and w i , respectively.
The similarity in this work between words and sentences (which we deﬁne later in Section 3.3) via a measure is different from the standard notion of distance in a Euclidean space and is related to metric learning methods [11].
Like in metric learning, FIRE is nonlinear with f realized as a multi-layer neural network.
Whereas metric learning methods share a large neural network for all inputs, FIRE assigns an individual small neural network to each word.
3.3 Composition of Words and Similarity of Two Sentences
A set of n words, Γ = [ w 1 , . . . , w n ] , including sentences, can also be represented in [ µ, f ( s )] notation when deﬁned as a weighted average of word representations { [ µ 1 , f 1 ( s )] , . . .
, [ µ n , f n ( s )] } , as follows:
[Equation]
where γ i are the weights.
In this work, we used SIF weights [2] as the γ i .
Note that this f ( s ) is multimodal and thus polysemous, because all the f i ( s ) are multimodal.
Figure 1 (right) shows such a superposition of the words river (blue) and ﬁnancial (red).
Here, river is at the lower mid, with f river ( s ) forming a diagonal river-like ﬂow from upper right to lower left, whereas ﬁnancial is at the upper left, with f ﬁnancial ( s ) forming a reddish area.
The color intensities and arrows represent the function values and gradients, respectively.
The ﬁeld superposing river and ﬁnancial is similar to the representation of bank on the left, demonstrating the effect of FIRE’s linear composition.
The similarity of two sets of words or sentences, Γ = [ µ, f ] and Γ (cid:48) =
[ µ (cid:48) , f (cid:48) ] , is measured by applying the similarity function (1).
Let n and n (cid:48) denote the respective numbers of words in the two sentences.
The words in Γ and Γ (cid:48) are assigned weights γ =
[ γ 1 , . . . , γ n ] T and γ (cid:48)
=
[ γ (cid:48) 1 , . .
, γ (cid:48) n (cid:48) ] T , respectively.
Then, as shown in Appendix B, Formula (1) is simply calculated as follows:
[Equation]
where Σ is the word similarity matrix, deﬁned as Σ ij = sim( w i , w j ) with w i and w j being words in sentences Γ and Γ (cid:48) , respectively.
In other words, the similarity between two sentences can be obtained from the pairwise word similarities between the sentences.
Implementation of FIRE 4.1 Nonlinear Functions As Planar Transformations
The simplest way to implement the function f ( s ) would be to construct a neural network NN with a D -dimensional input and 1 -dimensional output, i.e., f ( s ) =
NN ( s ; θ ) , where θ represents the parameters of the neural network.
A usual choice for the NN is a multi-layer perceptron (MLP).
A more elaborative way to produce the function would be to use the Jacobian of such an NN : (cid:83) → (cid:83) .
The trace of the Jacobian gives the divergence at s of the vector ﬁeld generated by the NN, which is a scalar in R :
[Equation]
As the Jacobian ∂ NN ( s ; θ ) /∂s can be decomposed into multiplication of the Jacobians of all the
NN’s layers, it can be computed efﬁciently.
In fact, it was empirically found to work better than the previous simplest MLP.
In particular, a normalizing ﬂow [22], i.e., a reversible neural network, can be used here as NN, as it naturally generates a vector ﬁeld (cid:83) → (cid:83) .
For our implementation, we adopt a negative multi-layer planar (MLPlanar) transformation [22], which is a simple kind of normalizing ﬂow.
Each layer of
MLPlanar is determined by two vectors v and u with a scalar b , as follows:
[Equation]
where θ = { v, b, u } is the parameter set.
1 The Jacobian of the l -th layer with input s ( l ) is
[Equation]
, where A ( l ) = tanh (cid:48) ( v T( l ) s ( l )
+ b ( l ) ) · u ( l ) v T( l ) is a rank-one matrix, with the derivative tanh (cid:48)
= 1 − tanh 2 .
Thus, f
[Equation]
s
[Equation]
∂
[Equation]
where s ( l ) =
Planar ( s ( l − 1) ) and s (0)
=
s .
Formula (7) uses the negative trace, so that f has modes that are mostly “peaks” as in Figure 1, rather than “valleys.”
Because similar words are located where f has higher values, the negative trace drives similar words toward a “peak,” which facilitates the separation of a word’s multiple meanings.
This enables FIRE to discover polysemy.
Formula (7) expands to a matrix polynomial of degree L , which can represent the multimodality of f ( s ) .
For the above case with D = 2 , evaluating f ( s ) takes O ( L ) time.
4.2 FIRE Training by Skip-Gram with Negative Sampling
For training FIRE, we adopt the Skip-gram algorithm with negative sampling (SGNS) [16], an unsupervised method.
More advanced training methods (e.g., GloVe ) can also be applied, which remains a future work.
In SGNS (one negative word per time), a three-word tuple ( w i , w p , w n ) is sampled from a corpus.
Here, w i is called the center word, w p is a positive word that co-occurs with the center word somewhere in the corpus, and w n is an arbitrary word, called the negative word.
We use only one negative word per positive, following the suggestion in [12].
The idea of SGNS is to optimize the parameters in word representations such that sim( w i , w p ) is high and sim( w i , w n ) is low, following the assumption that words co-occurring in the same neighborhood are more similar.
For example, the words bank and mortgage should have a high similarity because they co-occur frequently.
Following the implementation in [15], the optimization function with respect to similarity is
[Equation]
where σ ( x ) = log(1 + exp( x )) is the softplus function.
Evaluation of the similarity in Formula (3) between two words has O ( KL ) time complexity when D = 2 .
Note that Formula (8) is not the triplet loss in metric learning.
A triplet loss is typically used to evaluate a large neural network shared by all words, whereas in FIRE, each word is represented as an individual small neural network deﬁned by Formula (7).
4.3 Experimental Settings for Training
We obtained FIRE representations on two datasets: the large Wacky [4] dataset and the small text dataset, which contain about 3 billion and 17 million tokens, respectively.
For both datasets, we converted the text to lowercase and tokenized it with the NLTK toolkit [5].
We followed the settings used in previous works and replaced infrequent words with <unk> .
For Wacky , infrequent words were those occurring less than 100 times, forming a vocabulary of around 270, words; for text8 , the threshold was 5 times, forming a vocabulary of about 70,000 words.
As mentioned in Section 3.1, we used a 2-dimensional space (cid:83) =
[ − 4 , 4] 2 .
For the SGNS hyperparameters, we followed the settings recommended by [15].
We used one negative sample per sample, and the subsample rate for w i was 1e-5 for Wacky and 1e-4 for text8 .
The negative sampling probability was adjusted by a power of 0.75.
For training FIRE representations, we adopted the AdamW optimizer [13].
The OneCycle learning rate scheduler [24] was applied, with the maximum learning rate set to 0.005.
As noted above, the time complexity for training is O ( KL ) .
For the Wacky dataset, all methods were trained for three epochs, which took about 10 hours for FIRE with 50 parameters per word on a single GPU.
For text8 , FIRE was trained for 15 epochs.
In similarity benchmarks, conventional vectoral representations use the cosine similarity function, where sim( w i , w i ) =
1 ∀ i , to provide a natural regularization.
For fair comparison of FIRE with baselines, we used the following regularization for word similarity in our evaluation:
[Equation]
For sentences, substituting Σ in Formula (5) with the regularized version of word similarity gives the regularized sentence similarity.
Because our similarity function is point-wise, smoothing is also required.
Appendix C details this process.
Baselines
We qualitatively compared FIRE with previous semantic representations, as summarized in Table 1.
For fair comparison of such different representations, the most important factor is the number of parameters (sixth column).
For FIRE (bottom of the table), µ requires K locations with D dimensions, plus the number of weights m ( k ) , i.e., K , giving ( D + 1) K .
As for f ( s ) , v and u (Section 4.1) are
D -dimensional, are implemented as L layers, and have L intercepts b , giving (2 D + 1) L .
The total is thus (2 D + 1) L + ( D + 1) K parameters per word.
The second column shows whether the method must process a word with a context, and only BERTlarge has this requirement.
The third and fourth columns in Table 1 indicate whether a method has the qualities of additive compositionality and polysemy; FIRE is the only non-contextual method that theoretically accommodates both.
Furthermore, as shown by the ﬁfth column, FIRE and Word2Cloud are the only two that represent polysemy in an interpretable way, i.e., that are visualizable on a 2D plane without dimensionality reduction.
However, as will be explained in the next section,
Word2Cloud is not comparable with FIRE on word similarity benchmarks.
For an ablation study, we considered a variant of FIRE, denoted as FIRE/m , that ﬁxes m ( k ) i = 1 in
Formulas (2) and (3).
FIRE/m shares the properties of FIRE, but with fewer parameters per word.
[Table]
We adopted the baselines listed in Table 1 in this work, as explained in Appendix D. Brieﬂy,
Word2Gauss and Word2GM represent words as Gaussian mixtures, whereas Word2Cloud represents words as a Multinoulli distribution of K discrete points in a low-dimensional space.
For Word2Gauss and Word2GM, “/S” indicates a spherical covariance matrix rI requiring one parameter (i.e., r ), whereas “/D” indicates a diagonal covariance matrix diag ( r 1 , · · · , r D ) requiring D parameters.
For
Word2Cloud, we refer to the results reported in the original paper, because the original code is not available.
As Word2Cloud was only implemented on the relatively small text8 dataset, for comparison, we constructed FIRE models with the same number of parameters and trained them on the text dataset.
Evaluation on Basic Word Semantics
First, FIRE was compared with the baselines on 12 word similarity benchmarks to test the basic word semantic representational capability.
As these benchmarks are for non-contextual methods,
BERT-large was not included.
For each benchmark, we computed the similarity of a word to other words and generated a ranking of those words.
The gold standard was compared via Spearman’s correlation coefﬁcient ρ , with a higher ρ indicating a better method.
The results are listed in Table 2.
Each block groups methods with the same number of parameters.
Generally speaking, our method produced comparable or often even better results than the baselines.
For either #par = 50 or 100 (third or fourth block, respectively), FIRE was the only method whose performance was comparable to that of the vectoral methods.
For example, it achieved an average ρ of 47.9% and 49.8%, respectively, for #par =
40 and 50; these ρ values were only 1.1% and 0.6% less than those of Word2Vec
(second and third blocks).
These results show that our functions and measures have sufﬁciently large capacity even with a semantic ﬁeld as small as [ − 4 , 4] 2 .
In Appendix
E, we included further experimental results about K ’s effect.
Brieﬂy, our method is computationally efﬁcient even for a larger K , because the time complexity of FIRE is O ( K ) , rather than O ( K 2 ) as in
Word2GM and Word2Cloud.
Another perspective is to speciﬁcally compare the low-dimensional representations, i.e., those with small D (last block).
Word2Cloud is such a representation and is comparable to FIRE with D = 2 .
Although Word2Cloud was computed on the smaller text8 dataset, it produced better similarity scores than the 2-dimensional Word2Vec listed in the top row of Table 2.
When trained on text8 , however,
FIRE greatly outperformed Word2Cloud, even with a smaller number of parameters.
[Table]
Evaluation of Compositionality via Sentence Similarity Benchmark
FIRE’s linear compositional capability was evaluated using a sentence similarity benchmark.
A sentence representation is produced by composing words in conventional vectoral word representations, making full use of linear compositionality.
Similarly, FIRE represents a sentence by composing word representations as described in Section 3.3.
The similarity of two sentences is given by Formulas (5) and (9) (see Appendix C), with respect to the SIF weights [2].
We set α = 10 − 3 for SIF.
[Table]
We compared FIRE with vectoral word representations on the Word-in-Context (WiC) [21] and STS (2012-2016) benchmarks.
WiC is a binary classiﬁcation task asking whether a word’s meaning differs in two given context sentences.
The semantic similarity scores between sentence pairs were computed, with scores below a threshold indicating that the word’s meaning differs in the two sentences.
STS evaluates sentence similarity by the Pearson correlation coefﬁcient between the predicted similarity and the gold standard (see Appendix G).
Table 3 lists the results.
Note again that Word2Vec and FIRE are non-contextual, whereas BERT-large is contextual.
The second and third columns give the model’s number of parameters per word and its total parameters, respectively.
The fourth and ﬁfth columns give the accuracy and correlation scores, respectively.
BERT-large, a contextual method, has 20 times more parameters than the non-contextual Word2Vec and FIRE, and its accuracy and correlation were higher because of the context stored in this large memory.
Between the non-contextual methods, FIRE performed better than Word2Vec on WiC.
BERT’s performance was naturally the best because it has context; more importantly, this result demonstrates that FIRE’s compositionality is comparable to Word2Vec.
Evaluation of Polysemy
The quality of FIRE lies in its capability to capture polysemy.
Different meanings (i.e., senses) of a word are automatically represented as separate modes of a nonlinear function, as seen for bank in
Figure 1 with ﬁnance- and geography-related modes.
Appendix A gives more examples.
We quantitatively evaluated the polysemy of FIRE via the task of word sense detection , where the number of word senses is predicted by each method and compared with a gold standard via WordNet [17].
From the Core WordNet dataset, 2 with 5000 frequent senses of 3739 different words, we selected words annotated with at least three senses (266 words that are strongly polysemous).
From Core
WordNet and the whole WordNet, we also selected 276 words that are strongly monosemous.
We thus used 542 words, with various numbers of word senses.
FIRE was compared with four previous word representations: Word2Vec, Word2Gauss, Word2GM, and BERT.
All methods except BERT are non-contextual.
With the non-contextual methods, including
FIRE, for each target word, the most similar 1,000 words were acquired from the most frequent 50,000 words, and their locations were plotted in the 2D plane.
For Word2Gauss and Word2GM, we used the mean vectors of the Gaussian distributions.
See Appendix H for the details.
In Figure 2 (upper), we show the results, with dimensionality reduction by (a) principal component analysis (PCA) and (b) spectral embeddings, for three words: “bank” and “rock” are polysemous, while “glacier” is monosemous.
In contrast, BERT is a contextual model requiring context to count the number of senses.
For each target word, 1,000 context sentences were collected from the Wacky dataset, and corresponding
BERT vectors for the target word were generated.
For a target word that was split into subwords by BERT’s default tokenizer, we averaged the subword vectors to represent the target word.
The rightmost columns in Figure 2 (upper, a and b) show BERT’s results for the three sample words.
As mentioned above, for plotting on the 2D plane, we used two dimension reduction methods, PCA and spectral embeddings, to acquire 2D points from the vectors for BERT (1024-d) and the other methods.
For FIRE, the application of PCA is equivalent to performing rotation and normalization, but we nevertheless applied it for fair comparison.
For the details, see Appendix H.
Figure 2 (upper) clearly shows how well FIRE captured the different senses.
The term “bank” produced three senses, with those to the right corresponding to a river bank and a “large pile,” as listed in many dictionaries.
On the other hand, such clusters for Word2Vec and BERT are difﬁcult to see.
As
Word2Vec is monosemous, it is not expected to produce any clusters, while clear clusters are barely visible for BERT.
Although BERT performed well in contextual processing, this shows how context and polysemy are not necessarily the same.
As for Word2GM, it uses two Gaussian components for each word and is biased to show two clusters, even for the monosemous word “glacier.”
For all 542 words, the number of clusters, t , was counted with DBSCAN with different thresholds eps .
For all methods the number was counted on the PCA-reduced 2D planes.
For BERT, this PCA result was better than direct application of DBSCAN to the original high-dimensional spaces.
The number of clusters was then compared with the gold standard.
As the gold standard consisted of 49% (= 266 / 542) polysemous and 51% monosemous words, the result was judged accurate by t = 1 for monosemous words and t > 1 for polysemous words.
Figure 2 (lower) shows the results for the ﬁve methods, with the horizontal axes indicating eps .
The vertical axes of the lower-left and lower-right graphs show the binary accuracy and Pearson correlation coefﬁcient, respectively.
Word2Vec, Word2Gauss, and Word2GM performed at chance level for any eps value.
BERT was slightly better but could not detect the number of senses.
In contrast, FIRE showed a clear capability of identifying polysemy.
Conclusion
We proposed FIRE, a semantic ﬁeld representation that is deﬁned on a D -dimensional space, with a ﬁeld of word interactions.
A word is represented by a pair that consists of a set of multiple locations for the word and a function covering the semantic ﬁeld that represents the word’s context.
The interaction of words is measured via a similarity function, which is naturally deﬁned as the sum of the functions at the word locations.
The multiple possible word locations and the function’s multimodality account for the quality of polysemy, and the functional addition formally guarantees the quality of additive compositionality.
FIRE is implemented via a planar transformation of a neural network.
Multiple images of a 2D ﬁeld were shown to demonstrate the capability of polysemy representation.
FIRE was also evaluated through a large-scale analysis of word and sentence similarity benchmarks, in which it showed comparable or better results than conventional Word2XX representations.
Moreover, FIRE greatly outperformed BERT and Word2Vec in evaluating polysemy by well predicting the number of word senses.
These results provide evidence of FIRE’s capability to incorporate polysemy as a nonlinear quality with compositionality as a linear quality.
Our future work will entail extension of this framework to a more general representation, which will further improve the capability of FIRE.
Acknowledgements
This work was supported by JST, CREST Grant Number JPMJCR2114, Japan, and by JSPS, KAKENHI Grant Numbers JP20K20492 and JP21J11781.
B Derivation of Sentence Similarity
Formula (5) is derived as follows:
[Equation]
C Smoothing of Similarity Scores by Standardization
One way to improve our framework is to standardize the similarity function.
Formally, for a word w i , we apply the following transformation:
[Equation]
where the mean is deﬁned as
[Equation]
the variance is deﬁned as
[Equation]
and Var (cid:2) sim( w, w j ) (cid:3) is deﬁned similarly.
This standardization should be applied after the regularization given in Formula (9).
Thus, in a word similarity benchmark, the predicted word-wise similarity is computed in three steps:
compute the word similarity by Formula (1), 2.
regularize the similarity score via Formula (9), and 3. apply standardization via Formula (10).
Such standardization works because of our measure’s point-wise nature, such that the similarity function in Formula (1) is optimized only at a point.
In other words, f is optimized very locally, rather than globally in (cid:83) .
Standardization of the similarity function serves to provide another regularization against overﬁtting.
It is executed after word representations are obtained, and thus, it does not affect the training process.
A typical use of this standardization is for a word similarity benchmark, as in this work.
Such a benchmark deﬁnes a set of words on which we can perform standardization.
We thus deﬁne W as this set, and we compute the similarity of word or sentence pairs that are composed of words in W .
Similarity benchmarks for sentences also require this standardization procedure, with an additional step (i.e., step 2 in the following) for composing word similarities into sentence similarities:
compute the word similarity by Formula (1), 2.
compute the sentence similarity by composing the word similarity scores via Formula (5), 3.
regularize the sentence similarity via Formula (9), and 4.
apply standardization via Formula (10).
D Summary of Baseline Methods
Word2Vec[15] 4 A word is represented by a vector, trained via the Skip-gram algorithm.
GloVe[19] 5 This vectoral method is similar to Word2Vec but uses a different training objective function.
Word2Gauss [29] 6 A word is represented by a Gaussian random variable, parameterized by a mean vector and a covariance matrix.
There are two variants of Word2Gauss: Word2Gauss/S uses a spherical covariance matrix rI that requires one parameter (i.e., r ), whereas Word2Gauss/D uses a diagonal covariance matrix diag ( r 1 , · · · , r D ) that requires D parameters.
Word2GM [3] 7 A word is represented as a weighted mixture of K Gaussian distributions, each of which is parameterized as in Word2Gauss .
Word2GM also has two variants: Word2GM/S uses a spherical covariance matrix, and Word2GM/D uses a diagonal covariance matrix.
The original paper recommended setting K = 2 and using a spherical covariance matrix.
Accordingly, we compared that approach with FIRE in the main text.
Word2Cloud [10] A word is represented by a Multinoulli distribution of K discrete points in a low-dimensional space (e.g., R 2 ).
Word2Cloud also has K components.
The similarity between two words is evaluated via the Wasserstein distance, which has at least O ( K 2 ) time complexity even with a fast approximation algorithm.
The original paper used K = 64 .
BERT-large [8] The large case-insensitive version of BERT is composed of 24 Transformer layers that are stacked on vectoral word representations.
In each Transformer layer, words interact with other words to produce new vectors that contain contextual information.
The vectoral output for a word is called a contextualized word vector, which was tested in Section 8.
E Supplementary Evaluation Results for Word Similarity
In FIRE, K is the number of locations for each word and is an important factor in polysemy.
In comparison, Word2GM also implements polysemy, via mixtures.
As seen in Table 1 from the rows for Word2GM and FIRE, the mixture-based approach performed well even with small D .
From this perspective, we compared Word2GM and FIRE/m in terms of K , under the constraint of
D
= 2 .
Here, we used FIRE/m for fair comparison, so that we could construct models with a close number of parameters to Word2GM.
The number of NN layers, L , was also set so that both models would have roughly equal numbers of parameters, as listed in the second to ﬁfth columns of Table 4.
As mentioned in Section 4, our method is computationally efﬁcient even for a larger K : the time complexities of Word2GM and Word2Cloud grow by at least O ( K 2 ) , whereas FIRE has O ( K ) complexity.
For example, the Word2GM training took around 24 hours for K = 2 , but the time grew to around 9 days for K = 6 .
Accordingly, we only tested K = 2 , 3 , 4 , 5 for Word2GM, whereas for
FIRE, we tested K up to 15.
Figure 4 shows the average Spearman’s ρ across the tasks of the word similarity benchmark.
The values are listed in Table 4 (rightmost column).
For Word2GM, an increase in K did not improve the performance for K ≥ 4 .
In contrast, FIRE enabled implementation of large K (e.g., 10), for which the performance greatly improved.
Thus, FIRE has a larger potential to improve the performance on word similarity benchmarks, by increasing the parameter K .
[Table]
F Dimensionalities Other Than
[Table]
G Results for Each of 24 STS Datasets
The STS benchmark consists of 24 datasets released during 2012-2016.
Each dataset is a list of sentence pairs.
For each sentence pair, a human-annotated, gold-standard similarity score is provided.
We tested Word2Vec, BERT, and our FIRE on the 24 datasets.
Following the conventional setting, a model’s performance was evaluated by the Pearson correlation coefﬁcients between the predicted similarity scores and the gold standard.
Spearman’s correlation coefﬁcient ( ρ ), which we used for the word similarity benchmarks, was not used for evaluating sentences, because the sentence benchmarks contained many duplicate similarity scores that made Spearman’s ρ non-unique.
Table 6 lists the results.
Each row is a dataset, and each column is a sentence-embedding model.
The bottom row gives the average scores, which are given in Table 3 in the main text.
On average, FIRE performed slightly worse than Word2Vec, though it actually had better performance on many individual datasets.
[Table]
H Plotting on 2D Plane
In Figure 2 (upper), we showed qualitative results for FIRE and the other methods, where themost similar words out of the most frequent 50000 words were shown on a 2D plane.
Here, we describe the two dimensionality reduction methods that we used.
H.1 PCA
For Figure 2(a), we applied principal component analysis (PCA) to the word “vectors” and reduced the dimensionality to 2 to make the vectors visualizable on a 2D plane.
Because all methods are different, we adapted each one to extract representative vectors for PCA, as follows:
FIRE ( D = 2 , L = 4 , K = 1 )
We used the word location s ∈ R 2 as the vector.
Because we used
K = 1 , there was only one such vector for each of the 1000 words.
For FIRE, PCA is equivalent to performing rotation and normalization.
Word2Vec
The word vector in R 50 was used.
Word2Gauss/S
The mean vector of the Gaussian distribution in R 49 was used.
Word2GM
S
K = 2 )
Each word was represented by two independent Gaussian components, which represented different senses of the word.
For a target word such as “bank,” we considered 500 components that were similar to the ﬁrst component of “bank” and 500 that were similar to the second.
As with Word2Gauss, we used the mean vector to represent a component.
We used the
Word2GM/S model listed in Table 2 (third block), which assigns 50 parameters to each word; thus, the mean vectors were in R 23 .
BERT-large We collected 1000 sentences for each target word from the Wacky dataset.
A sentence was input to the BERT model
and a contextualized word vector of the target word was acquired, with 1024 dimensions.
H.2 Spectral Embedding of Word Similarity Matrix
Another way to visualize the most similar 1000 words was to use the similarity matrix between words.
For FIRE, the similarity was computed with Formulas (1) and (9).
For BERT, we used the cosine similarity between the contextualized word vectors.
A spectral embedding was acquired by applying SVD decomposition to the similarity matrix and taking only the ﬁrst two eigenvectors.
We used the