I MPOSING C ATEGORY T REES O NTO W ORDE MBEDDINGS U SING A G EOMETRIC C ONSTRUCTION
A BSTRACT
We present a novel method to precisely impose tree-structured category information onto word-embeddings, resulting in ball embeddings in higher dimensional spaces ( N -balls for short).
Inclusion relations among N -balls implicitly encode subordinate relations among categories.
The similarity measurement in terms of the cosine function is enriched by category information.
Using a geometric construction method instead of back-propagation, we create large N -ball embeddings that satisfy two conditions: (1) category trees are precisely imposed onto word embeddings at zero energy cost; (2) pre-trained word embeddings are well preserved.
A new benchmark data set is created for validating the category of unknown words.
Experiments show that N -ball embeddings, carrying category information, signiﬁcantly outperform word embeddings in the test of nearest neighborhoods, and demonstrate surprisingly good performance in validating categories of unknown words.
Source codes and data-sets are free for public access https://github.com/GnodIsNait/nball4tree.git and https://github.com/GnodIsNait/bp94nball.git .
I NTRODUCTION
Words in similar contexts have similar semantic and syntactic information.
Word embeddings are vector representations of words that reﬂect this characteristic ( Mikolov et al., 2013; Pennington et al., 2014) and have been widely used in AI applications such as question-answering (Tellex et al., 2003), text classiﬁcation (Sebastiani, 2002), information retrieval (Manning et al., 2008), or even as a building-block for a uniﬁed NLP system to process common NLP tasks (Collobert et al., 2011).
To enhance semantic reasoning, researchers proposed to represent words in terms of regions instead of vectors.
For example, Erk (2009) extended a word vector into a region by estimating the log-linear probability of weighted feature distances and found that hyponym regions often do not fall inside of their hypernym regions.
By using external hyponym relations, she obtained 95.2% precision and 43.4% recall in hypernym prediction for a small scale data set.
Her experiments suggest that regions structured by hyponym relations may not be located within the same dimension as the space of word embeddings.
Yet, how to construct strict inclusion relations among regions is still an open problem when representing hypernym relations.
In this paper, we restrict regions to be n dimensional balls ( N -ball for short) and propose a novel geometrical construction approach to impose tree-structured category information onto word embeddings.
This is guided by two criteria: (1) Subordinate relations among categories shall be implicitly and precisely represented by inclusion relations among corresponding N -balls.
This way, the energy costs of imposing structure will be zero; (2) Pre-trained word embeddings shall be well-preserved.
Our particular contributions are as follows: (1) The proposed novel geometric approach achieves zero energy costs of imposing tree structures onto word-embeddings. By considering category information in terms of the boundary of an N -ball, we propose a new similarity measurement that is
[Figure]
more precise and sensitive than conventional cosine similarity measurements.
(3) We create a large data set of N -ball embeddings using the pre-trained GloVe embeddings and a large category tree of word senses extracted from Word-Net 3.0.
The remainder of our presentation is structured as follows: Section 2 presents the structure of N -ball embeddings; Section 3 describes the geometric approach to construct N -ball embeddings; Section 4 presents experiment results; Section 5 brieﬂy reviews related work; Section 6 concludes the presented work, and lists on-going research.
R EGION -B ASED E MBEDDINGS
Tree structures occur in many applications and are used to describe, say, ﬁle systems, syntactic structures, taxonomies of plants or animals, or subordinate relations of governments.
Here, we use the tree structure of hyponym relations among word senses as to construct N -ball embeddings.
Formally, an N -ball of word sense w with central point ⃗A w and radius r w is written as !
w = B ( ⃗A w ; r w ) and deﬁned as the set of vectors whose Euclidean distance to ⃗A w is less than r w :
B ( ⃗A w ; r w ) ≜
f ⃗p j∥ ⃗A w (cid:0) ⃗p ∥ <
r w g .
N -balls are deﬁned as open-regions, as illustrated in Figure 1(a), so they are not RCC regions that can be either open or closed, or even a mixture, thus avoiding a number of problems ( Dong, 2008; Davis & Marcus, 2015).
2. R ELATIONS B ETWEEN N - BALLS
We distinguish two topological relations between N -balls: being inside and being disconnected from as illustrated in Figure 1(b, c). !
u
= B ( ⃗A u ; r u ) being inside !
w = B ( ⃗A w ; r w ) can be measured by the result of subtracting the sum of radius r u and the distance between their central vectors from radius r w .
Formally, we deﬁne D inside ( !
u ; !
w ) ≜ r w (cid:0) r u (cid:0) ∥ ⃗A u (cid:0) ⃗A w ∥ .
So, !
u is inside of !
w ( w is the hypernym of u ), if and only if D inside ( !
u ; !
w ) (cid:21) 0 .
If v is the hypernym of w , w is the hypernym of u , we have D inside ( !
u ; !
v ) >
D inside ( !
u ; !
w )
+ D inside ( !
w ; !
v ) (cid:21) D inside ( !
w ; !
v ) .
The direct hypernym of !
v , written as dh ( ! v ) , can be deﬁned as the !
x which produces the minimal positive value of D inside ( ! v ; !
x ) .
Formally, dh ( ! v ) ≜ arg min !
x ; ! x ̸
= !
v D inside ( ! v ; !
x ) (cid:21)
= min f !
x jD inside ( ! v ; !
x ) (cid:21)
^ !
x ̸
= !
v g .
Similarly, !
u disconnecting from !
w can be measured by the result of subtracting the distance between their center vectors from the sum of their radii.
Formally, we deﬁne D disc ( !
u ; !
w ) ≜ r w + r u (cid:0)∥ ⃗A u (cid:0) ⃗A w ∥ , That is, !
u disconnects from !
w , if and only if D disc ( !
u ; !
w ) (cid:20) 0 .
2. S IMILARITY M EASUREMENT
Similarity is normally measured by the cosine value of two vectors, e.g., Mikolov et al.
For N -balls, the similarity between two balls can be approximated by the cosine value of their central vectors.
Formally, given two N -balls !
u
= B ( ⃗A u ; r u ) and !
w = B ( ⃗A w ; r w ) , their cosine similarity
[Figure]
can be deﬁned as cos( ⃗A u ; ⃗A w ) .
One weakness of the method is that we do not know the boundary of the lowest cos value below which two word senses are not similar.
Using category information, we can deﬁne that two word senses are not similar, if they have different direct hypernyms.
Formally,
[Equation]
otherwise
C ONSTRUCTING N - BALL E MBEDDINGS
N -ball embeddings encode two types of information: (1) word embeddings and (2) tree structures of hyponym relations among word senses.
A word can have several word senses.
We need to create a unique vector to describe the location of a word sense in hypernym trees.
We introduce a virtual root ( *root* ) to be the parent of all tree roots, ﬁx word senses in alphabetic order in each layer, and number each word sense based on the ﬁxed order.
A fragment tree structure of word senses is illustrated in Figure 2(a).
The path of a word sense to *root* can be uniquely identiﬁed by the numbers along the path.
For example, the path from *root* to ﬂower.n.03 is [ entity.n.01, abstraction.n.06, measure.n.02, ﬂower.n.03 ] 1 , which can be uniquely named as [1,1,1,1]; We call this vector location code of ﬂower.n.03 .
The location code of the direct hypernym of ﬂower.n.03 is called Parent Location Code (PLC) .
PLC of ﬂower.n.03 is [1,1,1], PLC of whole.n.02 is [1,2].
As a word and its hypernym may not co-occur in the same context, their co-occurrence relations can be weak, and the cosine similarity of their word embeddings could even be less than zero.
For example, in GloVe embeddings, cos ( ice cream, dessert ) = (cid:0) 0 : 1998 , cos ( tuberose, plant ) = (cid:0) 0 : 2191 .
It follows that the hypernym ball must contain the origin point of the n -dimensional space, if this hypernym ball contains its hyponym, as illustrated in Figure 3 (a).
When this happens to two semantically unrelated hypernyms, N -balls of the two unrelated hypernyms shall partially overlap, as they both contain the original point.
For example, the ball of desert shall partially overlap with the ball of plant .
This violates our ﬁrst criterion.
To avoid such cases, we require that no N -ball shall contain the origin O .
This can be computationally achieved by adding dimensions and realized by introducing a spatial extension code , a constant non-zero vector, as illustrated in Figure 2(b).
To intuitively understand this, imagine that you stand in the middle of two objects A and B, and cannot see both of them.
To see both of them without turning the head or the eyes, you would have to walk several steps away so that the angle between A and B is less than some degree, as illustrated in
Figure 3(c).
[Figure]
3. T HE S TRUCTURE OF THE C ENTRAL V ECTOR
Following Zeng et al.
and Han et al.
(2016), we structure the central vector of an N -ball by concatenating three vectors: (1) the pre-trained word-embedding, (2) the PLC (if the code is shorter than the max length, we append 0 s till it reaches the ﬁxed length), (3) the spatial extension code.
3. Z ERO E NERGY L OSS FOR N -B ALL C ONSTRUCTION U SING G EOMETRIC A PPROACH
Our ﬁrst criterion is to precisely encode subordinate relations among categories into inclusion relations among N -balls.
This is a considerable challenge, as the widely adopted back-propagation training process ( Rumelhart et al., 1988) quickly reaches non-zero local minima and terminates 2 .
The problem is that when the location or the size of an N -ball is updated to improve its relation with a second ball, its relation to a third ball will deteriorate very easily.
We propose the classic depth-ﬁrst recursion process, listed in Algorithm 1 , to traverse the category tree and update sizes and locations of N -balls using three geometric transformations as follows.
Homothetic transformation (H-tran) which keeps the direction of the central vector and enlarges lengths of ⃗A and r with the same rate k .
H ( B ( ⃗A; r ) ; k ) ≜ B ( ⃗A ′ ; r ′ ) , satisfying (1) j ⃗A j j ⃗A ′
j = rr ′ =
k > 0 , and (2) ⃗A j ⃗A j = ⃗A ′ j ⃗A ′ j ;
Shift transformation (S-tran) which keeps the length of the radius r and adds a new vector ⃗s to ⃗A .
S ( B ( ⃗A; r ) ;⃗s ) ≜ B ( ⃗A ′ ; r ) , satisfying ⃗A ′
= ⃗A
+ ⃗s ;
Rotation transformation (R-tran) which keeps the length of the radius r and rotates angle (cid:11) of ⃗A inside the plane spanned by the i -th and the j -th dimensions of ⃗A .
R ( B ( ⃗A; r ) ; (cid:11); i; j ) ≜
B ( ⃗A ′ ; r ) , such that ⃗A k
= ⃗A ′ k
( k ̸
= i; j ), ⃗A ′
i
=
⃗A
i cos (cid:11) + ⃗A
j sin (cid:11) and ⃗A ′
j = ⃗A
j cos (cid:11) (cid:0) ⃗A
i sin (cid:11) .
To satisfy our second criterion, we do not choose rotation dimensions among the dimensions of pre-trained word embeddings.
Rather, to prevent the deterioration of already improved relations, we use the principle of family action: if a transformation is applied for one ball, the same transformation will be applied to all its child balls .
Among the three transformations, only H-tran preserves inclusion and disconnectedness relations among the family of N -balls as illustrated in Figure 3(d), therefore H-tran has the priority to be used.
In the process of adjusting sibling N -balls to be disconnected from each other, we apply H-tran obeying the principle of family action.
When an N -ball is too close to the origin of the space, a
S-tran plus R-tran will be applied which may change the pre-trained word embeddings.
Following the depth-ﬁrst procedure, a parent ball is constructed after all its child balls.
Given a child
N -ball B ( ⃗B; r 2 ) , a candidate parent ball B ( ⃗A; r 1 ) is constructed as the minimal cover of B ( ⃗B; r 2 ) , illustrated in Figure 3(b).
The ﬁnal parent ball B ( ⃗P; r p ) is the minimal ball which covers these already constructed candidate parent balls B ( ⃗P i ; r p i ) .
input : a tree pointed by root ; each node stores a word sense word embedding output : the N -ball embedding of node children (cid:0) get all children of ( root ) if number of ( children ) > 0 then foreach child 2 children do // depth first training one family ( child ) endif number of ( children ) > 1 then // adjusting siblings to be disconnected from each other adjust to be disconnected ( children ) end // create parent ball for all children root =
create parent ball of ( children ) else initialize ball ( root ) end
E XPERIMENTS AND E VALUATIONS 4. E XPERIMENT 1: C ONSTRUCT N - BALL E MBEDDINGS FOR L ARGE -S CALE D ATA S ETS
We use the GloVe word embeddings of Pennington et al.
(2014) as the pre-trained vector embedding of words and extract trees of hyponym relations among word senses from Word-Net 3.0 of Miller (1995).
The data set has 54 ; 310 word senses and 291 trees, among which root
entity.n.01 is the largest tree with 43 ; 669 word senses.
Source code and input data sets are publically available at https://github.com/GnodIsNait/nball4tree.git .
We proved that all subordinate relations in the category tree are preserved in N -ball embeddings.
That is, zero energy cost is achieved by utilizing the proposed geometric approach.
Therefore, the ﬁrst criterion is satisﬁed.
4. E XPERIMENT 2: T EST I F P RE -T RAINED W ORD -E MBEDDINGS A RE W ELL P RESERVED
We apply homothetic, shifting, and rotating transformations in the construction/adjusting process of N -ball embeddings.
Shifting and rotating transformations may change pre-trained word embeddings.
The aim of this experiment is to examine the effect of the geometric process on pre-trained word embeddings, and check whether the second criterion is satisﬁed.
Method 1 We examine the standard deviation (std) of the pre-trained word embedding in N -ball embeddings of its word senses.
The less their std, the better they are preserved.
The N -ball embeddings have 32,503 word stems.
For each word stem, we extract the word embedding parts from N -ball embeddings, normalize them, minus pre-train word embeddings, and compute standard deviation.
The maximum std is 0.7666.
There are 417 stds greater than 0.2, 6 stds in the range of (0 : 1 ; 0 : 2] , 9 stds in the range of (10 (cid:0) 12 ; 0 : 1] , 9699 stds in the range of (0 ; 10 (cid:0) 12 ] , 22,372 stds equals 0 .
With this statistics we conclude that only a tiny fraction ( 1 : 3% ) of pre-trained word embeddings have a small change (std 2 (0 : 1 ; 0 : 7666] ).
Method
The quality of word embeddings can be evaluated by computing the consistency (Spearman’s co-relation) of similarities between human-judged word relations and vector-based word similarity relations.
The standard data sets in the literature are the WordSim353 data set (Finkelstein et al., 2001) which consists of 353 pairs of words, each pair associated with a human-judged value about the co-relation between two words, and Stanford’s Contextual Word Similarities (SCWS) data set (Huang et al., 2012) which contains 2003 word pairs, each with 10 human judgments on the similarity.
Given a word w , we extract the word embedding part from its word senses’ N -ball embeddings and use the average value as the word-embedding of w in the experiment.
Unfortunately, both data sets cannot be used directly within our experimental setting as some words do not appear in the ball-embedding due to (1) words whose word senses have neither hypernym, nor hyponym in Word-Net 3.0, e.g. holy ; (2) words whose word senses have different word stems, e.g. laboratory, midday, graveyard, percent, zoo, FBI, ... ; (3) words which have no word senses, e.g. Maradona ; (4) words whose word senses use their basic form as word stems, e.g. clothes, troops, earning, ﬁghting, children .
After removing all the missing words, we have 318 paired words from WordSim353 and 1719 pairs from SCWS dataset for the evaluation.
We get exactly the same Spearman’s co-relation values in all 11 testing cases: the Spearman’s corelation on WordSim318 is 76.08%; each test on Spearman’s co-relations using SCWS1719 is also the same.
We conclude that N -ball embeddings are a “loyal” extension to word embeddings, therefore, the second criterion is satisﬁed.
4. E XPERIMENT 3: Q UALITATIVE E VALUATION OF N - BALL E MBEDDINGS
Following Levy & Goldberg (2014), we do qualitative evaluations.
We manually inspect nearest neighbors and compare results with pre-trained GloVe embeddings.
A sample is listed in Table 12 with interesting observations as follows.
Precise neighborhoods N -ball embeddings precisely separate word senses of a polysemy.
For example, the nearest neighbors of berlin.n.01 are all cities, the nearest neighbors of berlin.n.02 are all names as listed in Table 1.
Typed cosine similarity function better than the normal cosine function Sim 0 enriched by category information produces much better neighborhood word senses than the normal cosine measurement.
For example, the top-5 nearest neighbors of beijing in GloVe using normal cosine measurement are: china, taiwan, seoul, taipei, chinese , among which only seoul and taipei are cities.
The top-10 nearest neighbors of berlin in GloVe using normal cosine measurement are: vienna, warsaw, munich, prague, germany, moscow, hamburg, bonn, copenhagen, cologne , among which germany is a country.
A worse problem is that neighbors of the word sense berlin.n.02 as the family name do not appear.
Without structural constraints, word embeddings are severely biased by a training corpus.
Category information contributes to the sparse data problem Due to sparse data, some words with similar meanings have negative cosine similarity value.
For example, tiger as a ﬁerce or audacious person (tiger.n.01) and linguist as a specialist in linguistics (linguist.n.02) seldom appear in the same context, leading to (cid:0) 0 : 1 cosine similarity value using GloVe word embeddings.
However, they are hyponyms of person.n.01, using this constraint our geometrical process transforms the Nballs of tiger.n.01 and linguist.n.02 inside the N -ball of person.n.01, leading to high similarity value measured by the typed cosine function.
Upper category identiﬁcation Using N -ball embeddings, we can ﬁnd upper-categories of a word sense.
Given word sense ws , we collect all those cat s satisfying D inside ( ws; cat ) > 0 .
These cat s shall be upper-categories of ws .
If we sort them in increasing order and mark the ﬁrst cat with + 1 , the second with + 2 ..., the cat marked with + 1 is the direct upper-category of ws , the cat marked with + 2 is the direct upper-category of the cat with + 1 ..., as listed in Table 2.
[Table]
[Table]
E XPERIMENT 4: M EMBERSHIP V ALIDATION
The fourth experiment is to validate the category of an unknown word, with the aim to demonstrate the predictive power of the embedding approach ( Baroni et al., 2014).
We describe the task as follows:
Given pre-trained word embeddings E N with vocabulary size N , and a tree structure of hyponym relations T K on vocabulary W K , ( K < N ) .
Given w x = 2 W K and c 2 W K , we need to decide whether w x 2 c .
For example, when we read mwanza is also experiencing , we may guess mwanza a person, if we continue to read mwanza is also experiencing major infrastructural development , we would say mwanza is a city.
If mwanza is not in the current taxonomy of hypernym structure, we only have its word embedding from the text.
Should mwanza be a city, or a person?
Dataset From 54,310 word senses, we randomly selected 1,000 word senses of nouns and verbs as target categories, with the condition that each of them has at least 10 direct hyponyms; For each target category, we randomly select p % ( p 2 [5 ; 10 ; 20 ; 30 ; 40 ; 50 ; 60 ; 70 ; 80 ; 90] ) from its direct hyponyms as training data.
The test data sets are generated from three sources: (1) the rest 1 (cid:0) p % from the direct hyponyms as true values, (2) randomly choose 1,000 false values from W K ; (3) 1,000 words from W N which do not exist in W K .
In total, we created 118,938 hyponymy relations in the training set, and 17,975,042 hyponymy relations in the testing set.
Method We develop an N -ball solution to solve the membership validation task as follows: Suppose c has a hypernym path [ c; h 1 ; h 2 ; : : : ; h m ] and has several known members (direct hyponyms) t 1 ; : : : ; t s .
For example, city.n.01 has a hypernym path [ city.n.01, municipality.n.01, urban area.n.01, ..., entity.n.01 ] and a number of known members oxford.n.01, banff.n.01, chicago.n.01 .
We construct N -ball embeddings for this small tree with the stem [ c; h 1 ; h 2 ; : : : ; h m ] and leaves t 1 ; : : : ; t s , and record the history of the geometric transformations of each ball.
Suppose that w x be a member of c , we initialize the N -ball of w x using the same parameter as the N -ball of t 1 , and apply the recorded history of the geometric transformations of t 1 ’s N -ball for w x ’s N -ball.
If the ﬁnal N -ball of w x is located inside the N -ball of c , we will decide that w x is the member of c , otherwise not.
This method can be explained in terms of Support Vector Machines (Shawe-
[Figure]
Taylor & Cristianini, 2004) as follows: the boundary of c ’s N -ball is supported by all N -balls of its known members.
If the unknown word were a member of c , its N -ball shall have the same PLC as the member of c , and according to the principle of family action, it shall have the same geometric transformation, and will contribute one candidate parent ball.
We can introduce a ratio (cid:13) (cid:21) 1 to zoom-out the boundary of c ’s N -ball.
Evaluation and Analysis In the experiment, results show that the N -ball method is very precise and robust as shown in Figure 4(a): the precision is always 100% , even only 5% from all members is selected as training set.
The method is quite robust: If we select 5% as training set, the recall reaches 76 : 8% ; if we select 50% as training set, the recall reaches 96 : 7% .
Theoretically, the Nball method can not guarantee 100% recall as shown in Figure 4(b).
If p < 70% , the population standard deviation (pstdev) decreases with the increasing percentage of selected training data.
When p > 70% , there is a slight increase of pstdev.
The reason is that in the experiment setting, if more than 80% of the children are selected, it can happen that only one unknown member is left for validating.
If this single member is excluded outside of the category’s N -ball, the recall drops to 0 , which increases pstdev.
The experiment result can be downloaded at https://figshare.
com/articles/membership_validation_results/7571297 .
In the literature of representational learning, margin-based score functions are the state-of-the-art approach (Gutmann & Hyv ¨ arinen, 2012):
The score of a positive sample shall be larger than the score of a negative sample plus a margin.
This can be understood as a simple use of categorization – no chained subordinate relations, no clear membership relations of negative samples, no requirement on zero energy loss.
However, when category information is fully and strictly used, the precision will increase signiﬁcantly, and surprisingly reach 100% in this experiment.
R ELATED W ORK
Lenci & Benotto (2012) explored the possibility of identifying hypernyms in distributional semantic model; Santus et al.
(2014) presented an entropy-based model to identify hypernyms in an unsupervised manner; Kruszewski et al.
(2015) induced mappings from words/sentences embeddings into
Boolean structure, with the aim to narrow the gap between co-occurrence based embeddings and logic-based structures.
There are some works on word embedding and knowledge graph embedding using regions to represent words or entities.
Athiwaratkun & Wilson (2017) used multi-modal
Gaussian distribution to represent words; He et al.
embedded entities using Gaussian distributions; Xiao et al.
(2016) used manifolds to represent entities and relations; Nickel & Kiela (2017) used Poincar´e balls to embed tree structures.
Mirzazadeh et al.
(2015) share certain common interest with the presented work in embedding constraints.
However, in none of these works, structural imposition at zero-energy cost is targeted.
C ONCLUSION AND O N - GOING W ORK
We proposed a novel geometric method to precisely impose external tree-structured category information onto word embeddings, resulting in region-based ( N -ball embeddings) word sense embeddings.
They can be viewed as Venn diagrams (Venn, 1880) of the tree structure, if zero energy cost is achieved.
Our N -ball method has demonstrated great performance in validating the category of unknown words, the reason for this being under further investigation.
Our on-going work also includes multi-lingual region-based knowledge graph embedding where multiple relations on directed acyclic graphs need to be considered.
N -balls carry both vector information from deep learning and region information from symbolic structures.
Therefore, N -balls establish a harmony between Connectionism and Symbolicism, as discussed by Marcus (2003), and thus may serve as a novel building block for the commonsense representation and reasoning in Artiﬁcial Intelligence.
N -balls, in particular, contribute a new topic to Qualitative Spatial Reasoning (QSR) that dates back to Whitehead (1929).
A CKNOWLEDGMENTS
Partial ﬁnancial support from NSFC under Grant No. 61472177 and 61661146007, and from BMBF (Bundesministerium f¨ur Bildung und Forschung) of Germany under grant number 01/S17064 and 01/S18038C are greatly acknowledged.
We are greatly indebted to Dagstuhl Seminar {+
+}
CrossLingualCross-Media Content Linking: Annotations and Joint Representations for fruitful and interesting discussions.
We are also thankful for critical comments from three anonymous reviewers, and for the OpenReview policy.