CoKE : Extending Contextualized Knowledge Embeddings for Word Sense Induction
Abstract
Word Embeddings are able to capture lexico-semantic information but remainﬂawed in their inability to assign unique representations to different senses ofa polysemous words.
They also fail to include information from well curatedsemantic lexicons and dictionaries.
Previous approaches that integrate polysemyand knowledge bases fall distinctly under two categories - a)retroﬁtting vectors toontologies or b)learning from sense tagged corpora.
While embeddings learnedfrom these methods are superior in understanding contextual similarity, they areoutperformed by single prototype word vectors on several relatedness tasks.
In thiswork, we introduce a new approach that can induce polysemy to any pre-trainedembedding space by jointly grounding contextualized sense representations andword embeddings to a knowledge base.
Along with word sense induction, theresulting representations reduces the effect of vocabulary bias that arises in naturallanguage corpora and in turn embedding spaces.
By grounding them to knowledgebases they are able to learn multi-word representations and are also interpretable.
We evaluate our vectors across 12 datasets on several similarity and relatednesstasks along with two extrinsic tasks ,we also evaluate against other transfer learningmethods and ﬁnd that our approach consistently outperforms current state of theart.
Related Work and Introduction
Distributed representations of words (Mikolov et al., 2013b) have proven to be successful in addressingmultiple drawbacks of symbolic representations which treats words as atomic units of meaning.
Bygrouping similar words and capturing analogical and lexical relationships, they are a popular choicein several downstream NLP applications.
While these embeddings capture meaningful relationships they come with their own set of drawbacks.
For instance, complete reliance on natural language corpora ampliﬁes existing societal and vocabularybias that are inherent in datasets.
A study by Bolukbasi et al., 2016 discussed societal biases in theform of gender stereotypes present in these VSMs.
Vocabulary bias is caused by words not seen inthe training corpora and also extends to bias in word usage where some words, often morphologicallycomplex words, are used less frequently than other words or phrases with the same meaning.
Thisalso becomes evident in the relatively lower performance of word embeddings on the rare wordsimilarity task (Luong et al., 2013b).
A recent approach by (Bojanowski et al., 2016a) proposesusing character n-gram representations to address the problem of out-of-vocabulary and rare words.(Faruqui et al., 2014) also proposed retroﬁtting vectors to an ontology.
However, these methods don’taccount for polysemy.
Polysemy is an important feature of language which causes words to have different meanings basedon the context in which they occur.
For instance, the word ‘bank’ can mean ‘ﬁnancial institution’ or
‘land on either side of a river’.
A well known drawback with word embeddings is the assignment of asingle vector representation to a word type, irrespective of polysemy.
A large body of work has goneinto developing word sense disambiguation systems to identify the correct sense of a word based onit’s context.
The availability of such disambiguation systems coupled with the growing reliance of
NLP systems on distributional semantics has led to an increasing interest in obtaining powerful senserepresentations.
Some of the previous work that has gone into learning sense representations includes unsupervisedlearning techniques to cluster contexts and learn multi prototype vectors(Reisinger and Mooney(2010) , Huang et al.
(2012) and Wu and Giles (2015)).
However, a common drawback with thecluster based models is the difﬁculty in deciding the number of clusters apriori.
( Neelakantan et al.(2015) , Tian et al.
,Cheng and Kartsaklis (2015) also learn multiple word embeddings bymodifying the Skip-Gram model.
These approaches yield to sense representations that are limited interms of interpretability which makes it challenging to include in downstream tasks.
As a remedy to limitation in sense tagged corpora, Jauhar et al. (2015) and Rothe and Schütze (2015)explored grounding word embeddings to ontologies to obtain sense representations.
As a result, thesetechniques drastically improved performance on several similarity tasks but an observed pattern isthat this leads to compromised performance on word relatedness tasks(Faruqui et al.
(2014), Jauharet al.
We suspect this is a result of directly modifying word embedding spaces based onontology structure.
In this work, we present a novel approach that uses knowledge bases and sense representations todirectly induce polysemy to any predeﬁned word embedding space.
We show how our approachallows the integration of ontological information and leads to improvements in both word similarityand relatedness tasks.
The advantages of this are plenty, it allows the integration of knowledge intoembedding spaces and can readily induce polysemy in them.
We thus rely on a) Sense tagged corporato obtain contextualized sense representations.
The objective of which is to capture sense relationsand interactions in naturally occurring corpora.
The sense representations are interpretable and havelexical mappings to a knowledge base.
We use them to induce polysemy in word embedding spaces.b)
Pretrained word embeddings to capture many useful lexical relationships that are inherent in themon account of being trained on large amounts of data.
These relationships are not effectively capturedby sense representations due to the limited size of sense tagged corpora they are trained on.
c)
Lastly,in order to account for the vocabulary bias which causes similar meaning words to be farther apart inembedding spaces as a result of bias in co-occurrence statistics found in corpora, we use a knowledgebase to jointly ground word and sense representations.
We thus obtain unique multiple word sense representations that show superior performance insimilarity, relatedness and extrinsic tasks.
They also show performance beneﬁts when used withtransfer learning methods like CoVE (McCann et al.
and ELMo (Peters et al.
Methodology2.1 Lexicon Building
We use WordNet(Miller (1995)) as our primary knowledge base source.
However, in order to obtainrepresentations that cater to both similarity and relatedness, we modify the synset nodes in WordNet.
A synset in WordNet is represented by a set of synonyms.
We observe that these synonym sets includewords of same meaning without differentiating between their syntactic forms.
For instance, thesynset operate.v.01 , deﬁned as ’direct or control; projects, businesses’ has both run and running in itssynonym sets.
In practice though, each syntactic form of a word has different semantic distributions.
For instance, in this case run is found to most likely occur with words such as lead and head ascompared to it’s alternate form running which is more likely to appear with words such as managing ,administrating , leading .
To account for this, we extend WordNet nodes to also include the syntacticform information and call a synset, syntactic form pair "sense-forms".
The extended WordNet nodesis depicted in Figure 1.
2.2 Sense-Form Embeddings
We use a concatenation of two sense tagged corpora, SemCor(Ciaramita and Altun (2006)) and
OMSTI(Taghipour and Ng (2015)) to obtain sense representations.
To better capture different relational and interaction information between senses, we pre-process the corpora by replacing everyword and synset pair with it’s respective sense tag and syntactic form.
We then use the Word2Vectoolkit(Mikolov et al.
(2013b))
with the Skip Gram objective function with Negative Sampling toobtain our "sense-form" representations.
2.3 Thesaurus Inclusion
While WordNet provides valuable structural information, the stringent structure leads to a ratherlimited synonym set.
We thus make use of an external thesaurus 1 to augment each synonym set.
2.4 Word Sense Induction
Now that we have sense and word representations, we ground the two in WordNet and obtain multiword sense representations.
Thus, for given a word with word embedding v w and sense-form withsense-form embedding v s , we obtain unique word speciﬁc sense representations v s w as follows.
[Equation]
Arora et al.
(2018) infact observe that a word vector lies in the linear superposition of it’s senses as :
[Equation]
The coefﬁcients contributing to each sense of a word is the ratio of frequency of the word senses.
Thus given a word with 2 senses, the coefﬁcients follow a probability distribution as 1 − clog ( r )where r is the ratio of frequencies of the two senses for some constant c, where r < 10 1 /c .
We thusmimic this property and ground our sense representations to WordNet to obtain :
[Equation]
To start of, we ﬁrst rank all the sense-forms a given word is found in decreasing order of likelihood.
WordNet ranks synsets for a word based on the frequency that a word has occurred with respect to asynset sense.
We thus exploit this property of WordNet to rank sense forms.
Since we use an externalthesaurus to augment synonym sets, a word typically has senseform not found in the original WordNetlist.
To account for this, we use the ontology structure to ﬁnd the closest synset from the original list to the senseform and use it’s rank.
By conditioning rank on structure, this leads to ontology groundedsense forms.
Once we obtain these word speciﬁc sense representations, we transfer this to pre existingembedding spaces to induce polysemy.
To best combine information from the two vectors spaces, weexperiment with two vector manipulations, concatenation and modiﬁcation.1)
Concatenation
: We simply concatenate the word speciﬁc senseform vectors to the respectiveword’s pretrained word embedding
[Equation]
2) Modiﬁcation :
[Equation]
The combination is thus done as a post processing step after obtaining word speciﬁc sense representations.
Experimental Setup
In this section, we describe the experiments done to evaluate our multi sense word embeddings.
Weuse an array of existing word similarity and relatedness datasets to conduct intrinsic evaluation and 4datasets across 2 tasks for extrinsic evaluation.
3.1 Intrinsic Evaluation
3.1.1 Word Representations
We pick three different embeddings(Pennington et al.
(2014) , Bojanowski et al.
(2016b) , Mikolovet al.
(2013a))
of 300 dimension to run our experiments
3.1.2 Similarity Measures
Given a pair of words w with M senses and w (cid:48) with N senses, we follow Reisinger and Mooney(2010) and use two metrics for computing similarity scores without using context.
[Equation]
AvgSim computes word similarity as the average similarity between all pairs of sense vectors.
Whereas MaxSim computes the maximum similarity over all pairwise sense vector similarities.
3.1.3 Word Similarity
We evaluate our embeddings on several standard word similarity datasets namely, SimLex (Hill et al.(2015)), WordSim-353(Gabrilovich and Markovitch), WS-S, MC-30(Miller and Charles (1991)) ,
RG-65 (Rubenstein and Goodenough (1965)), YP-130 (Yang and Powers (2006)),SimVerb(Gerz et al.(2016))and Rare Word(RW) similarity (Luong et al. (2013a)).
Each dataset contains a list of word pairs with a human score of how related or similar the twowords are.
We calculate the Spearman correlation between the labels and the scores generated by ourmethod.
We evaluate with another baseline using random vectors as senseform vectors along withword embeddings.
MaxSim shows bigger improvements on all datasets, except for WordSim -353and
Rare Word and we thus use AvgSim for it.
The results are outlined in Table 1.
3.1.4 Word Relatedness
Integration of our vectors also shows improvements in word relatedness tasks.
As our benchmark, we evaluate on WS-R (relatedness) , MTurk(771) (Halawi et al.
, MEN(Bruni et al.
Vector
WS-S
RGRW
SLYP
MC
SVSkip-Gram 76.
+RANDOM 52.
+CoKE(C)
+CoKE(M) 73.
Glove 79.
+RANDOM 52.
+CoKE(C)
+CoKE(M) 81.
FastText 77.
+RANDOM 60.
+CoKE(C) 76.
+CoKE(M) 76.
Vector
WS
[Table]
MEN
MTSGS
SG 58.
+RANDOM 44.
+CoKE(C) 58.
[Table]
+CoKE(M)
[Table]
+RANDOM
[Table]
+CoKE(C)
[Table]
+CoKE(M)
[Table] [Table]
(2012)),SGS130 ( Szumlanski et al. (2013)), this dataset also includes phrases
We evaluate theperformance of our method against standard pretrained word embedding using spearman correlation.
We depict the performance of MaxSim on MT(771) and SGS130 and AvgSim for WS-R and MEN.
The results are outlined in Table 2.
3.1.5 Word Similarity for Polysemous Words
We use the SCWS dataset introduced by Luong et al.
(2013a), where word pairs are chosen to havevariations in meanings for polysemous and homonymous words.
We compare our method with otherstate of the art multiprototype models .We
ﬁnd that our model performs competitively with previousmodels.
We use the SkipGram(SG) word embedding with our method to allow for fair comparison with the previous methods listed which uses SkipGram based retroﬁtting to WordNet.
The spearmancorrelation between the labels and scores are seen in Table 3.
3.2 Extrinsic Evaluation
3.2.1 Datasets
For sentiment analysis we use the Stanford Sentiment Treebank dataset(Socher et al.
We trainseperately and test on the Binary Version(SST-2) as well as the ﬁve class version(SST-5).
For questionclassiﬁcation, we evaluate performance on the TREC(Voorhees (2001)) question classiﬁcation datasetwhich consists of open domain questions and semantic categories.
[Table]
3.2.2 Performance Comparisons
We conduct two experiments to evaluate the usefulness of our vectors.
The ﬁrst comparison is againstusing pre-trained word embeddings alone(Pennington et al.
For this experiment, we representeach word as the average of all it’s sense vectors learned by CoKE.
Recent trends have also lead to an increasing interest in transfer learning.
Both CoVE( McCann et al.(2017)) and ELMo( Peters et al.
(2018) )show signiﬁcant improvements in extrinsic tasks by inclusionwith pre-trained word embeddings.
As shown by Peters et al.
(2018), these systems inherently act asword sense disambiguation and representation systems.
They give word representations conditionedon
the context it occurs in.
However, they rely entirely on distributional semantics and could beneﬁtfrom including knowledge base information.
Thus in our second experiment, we ﬁrst train and testusing vanilla CoVE and ELMo representations.
We then use them as disambiguation systems but replace word representations with CoKE embeddings instead and show performance improvements. For disambiguating using CoKE and ELMo, we use the same approach as outlined in Peters et al.(2018).
We ﬁrst use the sense tagged corpus to compute CoVE or ELMo word representationsand then take the average representation for each sense to obtain sense representations.
Duringdisambiguation, we again use CoVE or ELMo architecture to compute representations for a givenword and then take the nearest neighbor sense from the training set to get the correct sense.
We thenuse respective CoKE embeddings on the disambiguated data.
3.2.3 Training Details
To test for performance of different embeddings on datasets, we implement a an LSTM(Hochreiterand Schmidhuber (1997)) with a hidden size of 300 and run our experiments.
Parameters wereﬁne-tuned based speciﬁc to ttask and embedding type.
Qualitative Analysis
In this section, we look at some visualizations of senses induced and show how they are easilyinterpretable.
Since the sense tags have lexical mappings to an ontology,they can be looked up toﬁnd meanings.
Moreover, the semantic distribution also plays a role in obtaining meaningful senseclusters.
We analyze two things 1) Sense clusters induced 2)How using different sense forms affectrepresentations and sense interactions in their respective word forms.
For all our analysis, we use theconcatenated version of CoKE + GLoVE embeddings and do a dimensionality reduction.
4.1 Sense Clusters
: We look at the sense clusters formed by our word speciﬁc senses embeddings for the word "rock".
The clusters for the word "rock" is depicted in Fig2, the multiple ﬁne grained embeddings clusterto form 5 basic sense meanings.
We see three distinct clusters that dominate. "
Cluster#2" can beinterpreted as all synsets that speak of rock as a "substance".
In , "Cluster#3", the synsets clustertogether to speak of rock as "music".
An interesting property can be observed comparing "Cluster#1"and "Cluster#5" .
The senses found in both of these clusters interpret "rock" as "movement/motion".
However the two distinct clusters also capture the kind of motion .
For instance ,the senses roll.v.
13and rock.v.
01 in "Cluster#5" map speciﬁcally "sideways movement".
While the senses in "Cluster#1" map to glosses "sudden movements"(convulse , lurch,move , tremble) and "back and forthmovements(wobble , rock)".
Another interesting property is depicted by "Cluster#4", although theyare more synonymous in meaning to rock as a "substance", the senses for gravel cluster very closelyto senses mapping to gloss "jerking" movements capturing deeper relations between sense.
4.2 Sense Forms
[Figure]
In this section, we analyze how different sense form representations interact for synonyms within asense.
We do so by considering the word forms "plan" and "planning" both of which are synonyms oftheir respective senseforms of "mastermind.v.01" ( Gloss : plan and direct ,a complex undertaking).
In order to observe difference in relationships of word forms, we consider only common synsets in"plan" and "planning" for visualization and observe the interactions with each other.
For the word"plan" as shown in Figure 5, we observe that the synset "mastermind" is closer in proximity to synsetsthat map to words like "plan" , "sketch" , "prepare".
In contrast the same synset in the embeddingspace for "planning" as shown in Figure 6 interacts closely with synsets that are analogous to "projectplanning" , "scheduling" , "organizing".
This shows how using different sense form representations,leads to different interactions among the same group of synsets for different words and allows forbetter interaction that is more unique to each word.
Conclusion
In our work, we explore the possibility of obtaining multiword prototypes from embedding spacesby directly transfer learning from an ontology.
The prototypes allow ease of use with WSD systems, can easily be used in downstream applications since they are portable and are ﬂexible to use in awide variety of tasks.
While previous work on polysemy falls under three distinct clusters of learningmulti word sense representations, resource speciﬁc sense vectors and grounding vectors directly tolexicons.
Our work lies in the intersection.