Assessing Wordnets with W ord N et Embeddings (2019)

Abstract: An effective conversion method was proposed in the literature to obtain a lexical semantic space from a lexical semantic graph, thus permitting to obtain WordNet embeddings from WordNets. In this paper, we propose the exploitation of this conversion methodology as the basis for the comparative assessment of WordNets: given two WordNets, their relative quality in terms of capturing the lexical semantics of a given language, can be assessed by (i) converting each WordNet into the corresponding semantic space (i.e. into WordNet embeddings), (ii) evaluating the resulting WordNet embeddings under the typical semantic similarity prediction task used to evaluate word embeddings in general; and (iii) comparing the performance in that task of the two word embeddings, extracted from the two WordNets. A better performance in that evaluation task results from the word embeddings that are better at capturing the semantic similarity of words, which, in turn, result from the WordNet that is of higher quality at capturing the semantics of words.

## introduction lexical semantics studies the semantic properties of lexical units, and is often defined as the study of word meaning. given its importance, the computational representation of lexical meaning is a core challenge in natural language processing (nlp). since the meaning of a word is strongly related to the meaning of other words, the relations between words are a key ingredient for the representation of their meaning. there have been different types of representations proposed for lexical semantics, which, in general, can be viewed as pertaining to one of three main family of representations, namely semantic networks @xcite , feature-based models @xcite , and semantic spaces @xcite . semantic networks are a type of approach for lexical semantics that is based on graphs. in a nutshell, a lexical unit, typically a word, is recorded as a node in a graph while the semantic relations among words, such as hyponymy or synonymy, etc., are recorded as labeled edges among the nodes of the graph. one of the most popular semantic networks is wordnet @xcite . it stands out as being a lexical semantics network based on non trivial linguistic intuitions of human experts. feature-based models representing lexical semantics, in turn, resort to a hash table that stores the lexical units as keys, and the semantically related units as the respective values. small world of words @xcite is an example of such a model. in its development, the semantic features (related words) of a lexical entry can be obtained straightforwardly from laypersons by using the lexical entry as a cue to evoke possible words associated to it. finally, in semantic spaces, the meaning of a lexical unit is represented as a vector in a high dimension space -also known as word embedding -, typically obtained on the basis of the frequency of its co-occurrence with other lexical units, resorting to a large collection of documents. word2vec @xcite is an example of a method to obtain semantic spaces. bridging between these different types of lexical meaning representations is instrumental for a wider use of all the existing lexical semantics resources. unifying this knowledge in one lexical semantic representation would carry an immediate impact across a range of nlp tasks. an existing form of (partial) bridging is ob-tained with the conversion of one type of representation to another as in @xcite , with the wnet2vec methodology. wnet2vec permits the conversion from lexical semantic networks to lexical semantic spaces, termed as wordnet embeddings. the success of this type conversion can be measured by using the typical semantic space evaluation process. that is obtained by comparing the semantic similarity scores between the vectors of words arranged in pairs against the gold scores of semantic similarity among the words in the pairs, which were obtained from human subjects. the evaluation of the semantic similarity task based on the semantic space wnet2vec used the simlex-999 @xcite , a mainstream semantic similarity data set composed of 999 pairs of words with a correspondent similarity strength value. semantic similarity detection with wnet2vec @xcite shows an almost 20% superior result against a strong baseline, namely google's word2vec semantic space, which is trained on a very large collection of 100 billion token texts. our goal in the present paper is to propose the exploitation of this conversion methodology as the basis for the comparative assessment of word-nets: given two wordnets, for the same language, their relative quality in terms of capturing the lexical semantics of that language, can be assessed by (i) converting each wordnet into the corresponding semantic space (i.e. wordnet embeddings), (ii) evaluating the resulting embeddings in the semantic similarity prediction task; and (iii) comparing the performance in that task of the two word embeddings, extracted from the two wordnets. a better performance results from the word embeddings that better capture the semantic similarity of words, which, in turn, results from the wordnet that is of higher quality at capturing the semantics of words. in order to illustrate this proposed methodology for the comparative assessment of wordnets with a first exercise with its application, we resort to two wordnets of the same language, portuguese, developed under two distinct methodologies, namely mwn.pt -hand-crafted -and own-pt -built (semi-)automatically. the next section 2 reports on the conversion of the hand-crafted wordnet to the respective word-net embeddings and on the performance of the latter in the semantic similarity prediction task. the following section the same exercise is undertaken but now with the wordnet built (semi-)automatically. sections 4 and 5 present, respectively, the discussion of the results and the related work. the conclusions are presented in section 6. ## embeddings from hand-crafted wordnet the multiwordnet of portuguese (mwn.pt) is developed under the same methodological principles as the seminal princeton english wordnetincluding the resorting to manually validated representations. its synsets are aligned with the translationally equivalent synsets in princeton word-net. it is available from elra-european language resources association. foot_0 besides the difference in the language covered, mwn.pt differentiates to princeton wordnet by being smaller, encompassing 17k concepts/synsets (against over 120k of princeton), by encoding only synonymy and hyponymy/hypernymy (against some 25 semantics relations in princeton wordnet), and by including only nouns (against all open categories), and includes mostly the sub-ontologies of person, organization, event, location and art works. hence, it offered interesting contrasting conditions to proceed with an empirical study of the strength of the wnet2vec methodology when applied to quite different and more challenging empirical settings than the one originally resorted to in @xcite to convert the princeton wordnet into its wordnet embeddings. to obtain word embeddings, the mainstream methods have used the frequency of co-occurrence in large corpora between the target word and its neighboring words to construct the respective vector. instead of texts and the frequency of cooccurrence between words, wnet2vec resorts to lexical semantics graphs and the knowledge encoded in them, using the semantic networks as the empirical source to obtain the vectors of the corresponding semantic space. the key insight in the conversion process is that a stronger semantic affinity between two lexical units is found between nodes that are closer and have a higher number of connecting paths. in a nutshell, the wnet2vec methodology starts by creating a matrix with all of the possible semantic relations between all the words, resulting in an adjacency matrix m . then it populates each cell m ij of the matrix resorting to a wordnet, in the present experiment mwn.pt, as the semantic graph g. each cell m ij is set to 1 if and only if there is a direct edge between synsets including the two words word i and word j the cell encodes/represents. words present in the same synset have a synonym relation and thus are assigned a value of 1. if there is no edge between the two words that cell is set to 0. for all nodes not directly connected, that is connected through other nodes in between, the representation of their affinity strength is obtained by following the cumulative iteration: m n is the matrix where every two words, word i and word j , are transitively related by n edges. i represents the identity matrix and α is used as a decay factor for longer paths. the iteration converges into the matrix m g , obtained by an inverse matrix operation: after the convergence, a positive point-wise mutual information transformation (pmi+) is applied to reduce the frequency bias, followed by an l2-norm to normalize each line of m g , and finally, a principal component analysis (pca) is applied to reduce the dimension of the vectors. further details on this conversion can be found in @xcite . ## embeddings from (semi-)automatic wordnet given the lessons learned with the creation of a semantic space from the mwn.pt, in the second phase of our experiments we applied the same conversion methodology to another portuguese word-net, the openwordnet-pt (own-pt) @xcite . while mwn.pt was built manually by resorting to human experts labor, own-pt is different in that it resorts to (semi-)automatic and machine learning methodologies, and has a dimension that is over three times larger -over 54k words (against over 15k in mwn.pt) -, thus offering an interesting case for empirical study. we resorted to openwordnet-pt in the lmf format @xcite , whose last release in this format we found is from october 2018. to reuse the scripts ready for the conversion to wordnet2vec, we converted this lmf format into a princeton wndb format, having retained 54390 words. 4 this conversion was done by iterating over the lexicon and keeping track of lexical entries and their lemmas and senses, according to a unique id to differentiate between them, and also keeping a log of the semantic relations between synsets. only two semantic relations present in own-pt are not represented in the final converted network, due to them not being present in the princeton wndb format foot_4 . those two semantic relations are "exemplifies" and "is exemplified by". 6 for the sake of comparability, and given the different sizes of the two wordnets for portuguese, three experiments were performed with own-pt. in a first experiment, the 54390 words of own-pt in the wndb format were used. in a second experiment, a subset of own-pt was selected with the same number of words of mwn.pt (15886). the words that are common to both wordnets were selected. given that not all of the mwn.pt words exist in the own-pt, further words were selected from own-pt to attain the aimed dimension. remaining synsets were ordered from the ones with more outgoing edges to less outgoing edges and the words from the more connected synsets were selected until the intended dimension was reached. in previous experiments with english @xcite , it became apparent that selecting words from synsets with more outgoing edges leads to semantic spaces with better performance in the semantic similarity task. in a third experiment, a subset of equal dimension to the mwn.pt set was again extracted, this time with the simpler methodology of the second part of the selection undertaken in the second experiment: synsets were ordered from the ones with more to less outgoing edges and the words from the more connected synsets were selected until the intended dimension was reached. ## discussion the result of these experiments with mwn.pt is in line with the results of the experiments with english @xcite , even though now the experiment was with another language and with wordnets that are quite different in dimension and coverage than the english one. when evaluated in the semantic similarity task with a mainstream test dataset, the semantic space obtained from a concept-based semantic network with wnet2vec methodology outperforms the strong baseline consisting of a semantic space obtained from mainstream corpus-based methods, namely with word2vec trained with a very large collection of text, with 2.2b tokens in the present case. the results of the subsequent experiments with own-pt are also in line with those findings. even though it was built with a methodology resorting to heuristics and (semi-)automatics methods, the semantic space obtained from a second concept-based semantic network of portuguese with wnet2vec methodology also outperforms the same strong baseline. in this connection, we offer the observation that when a subset of the english wordnet was experimented using a number of synsets (25k) that is closer to our experiments reported here, a 0.45 score was obtained (against 0.53 with 60k synsets) @xcite . this may indicate that improving the existing wordnets of portuguese with a larger number of lexical units and relations may bring even better performance. additionally, the results of the experiments reported above suggest that when using the wnet2vec methodology to obtain a semantic space from a semantic graph, under comparable experimental circumstances (i.e. over 15k words that hold higher number of relations), 15% better semantic similarity performance scores are obtained with a manually crafted wordnet -0.46 with mwn.pt -than with a wordnet obtained (semi-) automatically -0.40 with own-pt. this is in line with what is expected given the noise introduced by the (semi-)automatic methods used in the construction of wordnets. what is new with respect to the methodology proposed here is that there is now a quantitative way to assess the difference between wordnets in what concerns their different quality at capturing the semantics of words. ## related work a proposal for the conversion from the princeton wordnet to a semantic space different from the one used here can be found in @xcite . that is different in that in this other proposal the conversion from semantic graph to semantic space is not direct. first, a synthetic corpus is generated by a random walk in the word-net. then on the basis of that artificial text, common corpus-based techniques are used to obtain the word embeddings. in (gonc ¸alo oliveira, 2018), in turn, another approach was used to obtain a semantic space from portuguese semantic networks also resorting to a random walk but, differently from the approach mentioned above, via direct conversion. instead of using a concept-based semantic network (word-net) as in our study reported here, semantic networks based on words only (no synsets) were used and converted to semantic spaces. also, a different method than ours was used, a random walk with 30 iterations. although these differences render the results not comparable, it may be still interesting to draft some observations with the necessary caution and grains of salt. the best accuracy score reported in (gonc ¸alo oliveira, 2018) with lx-simlex-999 is 0.61 in terms of spearman's coefficient. this score is obtained with a network with more than 200k words, more than ten times larger than the network used in our study reported here, with approximately 17k synsets. the system that, in turn, is reported there as having a performance score of 0.45, in line with the score of 0.46 we found here for a 17k network, was trained over a network five times larger than the one used here. this may be another sign of the higher quality of (hand-crafted) wordnets at recording the lexical semantics of words. in future work, it will be interesting to undertake further experiments to try to understand to what extent the strength of the findings reported here are due to intrinsic strength of the conversion algorithm adopted here or to the intrinsic quality of the semantic networks used, or just of a bit of both factors and of their combination. ## conclusions in a previous study in the literature @xcite , a conversion method (wnet2vec) was explored to obtain a semantic space (aka word embeddings) from a semantic graph, by applying it to the english princeton wordnet. the wordnet embeddings wnet2vec thus generated, on the basis of 60k synsets, outperforms a strong baseline which is a corpus-based word embedding word2vec, based on 100b words. it outperforms in the semantic similarity detection task over the mainstream simlex-999 test dataset, with an accuracy score of 0.50 against 0.44 in terms of spearman's coefficient, for wnet2vec and word2vec respectively . in the present paper, we experimented with this conversion method under further empirical conditions. we applied it over a wordnet manually built under the same construction principles as princeton wordnet (over 120k synsets) only that it is more than seven times smaller (17k synsets) and is for another language, namely portuguese. we experimented also with another wordnet for portuguese but constructed under an alternative approach that resorts to (semi-) automatic methods. the wordnet embeddings obtained were tested under the semantic similarity task over the portuguese translation of the mainstream simlex-999 test dataset @xcite . the baseline was the word embeddings obtained with the corpus-based word2vec procedure over a 2.2b words corpus of portuguese @xcite . the results obtained are in line with earlier findings. the wnet2vec conversion method to obtain a semantic space from a semantic network is very effective. the semantic similarity detectors based on word embeddings wnet2vec -obtained from any of the wordnets experimented with in this paper -outperform the strong baseline detector based on the corpus-based word embeddings word2vec. the semantic similarity detector based on the manually built wordnet, in turn, outperformed the detector based on the wordnet that was built (semi-) automatically. these results suggest that, when using the wnet2vec methodology to obtain a semantic space from a semantic graph, under comparable experimental circumstances, better semantic similarity performance scores are obtained with a manually crafted wordnet rather than with a wordnet obtained (semi-) automatically. this is as expected given the noise introduced by automatic methods. what is new with respect to the assessment methodology proposed here is that it offers a new quantitative way to evaluate the difference between wordnets in what concerns their different quality at capturing the meaning of words.