Are Embedded Potatoes Still Vegetables? On the Limitations of W ord N et Embeddings for Lexical Semantics (2023)

Abstract: Knowledge Base Embedding (KBE) models have been widely used to encode structured information from knowledge bases, including WordNet. However, the existing literature has predominantly focused on link prediction as the evaluation task, often neglecting exploration of the models’ semantic capabilities. In this paper, we investigate the potential disconnect between the performance of KBE models of WordNet on link prediction and their ability to encode semantic information, highlighting the limitations of current evaluation protocols. Our findings reveal that some top-performing KBE models on the WN18RR benchmark exhibit subpar results on two semantic tasks and two downstream tasks. These results demonstrate the inadequacy of link prediction benchmarks for evaluating the semantic capabilities of KBE models, suggesting the need for a more targeted assessment approach.

## introduction knowledge base embedding (kbe) models aim to encode the structured information contained in knowledge bases as vectors. these models are often claimed to possess rich semantic information, and have been applied to various downstream semantic tasks, such as entity typing @xcite , entity alignment @xcite , rule mining @xcite , and conceptual clustering @xcite . if the knowledge base encodes semantic relations (hypernymy, meronymy, and so on), a kbe model should represent these relations. the assumption is that this knowledge should be helpful for more general semantic modelling. a further assumption is that evaluating on the training task, i.e. relational link prediction, is suitable as a proxy for other tasks. our aim is to test these implicit assumptions. we focus on models of the princeton wordnet @xcite . typically, kbes are evaluated on ## kbe models a knowledge base consists of triples (s, r, o), denoting the subject node s being connected by the relation r to the object node o. kbe models aim to learn an embedding for each node and relation, so that it can output higher scores for valid triplets and lower scores for invalid ones. in this paper, we compare several common models, including transe @xcite , dist-mult @xcite and murp @xcite , as well as a function-based model, funce, adapted from chen (2021). we also discuss gnn-based and distributional approaches. transe embeds both nodes and relations as real vectors, with scores defined by translation: -|v s +v r -v o | 2 . distmult also uses real vectors, but instead defines scores by componentwise multiplication: some kbe models have been designed to meet specific geometric and linguistic criteria @xcite . for the multirelational setting, @xcite introduce murp, which encodes relations as möbius transformations, with scores defined using the distance between transformed points. @xcite analyse the performance of different scoring functions on different categories of relations. their findings underscore that models should be selected for specific predictive needs: distmult performs better for some relation types, murp for others. similarly, @xcite proposed to use different scoring functions for different wordnet relations, motivated by theoretical work on functional distributional semantics @xcite . building on this, we introduce funce, which embeds a node as a real-valued function f : r @xmath0, 1], and uses a different scoring function for different categories of relation, according to linguistic criteria: hypernymbased, synonym-based, or other (see appendix a). such a function can be interpreted as the membership function for fuzzy region @xcite . in distributional semantics, @xcite has argued in favour of such representations (for a survey, see @xcite . membership functions can naturally represent tree hierarchies (like hyperbolic embeddings) but can also naturally represent multiple inheritance hierarchies. using a real-valued space makes it easier to switch scoring functions for different types of relation. more precisely, f is a hypernym of g if f (x) ≥ g(x) for all x. for simplicity, we embed nodes as logistic regression classifiers, defined by a weight vector v and bias b, so f (x) = sigmoid(v.x + b). this allows hypernymy to be calculated easily: restricting the input to the unit sphere, f is a hyper- we can use this expression as a score for the hypernymy relation. numerically, this is the same hypernymy condition as for ball embeddings @xcite , if we take v to be the centre of the ball and b to be the radius. for synonym-based relations, we use the distmult score function, and for other relations, we use the transe score function. we further experiment with recent works that apply graph neural networks (gnns) as node encoders, which can potentially leverage the structural context of each node @xcite . in particular, we consider kbgat @xcite and rgat @xcite , both of which were found to have high performance on link prediction for wn18rr. finally, some work in distributional semantics has compared models trained on corpora against models trained on linguistic resources like word-net. @xcite create sparse binary vectors, based on manually defined features, and find that they are competitive with distributional vectors on various evaluation datasets. @xcite learn synset embeddings from word-net by utilizing path-length based similarity measures. @xcite present wnet2vec, which generates word embeddings following the intuition that semantically similar nodes are connected by more paths and shorter paths. they show that the resulting embeddings outperform several distributional methods on word similarity metrics. we consider wnet2vec as an additional baseline. ## related research on evaluation a growing body of research advocates a reassessment of the evaluation protocol for kbe models, even if link prediction is taken as the ultimate goal. @xcite and @xcite argue that current benchmarks contain too many trivial and too few difficult predictions, advocating for more realistic evaluation. @xcite discuss how random splits alter the graph topology from the original knowledge base, and show that correcting for this allows simple translation-based models to outperform more complicated ones. finally, @xcite and @xcite show that performance on link prediction benchmarks is strongly determined by hyperparameters and training mechanisms. going beyond link prediction, jain et al. (2021) challenge the prevalent belief that kbe models can properly capture the semantics of kb entities and relations. by evaluating common kbe models on entity typing and entity clustering benchmarks, the authors discovered that their results were no better than those achieved by simple statistical models. similarly, @xcite designed tests for specific capabilities of kbe models, such as capturing symmetric relations or hierarchical relations. they found that performance on link benchmarks did not reflect performance on their tests. together, these studies suggest that link prediction is a narrow evaluation metric, even when considering tasks defined on the knowledge base itself. our work differs from these studies as we primarily focus on external evaluation datasets, investigating the limitations of wordnet embeddings in representing lexical semantics. specifically, we apply wordnet kbe models to semantic evaluation datasets and downstream tasks. differences between training-related and downstream measures of performance have been found for several nlp tasks. chiu et al. (2016), qiu et al. (2018), and torregrossa et al. (2020) found that performance on common benchmarks for word embeddings do not correlate well with downstream performance. similar results have been demonstrated especially for intrinsic measurements of bias and fairness, which often struggle to adequately capture downstream failures @xcite . ## experimental setup to demonstrate how performance on link prediction benchmarks is inadequate for assessing semantic capabilities, we evaluate common link prediction models on several semantic tasks. we use two tasks directly testing semantic understanding (word similarity, word analogy), and two downstream tasks using the embeddings (pos-tagging, ner). these tasks serve as representative examples of challenges that semantic embedding models should be able to tackle effectively. ## wordnet potatoes and kbe model architectures: a problematic mash as a concrete demonstration of how kbe models fail to accurately represent semantics, we consider the case of hypernymy. wordnet encodes a tree defined by the hyponymy and hypernymy relations, with concepts near the leaves representing more concrete forms of those occurring near the root. this is a transitive relation: a potato is a concrete example of a vegetable, and therefore also a con-crete example of food. to accurately model the semantic information in wordnet, models must encode this hierarchical structure in their latent spaces. unfortunately, models are typically not designed for this. we find that some of our selected models exhibit unexpected behavior when encoding hypernymy, and almost all of them fail to properly capture the transitive property. ## further analysis: subtasks under the spudlight an interesting question is how uniform our findings are across various subtasks of our evaluation tasks. conveniently, simlex-999 and bats allow for such analysis, respectively across pos-tag and question category. detailed model results for these can be seen in analogy questions in bats are classified into four groups according to the original paper: derivational, lexicographical, encyclopedic, and inflectional. we exclude the inflectional group due to its limited coverage in wordnet. as per for both datasets, the ranking of the models by performance is broadly consistent between the data subsets. the impact of the additional relations in wn25 is inconsistent, generally improving performance on bats but decreasing it on simlex999. ## conclusion our study has highlighted the substantial discrepancy between the performance of kbe models on the wn18rr benchmark versus their performance in semantics and downstream tasks. specifically, we have shown that kbe models, when designed absent of linguistic insights, are capable of delivering high performance on link prediction metrics while also underperforming and demonstrating unexpected behaviors in semantic analysis. our findings demonstrate the deficiency of the wn18rr benchmark in evaluating specific semantic capabilities of kbe models and underscore the potential advantages of constructing models tailored for semantic tasks. consequently, we advocate for the development of targeted benchmarks that assess specific semantic abilities of wordnet kbe models, which enable model designers to calibrate and optimize their models in alignment with precise semantic requisites.