Probing Taxonomic and Thematic Embeddings for Taxonomic Information (2023)

Abstract: Modelling taxonomic and thematic relatedness is important for building AI with comprehensive natural language understanding. The goal of this paper is to learn more about how taxonomic information is structurally encoded in embeddings. To do this, we design a new hypernym-hyponym probing task and perform a comparative probing study of taxonomic and thematic SGNS and GloVe embeddings. Our experiments indicate that both types of embeddings encode some taxonomic information, but the amount, as well as the geometric properties of the encodings, are independently related to both the encoder architecture, as well as the embedding training data. Specifically, we find that only taxonomic embeddings carry taxonomic information in their norm, which is determined by the underlying distribution in the data.

## introduction research on probing @xcite @xcite has gained significant momentum in the nlp community in recent years, helping researchers explore different aspects of text encodings. while its potential for application is broad, there are still many nlp tasks the framework has not been applied to. specifically, it seems the majority of impactful probing work focuses on analysing syntactic properties encoded in language representations, yet the rich and complex field of semantics is comparably underrepresented @xcite . one particular semantic problem that has not been explored at all in the context of probing is the distinction between the taxonomic and thematic dimensions of semantic relatedness @xcite : words or concepts which belong to a common taxonomic category share properties or functions, and such relationships are commonly reflected in knowledge-engineered resources such as ontologies or taxonomies. on the other hand, thematic relations exist by virtue of co-occurrence in a (linguistic) context where the relatedness is specifically formed between concepts performing complementary roles in a common event or theme. this distinction informs the theoretical basis of our work, as we wish to explore the tension between taxonomic and thematic representations by examining how their information is structurally encoded. indeed, the vast majority of pretrained language models (ptlms) are trained solely on natural language corpora, meaning they mainly encode thematic relations. consequently, most probing work is applied to thematic embeddings, while taxonomic embeddings remain unexplored. we thus use the probing framework to study and compare taxonomic and thematic meaning representations. in addition, one aspect of embeddings that has not received much attention is the contribution of the vector norm to encoding linguistic information. we have recently highlighted this gap in the literature and developed an extension of the probing method called probing with noise @xcite , which allows for relative intrinsic probe evaluations that are able to provide structural insights into embeddings and highlight the role of the vector norm in encoding linguistic information. we find taxonomic embeddings to be particularly interesting for probing the role of the norm, as we suspect that the hierarchical structure of a taxonomy is well suited to be encoded by the vector norm-given that the norm encodes the vector's magnitude, or distance from the space's origin, it is possible that the depth of a tree structure, such as a taxonomy, could be mapped to the vector's distance from the origin in some way 1 . applying the probing with noise method to taxonomic embeddings on a taxonomic probing task could shed some light on this relationship. in order to draw broader comparisons, we apply the same evaluation framework to taxonomic and thematic sgns and glove embeddings. ## related work hypernymy, understood as the capability to relate generic terms or classes to their specific instances, lies at the core of human cognition and plays a central role in reasoning and understanding natural language @xcite . two words have a hypernymic relation if one of the words belongs to a taxonomic class that is more general than that of the other word. hypernymy can be seen as an is-a relationship, and more practically, hypernymic relations determine lexical entailment @xcite and form the is-a backbone of almost every ontology, semantic network and taxonomy @xcite . given this, it is not surprising that modelling and identifying hypernymic relations has been pursued in nlp for over two decades @xcite . while research on hypernym detection has been plentiful, work applying any probing framework to identify taxonomic information in embeddings is scarce, and the existing work does nor probe for it directly, but rather infers taxonomic knowledge from examining higher-level tasks. for example, ettinger (2020) identified taxonomic knowledge in bert, but rather than using a probing classifier, bert's masked-lm component was used instead and its performance was examined on a range of cloze tasks. one of the relevant findings was that bert can robustly retrieve noun hypernyms in this setting, demonstrating that bert is strong at associating nouns with their hypernyms. @xcite build on ettinger's work and investigate whether probing studies shed light on bert's systematic knowledge, and as a case study examine hypernymy information. they devise additional cloze tasks to test for prediction consistency and demonstrate that bert often fails to consistently make the same prediction in slightly different contexts, concluding that its ability to correctly retrieve hypernyms is not a reflection of larger systematic knowledge, but possibly an indicator of lexical memorisation @xcite @xcite . aside from this recent focus on bert, little work has been done in the space of probing embeddings for hypernym information. however, work on modelling hypernymy has a long history that stretches back before large ptlms and includes pattern-based approaches @xcite @xcite @xcite @xcite @xcite that are based on the notion of distributional generality @xcite , as well as distributional approaches @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . we highlight the work of @xcite , who demonstrated that it is possible to predict a specific semantic relation between two words given their distributional vectors. their work is especially relevant to ours as it shows that the nature of the relationship one is trying to establish between words informs the operation one should perform on their associated vectors, e.g. summing the vectors works well for a co-hyponym task. we consider this in §3. in terms of evaluation benchmarks for modeling hypernymy, in most cases their design reduces them to binary classification @xcite @xcite , where a system has to decide whether or not a hypernymic relation holds between a given candidate pair of terms. criticisms to this experimental setting point out that supervised systems tend to benefit from the inherent modeling of the datasets in the task, leading to lexical memorization phenomena. some attempts to alleviate this issue involve including a graded scale for evaluating the degree of hypernymy on a given pair @xcite , or reframing the task design as hypernym discovery @xcite . the latter addresses one of the main drawbacks of the binary evaluation criterion and resulted in the construction of a hypernym discovery benchmark covering multiple languages and knowledge domains @xcite . however, in our experiments we wish to probe both taxonomic and thematic encoders. given that we are mostly using pretrained thematic and taxonomic embeddings (see §4), their vocabulary coverage might vary dramatically. we wish to mitigate confounders by comparing like for like as much as possible, so to retain a higher integrity of interpretation when comparing models, we prune the dataset to only use the intersection of vocabularies of all the used models-we only include word pairs that have a representation for both candidate words in all the embedding models. note here that one of the goals of our work is to use the probing with noise method to learn about embeddings and the way they encode different types of information in vector space. we assert that a prediction of the relationship between a pair of words cannot be fairly done without the classifier having access to representations for both words in the pair. yet, our probe is a classifier which can only take a single vector as input. informed by the work of @xcite we considered options such as averaging or summing the individual word vectors, but found that these were not suitable for our framing as they muddled the notion that the classifier is receiving two separate words as input. we instead concatenate the word vectors in question and pass a single concatenated vector to the classifier (similar to approaches used by @xcite ). this approach allows us to formulate the task as a positional classification task: given a pair of words, is the first one the hypernym or the hyponym of the other? we can then assign each instance in the corpus a binary label-0 or 1-representing the class of the first word in the pair. the probe can then predict if the left half of the vector is the hyponym (0) of the right half, or whether it is its hypernym (1). finally, the inherent tree structure of wordnet means that a smaller number of words will be hypernyms, while a larger number will be hyponyms. we want to avoid the probe memorising the subset of words more likely to be hypernyms, but rather to learn from information encoded in the (differences between) vectors themselves. in an attempt to achieve this, we balance out the ratio of class labels by duplicating the dataset and swapping the hypernym-hyponym positions and labels. before duplicating, we also define a hold-out test set of 25,000 instances, so as to exclude the possibility of the same word pair appearing in both the train and test split-thus, the probe will be evaluated only on unseen instances. this duplication resulted in a final dataset of 493,494 instances, of which 50,000 comprise the test set and 443,494 comprise the training set. here are some example instances: ## experimental results experimental evaluation results for taxonomic and thematic embeddings on the hypernym-hyponym probing task are presented in tables 1 and 2 . note that all cells shaded light grey belong to the same sgns model them tax @xmath0. pred. .5000 .0009 .4997 .0009 rand. vec. .5001 .0012 .5001 .0011 vanilla .9163 .0004 .9256 .0003 abl. n .9057 .0004 .9067 .0005 abl. d .5039 .0008 .5294 .0010 abl. d+n .4998 .0010 .5002 .0009 sgns starting with thematic sgns (them), ## discussion there are a number of points to take away from our experimental results. firstly, we see that both vanilla thematic embeddings encode taxonomic information and the glove vanilla model significantly outperforms the sgns vanilla model. this is at least partially due to the fact that the pretrained sgns and glove thematic embeddings were trained on unrelated corpora, which differ in terms of size, topic and coverage: the corpus that glove was trained on is over 8 times larger than the one used to train the sgns model, and belongs to a different, much more varied genre of text data. thus, word representations derived from these resources are likely very different and it is possible that due to the broader scope and much larger size of the glove corpus, the glove representations reflect more taxonomic knowledge. however, these encoders exhibit the opposite behaviour when trained on the same wordnet random walk pseudo-corpus: expectedly, vanilla taxonomic sgns scores improve upon its thematic version, yet vanilla taxonomic glove scores significantly underperform compared to thematic. while we would expect it to mirror what was observed in sgns, taxonomic glove is in fact our worstperforming vanilla model. given the significant differences in model architectures, it is possible that this unexpected behaviour is due to an interaction between the architecture and training data 16 . while this may play a role, we suspect that the dominant factor is rather training corpus size. the word-net pseudo-corpus used for training taxonomic embeddings was only about 9 million tokens in size (which is sufficient to encode taxonomic relations, as shown by @xcite ), whereas sgns and glove were trained on 100 and 840 billion tokens respectively. it is not surprising that glove trained on a small and relatively sparse pseudo-corpus underperforms compared to training on a large natural corpus. if anything, it is encouraging that sgns trained on a 9-milliontoken pseudo-corpus outperforms one trained on a 100-billion-token natural corpus. another important finding from our experiments is the strong evidence that word embedding models can use the norm to encode taxonomic information, regardless of what is encoded in the vector dimensions. we find the clearest example of this in taxonomic glove after ablating dimension information, where the score remains as high as ≈0.66, meaning that the difference of 0.16 points is solely due to information in the norm. this is a very large difference given our understanding of the underlying mechanics, where it is well known that dimensions contain most, if not all information relevant for a task (e.g. @xcite ), and this is much more than has been demonstrated on any of the sentence-level experiments in our previous work @xcite . additionally, this is the only case where deleting half of each word vector yields a significantly higher score (≈0.87) than ablating the norm (≈0.84). this suggests that more information is lost when the norm is ablated than when half of the dimensions are removed. this is a strong indicator that in this case the norm encodes information that is not at all available in the dimensions. certainly, the majority of the information in an embedding is and will always be encoded in the dimensions, but it is striking how much of it is present in the norm in this case. generally, when it comes to dimension deletion experiments, it is expected that the performance would drop dramatically in comparison to vanilla embeddings. however, an important takeaway is that in all settings the drop is much smaller than might be expected, being quite close to vanilla performance and largely above random performance. this points to a redundancy within the dimensions themselves, seeing as either half of the vector seems to carry more than half the information required to model the task, indicating that not many dimensions are needed to encode specific linguistic features. this is consistent with the findings of @xcite , who analysed individual neurons in ptlms and found that small subsets of neurons are sufficient to predict certain linguistic tasks. our deletion results certainly corroborate these findings, given how small the drop in the probe's performance is when half the vector is deleted. for additional insight into the norm, we examine the norm values. we calculate the norms of the individual hypernym and hyponym word vectors in our dataset and present the results in figure 1 . the median norm value shows that the difference between hypernym and hyponym norms seems to be minor in both thematic embedding types (glove: 6.26 and 6.24; sgns: 2.78 and 2.76), whereas the difference is an order of magnitude larger in both taxonomic representations (glove: 2.03 and 2.67; sgns: 5.64 and 5.80). the difference is also quite large between taxonomic glove and sgns, and it seems to be what is reflected in our experimental results, which show that glove stores the most hypernym-hyponym information in the norm. the median norm measurements show that, on average, the norm of hypernyms is larger than the norm of hyponyms. this means that hypernyms, which are higher up in the tree, are positioned further away from the origin of the vector space than hyponyms, which are positioned lower in the tree and are closer to the origin. notably, this is only true in taxonomic embeddings, but not the thematic ones, indicating that in taxonomic embeddings there is a mapping between the taxonomic hierarchy and distance from the origin. finally, in spite of the fact that taxonomic glove (tax) is the worst-performing vanilla model, it is interesting that its norm also encodes the most taxonomic information. we base our interpretation of this result on the following: i) in many embeddings there is a high correlation between the norm and word frequency @xcite , and ii) wordnet pseudo-corpora reflect hypernymhyponym frequencies and co-occurrences. we suspect the principal signal that plays a role in the way taxonomic embeddings encode taxonomic knowledge is precisely these word co-occurrences, which glove is designed to capture. in turn, the norm can be seen as analogous to the hierarchical nature of taxonomic relationships and becomes the most accessible place to store this information. the thematic corpora reflect thematic co-occurrences and frequencies and hence glove (them) does not store taxonomic information in the norm, as such relations are not hierarchical in nature. ## conclusion in this paper we applied the probing with noise method to two different types of word representations-taxonomic and thematic-each generated by two different embedding algorithms-sgns and glove-on a newly-designed taxonomic probing task. the overall findings are that (a) both taxonomic and thematic static embeddings encode taxonomic information, (b) that the norm of static embedding vectors carries some taxonomic information and (c) thus the vector norm is a separate information container at the word level. (d) while in some cases there can be redundancy between the information encoded in the norm and dimensions, at other times the norm can encode information that is not at all available in the dimensions, and (e) whether the norm is utilised at all is sometimes dependant on training data, not just the encoder architecture. we also show that in the case of sgns, taxonomic embeddings outperform thematic ones on the task, demonstrating the usefulness of taxonomic pseudo-corpora in encoding taxonomic information. indeed, this work serves to further emphasise the importance of the norm, showing that the taxonomic embeddings use the norm to supplement their encoding of taxonomic information. in other words, random walk corpora can improve taxonomic information in word representations, which is not always the case for natural language corpora.