Lexical Semantics with Vector Symbolic Architectures (2023)

Abstract: Conventional approaches to the construction of word vectors typically require very large amounts of unstructured text and powerful computing hardware, and the vectors themselves are also difficult if not impossible to inspect or interpret on their own. In this paper, we introduce a method for building word vectors using the framework of vector symbolic architectures in order to encode the semantic information in wordnets, such as the Open English WordNet or the Open Multilingual Wordnet. Such vectors perform surprisingly well on common word similarity benchmarks, and yet they are transparent, interpretable, and the information contained within them has a clear provenance.

## introduction semantic representations based on word vectors are the foundation of many, if not most, algorithms in computational linguistics that require some means of handling lexical semantics. however, the most widespread of these, word embeddings based on the prediction of masked words, have a number of significant limitations. these begin with how such embeddings are constructed. the training process for these vectors requires large amounts of textual data, often on the order of billions of tokens. the scale of data required then brings further problems along with it: it is difficult (often prohibitively) to audit and improve the training data, the large size of the datasets requires accordingly high performance computers capable of processing the data, and models can only be trained for languages and domains where such large quantities of naturally occurring data exist. then, once a model has been trained, the resulting vectors are difficult if not impossible to fully understand and interpret. individual vectors cannot be inspected by themselves, they can only be compared with one another in terms of their distance from one another, and it is unclear exactly what relation this distance represents: is it more like similarity or relatedness or something else? in this paper, we propose a way of building word vectors that aims to address these issues. our word vectors use a single manually constructed linguistic resource (though there is no reason why one should be limited to a single resource, in theory) and thus represent relationships between words whose provenance is clearly traceable. as the resource is updated and improved, or adapted to a particular domain, the vectors can be adjusted accordingly, even individually and incrementally, incorporating the new information. by constructing the vectors in this way, rather than from unstructured text, we can avoid some of the problems described above. the vectors described in this paper can be constructed on ordinary laptop computers and require no more than a few hundred megabytes of memory. since they are based on linguistic resources, it is much easier to know what information goes into the vectors and how it got there. for some languages, especially ancient and historical languages, it's much more feasible to extend a linguistic resource than to discover more text from which to build a corpus. the vectors we describe in this paper are built using techniques known as vector symbolic architectures or hyper-dimensional computing, which we apply to an established high-quality linguistic resource, the open english wordnet (oewn) @xcite . we'll describe how the vectors are constructed (section 2.2) and some of their interesting properties relating to interpretability (4.1). then we'll compare their performance on benchmarks for word embeddings to get an idea the code for the experiments described here can be found at https://git.noc.rub.de/ajroussel/vsa-wn . ## conclusion and outlook in this paper, we described how techniques of vsas could be applied to linguistic resources, such as wordnets, in order to create word vectors that can be used in a manner akin to word embeddings, but which offer some key advantages over conventional word embeddings. the performance of these vectors on established word similarity benchmarks is similar to other wordnet-based methods, if not strictly equal to them, but, in exchange for some 10 percentage points of correlation, the vectors produced by the method described here are inspectable, interpretable and can be improved incrementally. furthermore, in contrast to the more conventional approaches, these vectors require few resources and can be built using ordinary computer hardware. in principle, such models can be constructed from any wordnet. though larger wordnets such as the oewn contain many synsets and cover more lexical items, it appears that even wordnets with much fewer synsets can still leverage the underlying structure of the omw such that the derived vector representations can be useful. this suggests that the approach described here could be valuable in situations in which it is infeasible to compile a large enough corpus for usable word embeddings or in which one seeks more transparency as to the contents of word vectors than distributional methods generally allow. future work since the collection of semantic vectors functions somewhat like a queryable database, it would be interesting to investigate to what degree query vectors and comparison operations could be constructed in order to try and query relations other than similarity: for instance, by ex-amining a synset's distance from nodes that function as paradigm words for concreteness, similar to approaches described by @xcite , whether one could use this same model to compare the concreteness of concepts. alternatively, perhaps it could be possible to compare vectors that have been modified in some systematic way, perhaps certain relations have been unbound from the vectors, in order to emphasize some particular aspect of their semantics or to retrieve some particular relation between them. sometimes there is little to distinguish some synsets from one another, since the set of relations they participate in is very similar or the same, so that our current model sees them as equivalent. this is apparent in the query (7), where all of the second-best answers are presented as equally close, which is a little counterintuitive. one idea would involve making use of the definition text as a useful workaround. for example, one could observe which words occur in which definitions and infer new relations for definitions that share significant overlap. finally, there is a lot of potential in encoding the semantic information in wordnets with vsas in ways that are very different from the one we implemented in this study. exploring some of these alternative approaches is a promising avenue for future work.