Together We Make Sense–Learning Meta-Sense Embeddings (2023)

Abstract: Sense embedding learning methods learn multiple vectors for a given ambiguous word, corresponding to its different word senses. For this purpose, different methods have been proposed in prior work on sense embedding learning that use different sense inventories, sense-tagged corpora and learning methods. However, not all existing sense embeddings cover all senses of ambiguous words equally well due to the discrepancies in their training resources. To address this problem, we propose the first-ever meta-sense embedding method – Neighbour Preserving Meta-Sense Embeddings, which learns meta-sense embeddings by combining multiple independently trained source sense embeddings such that the sense neighbourhoods computed from the source embeddings are preserved in the meta-embedding space. Our proposed method can combine source sense embeddings that cover different sets of word senses. Experimental results on Word Sense Disambiguation (WSD) and Word-in-Context (WiC) tasks show that the proposed meta-sense embedding method consistently outperforms several competitive baselines. An anonymised version of the source code implementation for our proposed method is submitted to reviewing system. Both source code and the learnt meta-sense embeddings will be publicly released upon paper acceptance.

## introduction in contrast to static word embedding methods @xcite that learn a vector, that represents the meaning of a word, sense embedding methods @xcite @xcite learn multiple vectors per word, corresponding to the different senses of an ambiguous word. prior work has shown that sense embeddings are useful for tasks such as word sense disambiguation (wsd) and sense discrimination tasks such as word in context (wic) @xcite . however, existing sense embeddings are trained on diverse resources such as sense tagged corpora or dictionary glosses, with varying levels of sense coverage (e.g. fullycovering all synsets in the wordnet or only a subset), and using different methods. therefore, the performance reported by the existing sense embeddings on different downstream tasks and datasets vary significantly for different part-of-speech (pos) categories. moreover, it is not readily clear which sense embedding learning method should be used for disambiguating words in a given domain. meta-embedding learning has been successfully used to learn accurate and high coverage wordand sentence-level meta-embeddings by combining independently trained multiple source embeddings @xcite . however, to the best of our knowledge, meta-embedding learning methods have not been applied for sense embeddings before. compared to word-level meta-embedding, sense-level meta-embedding has two important challenges. challenge 1 (missing senses). compared to learning meta-word embeddings, where each word is assigned a single embedding, in static sense embeddings an ambiguous word is associated with multiple sense embeddings, each corresponding to a distinct sense of the ambiguous word. however, not all of the different senses of a word might be equally covered by all source sense embeddings. challenge 2 (misalignment between sense and context embeddings). in downstream tasks such as wsd, we must determine the correct sense s of an ambiguous word w in a given context (i.e. a sentence) c. this is done by comparing the sense embeddings for each distinct sense of w against the context embedding of c, for example, computed using a masked language model (mlm) such as bert @xcite . the sense corresponding to the sense embedding that has the maximum similarity with the context embedding is then selected as the correct sense of w in c. for sense embeddings such as lmms @xcite or ares @xcite this is trivially achieved because they are both bertbased embeddings and the cosine similarity between those sense embeddings and bert embeddings can be directly computed. however, this is not the case for the meta-sense embeddings that exist in a different vector space than the context embeddings produced by bert, where a projection between meta-sense and context embedding spaces must be learned before conducting wsd. to address these challenges, we propose neighbourhood preserving meta-sense embedding (npms) by incorporating multiple independently trained source sense embeddings to learn a metasense embedding such that the sense-related information captured by the source (input) sense embeddings is preserved in the (output) meta-sense embedding. npms can combine full-coverage sense embeddings with partial-coverage ones, thereby improving the sense coverage in the latter. npms does not compare the source embeddings directly but require the nearest neighbours computed using source and meta sense embeddings to be similar. we call this information preservation criteria, and use pairwise inner product (pip) to compare the similarity distributions (nearest neighbours) over senses between meta and source embedding spaces. this allows us to address challenge 1 using shared neighbours to compute the alignment between source and meta embedding spaces, without predicting any missing sense embeddings. to address challenge 2, npms requires meta-sense embedding of a word sense to be similar to the contextualised (word) embeddings of the words that co-occur in the same sentence. we call this contextual alignment, and learn the sense-specific projection matrices that satisfy this criteria. this ensures that meta-sense embeddings could be used in downstream tasks such as wsd or wic, where we must select the correct sense of an ambiguous word given its context. we evaluate npms on wic and wsd tasks against several competitive baselines for metaembedding. experimental results show that npms consistently outperforms all other methods in both tasks. more importantly, we obtain state-of-theart (sota) performance for wsd and wic tasks, reported by any static sense embedding method. source code for the proposed method is publicly available. foot_0 2 related work our work is related to both static sense embeddings and meta-embedding learning as we review next. static sense embeddings assign multiple embeddings for a single word, corresponding to its distinct senses. @xcite proposed multi-prototype embeddings to represent word senses, which was extended by @xcite combining both local and global contexts. both methods use a fixed number of clusters to represent a word, whereas @xcite proposed a non-parametric model, which estimates the number of senses dynamically per each word. @xcite initialised sense embeddings by means of glosses from wordnet, and adapted the skip-gram objective @xcite to learn and improve sense embeddings jointly with word embeddings. @xcite used pretrained word2vec embeddings to compose sense embeddings from sets of synonymous words. @xcite created sense embeddings using structural knowledge from the babelnet @xcite . @xcite constructed sense embeddings by taking the average over the contextualised embeddings of the sense annotated tokens from semcor. @xcite used the lexical-semantic information in babelnet to produce sense embeddings without relying on senseannotated data. @xcite also proposed ares, a knowledge-based approach for constructing bert-based embeddings of senses by means of the lexical-semantic information in ba-belnet and wikipedia. meta embedding learning was first proposed for combining multiple pretrained static word embeddings @xcite . vector concatenation @xcite is known to be a surprisingly strong baseline but increases the dimensionality of the meta-embedding with more sources. coates and @xcite showed that averaging performs comparable to concatenation under certain orthonormal conditions, while not increasing the dimensionality. learning orthogonal projections prior to averaging has shown to further improve performance @xcite . globally linear @xcite , locally linear (bollegala and bao, 2018a) and autoencoder-based nonlinear projections @xcite have been used to learn word-level meta-embeddings. meta-embedding methods have been used for contextualised word embeddings @xcite and sentence embeddings @xcite . for an extensive survey on meta-embedding learning see @xcite . however, to our best knowledge, we are the first to apply meta-embedding learning methods to learn sense embeddings. ## meta-sense embedding learning to explain our proposed method in detail, let us first consider a vocabulary v of words @xmath0. we further assume that each word w is typically associated with one or more distinct senses s and the set of senses associated with w is denoted by s w . in meta-sense embedding learning, we assume a sense s of a word to be represented by a set of n source sense embeddings. let us denote the j-th source embedding of s by x j (s) ∈ r d j , where d j is the dimensionality of the j-th source embedding. we project the j-th source embedding by a matrix p @xmath1. the meta-sense embedding, m(s) ∈ r d of s is computed as the unweighted average of the projected source sense embeddings as given by (1). after this projection step, all source sense embeddings live in the same d-dimensional vector space, thus enabling us to add them as done in ( 1 ). an advantage of considering the average of the projected source embeddings as the meta-sense embedding is that, even if a particular sense is not covered by one or more source sense embeddings, we can still compute a meta-sense embedding using the remainder of the source sense embeddings. moreover, prior work on word-level and sentence-level meta-embedding have shown that averaging after a linear projection improves performance when learning meta embeddings @xcite @xcite . if we limit the projection matrices to be orthonormal, they can be seen as optimally rotating the source sense embeddings such that the projected source embeddings could be averaged in the metaembedding space. however, we observed that dropping this regularisation term to produce better metasense embeddings in our experiments. therefore, we did not impose any orthonormality restrictions on the projection matrices. we require a meta-sense embedding to satisfy two criteria: (a) sense information preservation and (b) contextual alignment. the two criteria jointly ensure that the meta-sense embeddings we learn are accurate and can be used in downstream tasks such as wsd in conjunction with contextualised word embeddings produced by an mlm. next, we describe each of those criteria in detail. ## conclusion we proposed the first-ever meta-sense embedding learning method. experimental results on wic and wsd datasets show that our proposed npms surpasses previously published results for static sense embedding, and outperforms multiple word-level meta-embedding learning methods when applied to sense embeddings. our evaluations were limited to english and we will consider non-english sense embeddings in our future work.