Modelling Lexical Ambiguity with Density Matrices (2020)

Abstract: Words can have multiple senses. Compositional distributional models of meaning have been argued to deal well with finer shades of meaning variation known as polysemy, but are not so well equipped to handle word senses that are etymologically unrelated, or homonymy. Moving from vectors to density matrices allows us to encode a probability distribution over different senses of a word, and can also be accommodated within a compositional distributional model of meaning. In this paper we present three new neural models for learning density matrices from a corpus, and test their ability to discriminate between word senses on a range of compositional datasets. When paired with a particular composition method, our best model outperforms existing vector-based compositional models as well as strong sentence encoders.

## introduction an integral part of natural language understanding is the ability to handle lexical ambiguity. words can have multiple meanings, and the precise meaning of a word only becomes clear when we see it in use -the surrounding context disambiguates it. word sense disambiguation (wsd) is said to be an 'ai complete' problem @xcite , that is, a problem that is at least as hard as any other problem in ai, and as such has been the subject of extensive research. standard approaches treat wsd as a classification problem: given a word in context, the task is to classify it into one of a range of possible senses @xcite @xcite . a more difficult task is to disambiguate every word in a sentence @xcite . a summary of the state of the art is given in @xcite . more recently, neural approaches @xcite use contextualised embeddings as input to wsd systems, together with knowledge from wordnet. other neural approaches generate multiple sense vectors per word @xcite or vectors representing a context @xcite . disambiguation can be costly. each word should be disambiguated with respect to the correct senses of the other words in the sentence, meaning that the computational complexity of the task can become problematic @xcite . within a compositional framework, the idea is for words to disambiguate automatically in the process of composition @xcite @xcite . within purely vector-based models, the amount of ambiguity that a word vector can represent is limited. @xcite argue that distributional vectors work well for polysemy, but less so for homonymy. @xcite extend the vector based model of meaning to encompass homonymy by using the notion of a density matrix. these can be used to encode a probability distribution over possible meanings of a word in a single representation. density matrices can also be accommodated within a compositional framework, allowing the ambiguity encoded in the matrix to be resolved via composition. we use density matrices within a compositional distributional framework to model word and sentence meaning. we propose three new models for building density matrices, based on neural word embedding models. we survey several composition methods for density matrices and evaluate how well our density matrices encode ambiguity and to what extent the composition methods achieve disambiguation, on four disambiguation datasets that test disambiguation in a compositional setting. one of our models (multi-sense word2dm) emerges as the best model overall. when paired with a particular composition method (phaser), multi-sense word2dm outperforms all other models (including existing baselines and high-performing sentence encoders) on most of the disambiguation tasks. ## background compositional distributional models come in a range of flavours. @xcite use simple element-wise operations on vectors. more recently, neural models of composition @xcite and large networks such as bert @xcite have been extremely successful. a third flavour is the typelogical tensor-based models of composition @xcite @xcite . the tensor-based model of composition works as follows. we choose a vector space n for nouns, and another s for sentences, and represent relational words as multilinear maps over these spaces. intransitive verbs are represented as linear maps @xmath0, i.e. matrices in n ⊗ s. transitive verbs are represented as maps from two copies of n to s, i.e. order 3 tensors or 'cubes' of parameters in n ⊗ s ⊗ n . composition is performed via tensor contraction -an extension of matrix multiplication. matrices and tensors require many parameters. to alleviate this problem, @xcite ; @xcite develop ways of building matrices and tensors from word vectors, some of which are described in section 4.1.1. we use an extension of the tensor-based approach, based on methods given in @xcite ; @xcite . nouns and sentences are represented as density matrices and relational words (adjectives, verbs, etc.) are represented as completely positive maps, which take density matrices to density matrices. representing words with density matrices a density matrix over r n is a matrix of the form to represent words, we view each word as a probability distribution over senses, and we view the vectors -→ v i in equation ( 1 ) as representing its different senses. for example, the word bright could mean shiny or clever. suppose that when bright is used, it is twice as likely to mean shiny as it is to mean clever. the density matrix for bright, denoted bright , is computed as follows: composition with density matrices since we are working with density matrices, nouns are now maps @xmath1, i.e. matrices in n ⊗ n . sentences are matrices in s ⊗ s. this means that intransitive verbs are order-4 tensors that take a noun density matrix as input and give back a sentence density matrix. they live in a space n ⊗ n ⊗ s ⊗ s. transitive verbs are order-6 tensors. clearly, these spaces get very big very quickly. to deal with this increase in dimensionality, tricks to create completely positive maps out of density matrices have been proposed @xcite . this allows composition mechanisms to be specified at the level of density matrices, rather than having to work in the high-order spaces described above. we describe these composition mechanisms in section 3.2. other applications of density matrices in nlp include modelling entailment in a compositional setting @xcite @xcite @xcite . @xcite also use density matrices to model ambiguity, but in a different setting. wsd in compositional distributional semantics @xcite argue that compositional distributional semantic models are particularly able to pick out the more subtle shades of meaning termed polysemy. this idea is used in @xcite @xcite , where a range of semantic composition models are tested on datasets built to distinguish different senses of words in context. neural and distributional models for disambiguation are compared in @xcite , and the role of ellipsis in disambiguation is investigated in @xcite . ## experimental setup we evaluate our models on three tasks -word similarity, disambiguation, and a word level ambiguity analysis. the code for training and evaluating our models has been made available at https://github.com/francois-meyer/ lexical-ambiguity-dms. in this section we introduce the experimental setup used in all of these tasks. ## results we introduce each of the evaluation tasks and present our results. for multi-sense word2dm and bert2dm we trained four models each, with different hyperparameter settings (as described in section 4.2 and listed in ## conclusion and future work in this paper we addressed the problem of modelling ambiguity in nlp, and how ambiguous words can be disambiguated in context. we investigated density matrices as semantic representations for modelling ambiguity. our results confirmed the value of density matrices over vector-based approaches. equipped with a compositional frame- possible directions for future work includes extending our models to larger datasets and longer sentences, and modifying techniques for differing sentences as in the word in context dataset (pilehvar and camacho-collados, 2019) and other wsd tasks. we focused on ambiguity here, but it would be possible to do similar experiments focused on other aspects of meaning such as metaphor or entailment, and examining how these interact with composition. this allows us to rewrite equation 9 to find an equivalent objective function that requires fewer computations than straightforward matrix multiplication would. the objective function at each target-context prediction becomes by using the result of lemma a.2 we have reduced the number of matrix multiplications required for each target-context prediction from 3(k + 1) to (k + 1). density matrices are trained by maximising equation 10 with respect to the intermediary matrices b t , b c , b w 1 , ..., b w k over a large corpus. the model is trained using stochastic gradient descent. we now derive the gradients used to update b t during training, and subsequently show that these gradients lead to suboptimal updates to the density matrices during training. deriving the gradient with respect to b c and b w k would proceed similarly. to compute the gradients of equation 10 we first rewrite it in terms of the elements of the n × m matrices b t , b c , and b w k : the derivative of j with respect to b t pq can now be computed as follows: the last line in the above derivation is obtained by rewriting the summation expressions as equivalent matrix multiplications. we can now write the derivative of j with respect to the full intermediary matrix b t : as opposed to the gradients of word2vec (equation 7), the gradients of word2dm do not lead to simple and easily interpretable training updates. as discussed in the paragraph following equation 8, in word2vec the target vector is made more similar to the context vector and less similar to the negative context vectors. ideally we would like something similar to occur in word2dm with density matrices, but equation 13 shows that we lose the intuitive training updates of word2vec through the introduction of intermediary matrices. furthermore, we can show that the gradients of word2dm sometimes lead to unwanted consequences in training. consider the case where the density matrices of a target and context word are highly dissimilar. recall from equation 9 that the y is in equation 13 is the trace inner product of the density matrices a t and a c (the measure we use to quantify semantic similarity). the minimum value of the trace inner product of two density matrices is zero (this follows from the fact that density matrices are positive semi-definite), so two density matrices are highly dissimilar when their trace inner product is close to zero i.e. y ≈ 0. from equation 10 we can recall how y can be written in terms of the intermediary matrices: consider that y ≈ 0 if and only if the elements of b c b t are close to zero in value, since squaring the elements in the summation makes them all positive. we have established the following equivalence: so the target-context update becomes ineffective for true contexts. the update should make the density matrix of the target word more similar to that of the context word, but the gradient is so small that it makes this impossible. moreover, the more dissimilar the target and context density matrices are before the update, the less effective the update will be. this is the opposite of the intended effect (achieved by word2vec) in which the magnitude of the target-context update should increase if the target and context representations are less similar. this is an example of how the introduction of intermediary matrices in word2dm leads to suboptimal training updates. we ensure that our density matrices are positive semi-definite, but lose the guarantee that the algorithm will learn high-quality semantic representations.