Word associations and the distance properties of context-aware word embeddings (2020)

Abstract: What do people know when they know the meaning of words? Word associations have been widely used to tap into lexical repre- sentations and their structure, as a way of probing semantic knowledge in humans. We investigate whether current word embedding spaces (contextualized and uncontextualized) can be considered good models of human lexi- cal knowledge by studying whether they have comparable characteristics to human associa- tion spaces. We study the three properties of association rank, asymmetry of similarity and triangle inequality. We find that word embeddings are good mod- els of some word associations properties. They replicate well human associations between words, and, like humans, their context-aware variants show violations of the triangle in- equality. While they do show asymmetry of similarities, their asymmetries do not map those of human association norms.

## introduction what do people know when they know the meaning of words? lexical semantic knowledge is rich and structured and comprises the knowledge of the relation between a word and its related concept, the relationships between concepts among themselves and between words themselves. word associations -spontaneous elicitation of words by similarity, contrast or contiguity-have been widely used to tap into lexical representations and their structure, as a way of probing semantic knowledge in humans @xcite . to process language in a way that mirrors human expectations and to develop usable technology, we need computational representations of the meaning of words that correspond to speakers' knowledge of words @xcite . current word embeddings methods, without context or contextaware, represent lexical semantic knowledge as coordinates in a multi-dimensional space. work in cognitive psychology, however, has argued that human lexical knowledge and in particular word associations are not well represented by geometrical models @xcite @xcite . more precisely, similarity judgments of associations in humans have been shown not to exhibit the properties of true distances. for example, associations are not symmetric: speakers will indicate that north korea is more similar to china than china is to north korea. in an analysis inspired by @xcite 's critique of spatial measures of similarity in word associations, and by recent work on topic models and word embeddings @xcite , we compare current word embedding spaces to the large human norming study by @xcite and investigate whether these word embedding spaces, especially those based on transformers, -the lexical representations used by almost all our current architecturesshare some properties with human word associations, as human-like representations of lexical meaning. ## word associations the task of word free association consists in providing a cue word to a speaker and asking to produce, fast and without thinking, other words that come to mind, called the target words. these collections of free associations present interesting characteristics. first, they present a certain stability across speakers, so that it is possible to determine the most associated word on average, the second most associated and so forth. second, they present interesting qualitative properties. they exhibit asymmetry of similarity judgments, as in the example on china and north korea above. finally, they also exhibit exp. description word embedding hypothesis method 1 find the top-k neighbors near the cue bert not in context, lemmatized and unlemmatized top-k near cue in bert are associates in human associations. knn, median rank and p@k 2.1 asymmetry of countries bert in context non-prominent countries are more similar to prominent countries than vice versa. cosine 2.2 asymmetries as frequencies output of exp.1 frequent words are more often associates of less frequent words than vice versa. cues and targets from nelson, cues and targets from exp. 1; hits of google's search engine. 2.3 asymmetries of hypernyms and hyponyms bert in context if @xmath0, hyponyms are more similar to hypernyms than the reverse. if @xmath1, the effect is the opposite. nouns and verbs from nelson; hypernyms/hyponyms from wordnet; cosine. 2.4 asymmetry as neighbourhood density bert not in context unlemmatized; fasttext (only for countries) a semantically richer word elicits a greater number of close neighbours than a fainter word. extraction of asymmetric pairs from nelson; knn; cosine (@xmath2.2) 3 violation of triangle inequality bert in context bert embedding space violates the triangle inequality. extraction of asymmetric triples; cosine, τ threshold. violation of transitivity (called in the literature, violation of the triangle inequality). for example, asteroid is highly associated with belt, and belt is highly associated with buckle, but asteroid and buckle show little association. feature-based explanations of these properties makes use of the richness of the representation, in terms of number of features and the proportion of shared features between two representations @xcite . for example, the asymmetry of similarity is explained by the assumption that only a few of the large number of features in speakers' mental representation of china are shared with north korea, while the representation of north korea involves a small number of features, many of which are shared with china. more recent approaches have proposed probabilistic representations in terms of topic models @xcite . violation of the triangle inequality is explained in these models as an effect of the fact that topic models represent different uses of a word as different topics, and different topics do not necessarily preserve similarity. we investigate whether current word embedding spaces can be considered good models of human lexical knowledge by studying whether they have comparable characteristics to human association spaces. we study the three properties of associations mentioned above: rank, asymmetry of similarity and violation of triangle inequality (lack of transitivity). more specifically, first, we verify that the notion of word association makes sense in word embedding spaces and compare if the target words that occur as preferentially associated with a cue word in a word embedding space correspond to the closest words in human association norms. then, we analyse asymmetry in similarity in several different ways: we look at its correlation with cue and target frequency, with neighbourhood densities and lexical entailment. finally, we model violations of triangle inequality by looking at how similarity of words spreads across word embedding spaces. a summary of the experiments discussed in more detail in the following sections is given in figure 1 . ## data to perform the studies indicated above, we use two sets of artificial word embedding data, and one set of human association norms. fasttext the original fasttext model is based on wikipedia dumps foot_1 in nine differerent languages including english @xcite . however, in this work, we used the pre-trained fast-text embeddings provided by the official site of fasttext, that we expressly do not modify. 3 the embeddings are trained on 600-billion tokens from commoncrawl foot_3 , resulting in two-million word vectors with subword information. @xcite propose a large dataset of free association, rhyme and word fragment norms, elicited from more than 6000 participants. the participants were asked to write the first word that came to mind when presented a particular stimulus word. more than 750'000 free associations (called targets) from a total of 5019 stimulus words (called cues) were collected. the related quantitative information (such as number of participants and measures of association strength) were calculated. 5 other word association norms exist, such as the small world of words @xcite , but in this study, we wanted to be able to compare, at least indirectly, our work to previous work where the nelson's norms were used @xcite . ## experiment 1: the notion of association to simulate the process of production of associates of a free association task, we used the knearest neighbours algorithm (knn) to find the topk words that are near the cue. the intuition is that words near the cue word in the embedding space are probably associates. we use cosine similarity as the metric to find the nearest neighbours. we compare if the target words that occur as preferentially associated with a cue word in a word embedding space correspond to the closest words in human association norms. for this comparison, we use two measures: median ranking and p@k, following @xcite . method as human associations in nelson are lemmatized, we tested both the lemmatized and unlemmatized versions on the "raw" bert word embeddings obtained from the vocabulary of the model. 6 the median rank is a measure of central tendency of the median rank in bert for the n-ranked associate target in nelson. for example, in figure 2 for the unlemmatized data, we would have to take the median across ranks of the ranks given in bert (1, 1, 2, 32, 22) for the first ranked target in nelson. this measure is calculated as follows. for each cue in nelson, we extract the top-k nearest neighbours in the bert space and rank the results by their cosine similarity in descending order. then, for each same cue and each one of its targets in nelson, we calculate the target's rank in bert (see figure 2 ). the median of these ranks is calculated for each of the nelson's rank (see using the same ranked lists, we calculate precision at k (p@k), where @xmath3, 2, 3, 4, 5. p@k tells us if the first associate in the human associa-6 these raw vectors come with the pre-trained model of bert. they are extracted from the vocabulary of the model, thus not in context. they can be downloaded from https: //github.com/ajitrajasekharan/bert_mask . their unlemmatized version uses the words and the embeddings as they come with the model. their lemmatized words are derived using the wordnet lemmatizer from nltk http://www.nltk.org/api/nltk. stem.html#module-nltk.stem.wordnet and the word embeddings corresponding to a lemmatised word is the sum of all the word embeddings of the unlemmatised word forms. ## experiment 2: asymmetries perhaps more interestingly than simple ranking measures, human word associations also exhibit peculiar qualitative properties that computational systems must also exhibit if they want to be considered human-like. these properties are especially interesting in a discussion of word embeddings as they seem to specifically defy a representation of the lexicon as a geometric space, in that they violate metric axioms, such as symmetry of distances and the triangle inequality. our intuition is, however, that context-aware word embeddings are no longer linear geometric representations in multi-dimensional space and that, as such, could mirror the geometricallywarped properties of human associations. for this reason, we use cosine, a symmetric similarity operator: any asymmetry found this way is to be ascribed to the context-aware vector and not to the similarity operator. here we want to model the asymmetric association between cue and target words @xcite . we want to model the intuition that these asymmetries stem from a richer and more specific representation for certain words (like china) and less specific or vaguer representation for less salient words (like north korea). ## experiment 3: violations of triangle inequality human word associations violate the triangle inequality, also called transitivity here. it is easy to find sets of words that have this property. for example, asteroid is highly associated with belt, and belt is highly associated with buckle, but asteroid and buckle have little association. the triangle inequality restricts the possible relationships between three words in embedding spaces: if w 1 and w 2 are highly associated and w 2 and w 3 are highly associated, then we expect w 1 and w 3 to be highly associated. however, as already motivated in section 5, our intuition is that context-aware word embeddings are no longer following the rules of linear geometric representations. for this reason, we use the cosine that, in addition to being symmetric, does not violate the triangle inequality. if any violation of the triangle inequality is found, it is attributable to the context-aware vector and not to the similarity operator. method we extracted a subset of the triples (w 1 , w 2 , w 3 ) from appendix a in @xcite 's norms for a total of 12664 triples, where both (w 1 , w 2 ) and (w 2 , w 3 ) are pairs of asymmetric nouns that share the same word w 2 . we call these "pivot triples". of these 12664 pivot triples only 263 are transitive, that is, they do not violate the triangle inequality and (w 1 , w 3 ) exists in the produces back the initial cue. for example, given the cue ball, one of the produced associates is the target baseball. conversely, baseball elicits ball as target. for these 6232 pairs, in 50.09% of cases, the target is denser than the cue. 12 for this experiment, we used fasttext, as countries cannot be used in context with this model and not all the countries were included in the bert "raw" word embeddings. human associations as a cue-target pair. 13 to calculate the bert word embeddings, we contextualise each pair in the three different contexts already shown for previous experiments. to determine if the two pairs of words violate the triangle inequality in bert space, we follow a procedure similar to griffiths et al. (2007)'s. we set a threshold τ and extract those (w 1 , w 2 ) and (w 2 , w 3 ) whose cosine is greater than τ. then, for each of these pivot triples, we quantify how many (w 1 , w 3 ) pairs exist in the associations as a cue-target pair. if the pair exists and (w 1 , w 3 ) is greater than τ, then bert's embeddings also show transitivity; if the pair exists, but (w 1 , w 3 ) is less than τ, then (w 1 , w 3 ) in bert does not reflect the transitivity found in humans; if the pair does not exist and (w 1 , w 3 ) is lesser than τ, then there is a lack of transitivity both in human associations and in bert embeddings. ## discussion we have found converging evidence for bert being like human word associations in ranks of associations, quantitatively and qualitatively. in studying whether bert similarity spaces are asymmetric, we find converging evidence to human experiments, using country names, both in the fact that the notion of salience influences the calculation of similarity and also in the fact that frequency correlates with the preferential direction of similarity. however, using a larger test set, we do not find convergence with frequency, or generality (hyperym/hyponym pairs of verbs and nouns) or neighbourhood density. finally, human word associations violate the triangle inequality. so do bert embedding spaces, for reasonably high values of the similarity measure. in conclusion, we confirm the properties of rank association and triangle inequality and also the influence of frequency for certain kinds of associations. we find, instead, that the property of asymmetric similarity does not appear to conform to the operationalisations we tested. ## related work recent interest in vectorial representations of words derives from the realisation that the meaning of words is much better represented when the rich networks of similarities and dissimilarities among words are taken into account. but the realisation that such notions are central to our word representations, that they can be estimated from a corpus and that such representations can be very technologically apt is not new. @xcite brought to the attention of the computational linguistic community a notion of word association as the informationtheoretic measure of mutual information estimated on large corpora. it was shown that if word order was taken into account the measure could be asymmetric. mutual information and other word association measures have been intensely studied to describe multi-word expressions or collocations, a stumbling block for many nlp applications (for a recent survey, see @xcite ). @xcite show that word embeddings are closely related to information-theoretic notions of mutual information, although denser and better performing in many tasks. our work builds on previous work in the nonassociationist tradition: word associations are a reflex of underlying properties and representations of words and their meaning @xcite @xcite and not the reverse. tversky and colleagues's body of work is centred in a contrast model, a model of objects, concepts and words based on features and not on a position in a multi-dimensional space. in this model, the similarity between objects increases with shared common features and decreases with distinctive features @xcite . from this point of view, computation of similarity is based on set-theoretic operations, rather than the computation of metric distances @xcite . in a probabilistic topic model, @xcite , words are a set of probability distributions on topics so that words that have a high probability under the same topic will be highly predictive of one another. the representations induced by the topic model and their correspondence to human memory are compared to a spatial representation model (latent semantic analysis, lsa) (landauer and dumais, 1997) and found to better reflect human association norms. current vector space semantic representations can be seen as inheriting from both the featurebased tradition and the similarity space tradition, exemplified by lsa. while initial similarity-space proposals like lsa represent words as atomic and occupying a single point in space, current geometric approaches represent words as vectors. the approaches proposed by @xcite ; @xcite ; @xcite are based on a distributed representation of words, and aim to produce vectors that represent a word, or the substrings that compose a word, with information about its surroundings, so that word vectors that share the same meaning tend to be close. a recent comparison of word embeddings, word2vec and glove, to nelson's norms indicates that vectorial representations that do not take context into account, unlike bert, still are unable to capture the triangle inequality @xcite . investigations of word associations also belongs to the growing literature of evaluating vector spaces for natural language applications. word associations are an interesting, intrinsic way to evaluate vector spaces @xcite , and have revealed important properties of these spaces, from gender stereotypes and demographic variation @xcite to their usefulness in the detection of puns @xcite , among many others. ## conclusions the work described in this paper starts from the assumption that word associations are the expression of underlying meaning properties of words. it confirms that context-aware word embeddings exhibit some properties of human association norms, despite being a vectorial representation of words in space. future work needs to clarify the underlying mechanisms that give rise to these properties, extend the study to new languages, leveraging also newer association norms @xcite . it will also extend the investigation of word associations to other properties, such as the minimal contrast rule -associations tend to establish a minimal contrast-or the marking rule -marked cues elicit unmarked targets more often than the reverse @xcite , and model documented differences between adults and children.