When is a bishop not like a rook? When it’s like a rabbi! Multi-prototype BERT embeddings for estimating semantic relationships (2020)

Abstract: This paper investigates contextual language models, which produce token representations, as a resource for lexical semantics at the word or type level. We construct multi-prototype word embeddings from bert-base-uncased (Devlin et al., 2018). These embeddings retain contextual knowledge that is critical for some type-level tasks, while being less cumbersome and less subject to outlier effects than exemplar models. Similarity and relatedness estimation, both type-level tasks, benefit from this contextual knowledge, indicating the context-sensitivity of these processes. BERT’s token level knowledge also allows the testing of a type-level hypothesis about lexical abstractness, demonstrating the relationship between token-level phenomena and type-level concreteness ratings. Our findings provide important insight into the interpretability of BERT: layer 7 approximates semantic similarity, while the final layer (11) approximates relatedness.

## introduction the rampant success enjoyed by contextualized language models (clms) like cove @xcite , elmo @xcite , bert @xcite , and roberta @xcite has precipitated a deluge of research into analyzing and interpreting their functionality. but to date, there has been little work analyzing their lexical semantic knowledge. this paper seeks to answer two questions: 1) is it possible to generate useful static word-type embeddings from bert activations for word tokens? 2) what sort of semantic relations are represented in embeddings generated from bert? 'useful' word embeddings are those which successfully represent target semantic relations and enable the testing of linguistic and cognitive hypotheses. many linguistic tasks and questions concern word meanings at the type level-that is, what a word means in general, abstracted away from any particular context. similarity, relatedness, and abstractness are often construed as properties at the type-level: most similarity and relatedness datasets contain judgments on isolated word pairs, and abstractness datasets contain judgments on isolated words. clms produce representations at the token level: the vector representation of a word varies depending on its context of occurrence. how can contextual representations be helpful for type-level tasks? similarity and relatedness are context-sensitive processes. for example, the relevant features of water used for calculating its similarity to land are different from those for calculating its similarity to coffee. bert's contextual knowledge is useful for dealing with the effects of this variation on similarity and relatedness judgments. relative abstractness/concreteness is another property often treated as a type-level phenomenon. however, this property may be detectable through token-level interactions. we hypothesize that a high degree of contextual variation may be an indicator of type-level abstractness. in other words, abstract words are more 'heterogeneous' than concrete words. traditional static representations, which represent words as infinitesimal points, are not suitable to test this hypothesis. in contrast, bert enables the representation of a single word as a constellation of points, with each point corresponding to a different usage or usage type. in bert, heterogeneity translates to the dispersion of tokens or prototypes in space. the most obvious extension of contextualized word embeddings to the type level is to build exemplar models from token representations. such models represent a word as the sum total of observed occurrences, i.e. the set of all token vectors. full exemplar representations are computationally expensive, and subject to noise from outliers. at the other end of the spectrum, some have tried averaging over exemplar models to generate a single vector for each word @xcite . however, these models do not leverage the contextual knowledge resident in berts later layers. our approach aims to capture the regularities in contextual variation while reducing the noise it introduces. a set of bert token vectors for a single word naturally tends to separate spatially into groups of similar usages, or 'usage types' @xcite . usage types often correspond to polysemous or homonymous senses, idiomatic constructions, and affordances. 1 figure 1 shows a t-sne visualization of tokens for madhouse and asylum. the dominant usage of asylum is as political refuge, but there is a cluster corresponding to asylum as an institution of confinement for people with psychiatric diagnoses, which shares similar negative connotations to madhouse, a derogatory colloquialism for such institutions. to test whether these usage types retain enough contextual information to aid in context-sensitive lexical tasks, we use k-means clustering of bert token representations to derive multi-prototype lexical embeddings. the embeddings are evaluated on the standard lexical tasks of similarity and relatedness estimation. clustered bert-based representations provide high-quality predictions of human judgments for both tasks. they are also employed to test a cognitive hypothesis which holds that one of the factors contributing to relative abstractness/concreteness a word is how much its meaning varies in context. we find that the average 1 affordances are the different ways a thing can present itself to an individual @xcite ). for instance, water can manifest as a drink, a chemical substance to be studied, a span to be travelled across, a medium for recreational sports, or a municipal resource, just to name a few. dispersion of tokens in a cluster bears a significant relationship to abstractness. the contributions of this paper are as follows: ## related work the pre-trained bert language model is a bidirectional transformer @xcite encoder. it has either 12 fully-connected layers (bert-base) or 24 (bert-large). the model is trained on two tasks: masked token prediction and next sentence prediction. for bert-base, the representation of an input sequence consists in a 12-layer activation network with 768-dimensional vectors for each input token (where tokens correspond to sub-word wordpieces @xcite ) at each of 12 layers. the preponderance of research surrounding analysis and interpretability of bert has been dubbed bertology, after the poster-child of the contextual revolution (cf. @xcite for an excellent survey). probing tasks most studies of word meaning in bert follow an agenda of extrinsic evaluation @xcite . tenney et al. (2019a) found that clms improve over non-contextual counterparts largely on syntactic tasks, with smaller gains on semantic tasks. tenney et al. (2019b) introduced edge-probing tasks to analyze the layer-wise structure of bert, and found that early layers perform syntactic tasks like part-of-speech and dependency tagging, while later layers encode information pertinent to semantic tasks like coreference resolution, relation labeling, and semantic proto-role labeling. we aim to advance this kind of structural analysis of bert through the intrinsic evaluation of representations at different layers. contextual word embeddings existing applications of clms to lexical tasks use exemplar models or single-prototype models. wiedemann et al. (2019) successfully employed a k-nearestneighbor approach to bert exemplar models for word sense disambiguation (wsd). coenen et al. (2019) created a visualization tool that generates a 'word cloud' from bert tokens, browsable by layer. foot_0 they also achieved a state-of-the-art f1 score on a wsd task with the simple scheme involving sense-annotated training data from peters et al. (2018). ethayarajh (2019 ) generated static embeddings from clm models, using the first pca component sets of token representations. @xcite experimented with averaging tokens for context-agnostic word vectors. this approach performs well on similarity and relatedness tasks compared to traditional static embeddings. however, averaging evaporates much of bert's contextual variation, cutting off its potential to aid in similarity estimation. it's no surprise that averages derived from earlier layers of bert performed best at similarity estimation. later layers demonstrate more contextual variation @xcite , making the mean less meaningful. ## multi-prototype bert embeddings as stated, bert token representations demonstrate greater contextual variation at each progressive layer @xcite . to capture the contextual knowledge of later layers, we construct multiprototype embeddings from many token representations of a single word. the usage types captured by clusters do not always correspond directly to dictionary word senses, but they often discriminate between senses of polysemous and homonymous words, metaphorical senses, as well as syntactic roles and constructions. @xcite . for this reason, countbased multi-prototype models have long been recognized for their usefulness to wsd tasks @xcite @xcite . we observed that natural clusters often capture cognitive affordances. whether or not they are linguistically significant, these loose categories likely play a role in cognitive processes like similarity. to recast our earlier observation: the affordance(s) of water that comes to mind in the comparison of water-land are different from the idea that surfaces when comparing watercoffee. we use clustering to approximate bert's usage-types and the affordances they capture, and demonstrate their usefulness for lexical tasks. ## abstractness and contextual variation to demonstrate the potential of multi-prototype embeddings, we next leverage our model to test a cognitive hypothesis about concreteness. specifically, we examine the difference between abstract and concrete words with respect to spatial dispersion among prototypes. roughly speaking, abstract concepts are classically characterized as those that are "neither purely physical nor spatially constrained" @xcite . concrete words are recognized and comprehended faster (west and @xcite , remembered longer @xcite @xcite , and more resilient to brain damage @xcite ) than abstract words. the dominant explanation for concreteness effects is called the dual-coding hypothesis @xcite @xcite . it holds that concrete and abstract concepts are organized differently in the brain: the former are grounded in experience, while the latter are based solely on other concepts. the sensory richness of concrete concepts explains their memory advantage. significant distributional-semantic research has tested the dual-coding hypothesis, with mixed results (cf. @xcite . dual-coding theory demands discrete types of conceptual representations, and raises an uncomfortable metaphysical issue of whether a concept can be about anything other than the purely physical. embodied views of abstract representation @xcite hold that both linguistic and experiential information contribute to rich representations for all concepts, and that the apparent distinction arises from statistical patterns in the proportion of sensorimotor to affective experiential information undergirding the concepts. the question then arises of how to account for concreteness effects without positing distinct representations. a recent psychological theory contends that concreteness effects are a consequence of 'situational systematicity' @xcite ): abstract concepts are constituted by a larger and more complex set of relationships, dispersed through space and time, and are therefore more subject to contextual variation. if the hypothesis is valid, it might contribute to an explanation of concreteness effects without reliance on the claim that abstract concepts lack experiential grounding: simple, systematic, contextually invariant concepts would naturally be easier to remember. testing the situational systematicity hypothesis demands a way to measure the complexity and contextual variability in concepts. multi-prototype bert meets this demand, at least for modeling variability, which can be viewed a proxy for complexity. single-prototype models cannot be used to test this hypothesis-they lack such fine-grained information about contexts of occurrence. ## conclusion we have presented evidence that the spatially complex lexical representations afforded by clms are useful in type-level lexical modeling, and in investigating type-level semantic questions. we first addressed the common tasks of predicting word similarity and relatedness, and demonstrated that bert produces high-quality multi-prototype word embeddings. we then used these representations test a hypothesis about word-level abstractness, and uncovered a significant connection between this type-level property and the relationships between tokens. multi-prototype embeddings represent word meaning as a constellation of points, as opposed to a single point. the strength of multi-prototype embeddings (and other alternatives such as probabilistic, gaussian, and exemplar-based models) lies in this word-internal complexity. a vector lacks internal complexity-it is infinitesimal. as such, it can only be compared to other vectors on the basis of their location in space. internally complex representations, on the other hand, can be compared on the basis of their internal geometry. when words have a shape, we can ask how the shape of words, or of lexical categories, compare to one another. computational lexical semantics ought to seek out and apply mathematical methods for comparing the higher-order structure or topology of word meaning in such models. future work in this experiment we limit the sample size for each word to 100, and use a fixed k for each word. while increasing the number of data points would perhaps lead to more natural fine-grained clusters, and therefore more sensible k-means clusters for a higher k, we predict that outliers would still lead to the over-prediction of similarities. in order to maximize the contribution to the longstanding debate between exemplar and prototype models for semantic memory, the effects of increased sample size should be evaluated. in addition, the optimal number of prototypes is not the same for every word. we hypothesize that the application of non-parametric and hierarchical clustering methods will demonstrate a tendency towards a relatively low number of prototypes, further validating multi-prototype models of semantic memory as well as improving performance. future work will apply clms to to other lexical tasks and questions, such as metaphor. metaphor interpretation is a context-sensitive process akin to similarity judgment. the rational speech act formalization of metaphor interpretation @xcite utilizes hand-crafted feature vectors; it might be extended by inducing metaphorical sense representations from bert-based prototypes.