Exploring Nominal Coercion in Semantic Spaces with Static and Contextualized Word Embeddings (2022)

Abstract: The distinction between mass nouns and count nouns has a long history in formal semantics, and linguists have been trying to identify the semantic properties defining the two classes. However, they also recognized that both can undergo meaning shifts and be used in contexts of a different type, via nominal coercion. In this paper, we present an approach to measure the meaning shift in count-mass coercion in English that makes use of static and contextualized word embedding distance. Our results show that the coercion shifts are detected only by a small subset of the traditional word embedding models, and that the shifts detected by the contextualized embedding of BERT are more pronounced for mass nouns.

## introduction the literature in formal semantics has debated for long on the distinction between count nouns and mass nouns, which has often been described as an opposition between discrete, countable objects and substances that cannot instead be divided into subunits. a notorious formal characterization of this intuition is provided by @xcite : mass nouns like wine are non-quantized, in the sense that each subpart of wine will still count as wine; on the other hand, count nouns like cat are quantized, because if you take a subpart of a cat, it will not count as a cat @xcite . according to such view, in other words, the two types of nouns denote in different domains with different properties. @xcite proposed instead a lexicalist perspective on the problem, where nouns are marked with a binary @xmath0. although the approaches adopt different criteria for defining the "countability" of the nouns, they both predict that count nouns will (mostly) appear in count contexts, and mass nouns will (mostly) appear in mass contexts. however, cases like the following are extremely frequent in natural language: since it is rare to find nouns that occur exclusively in either mass or count contexts, it makes more sense to talk about predominantly count and predominantly mass nouns. @xcite describes the idea of mass-count elasticity, meaning that any noun can be in principle mass or count, its status being determined at the level of the nominal phrase. when we say "predominantly" mass or count noun, therefore, we mean that a noun has the tendency to occur more frequently in one of the two context types. on such basis, the count-mass distinction can be intuitively seen as a continuum, with the nouns traditionally described in the literature being closer to one the two extremes @xcite . in this work, we investigate to what extent modern distributional semantic models -which are nowadays the standard for the representation of lexical meaning in nlp-encode the meaning shifts caused by mass-count coercion. we run two different experiments, making use respectively of static and contextualized word embedding models to identify the meaning shifts, and we study some of the potential factors that might influence the extent to which a noun is shifting. ## related work modern nlp widely adopts distributional semantic models (dsms) for the representation of lexical meaning, using vectors that are based on the cooccurrences patterns of words in large text corpora. vector representations are usually compared using the cosine of the angle between them, and the smaller the angle between two words, the closer their meanings will be @xcite . the literature on dsms has identified three generations of vector spaces @xcite . the first generation is typically referred to as count models @xcite , because the spaces are obtained from the extraction of co-occurrences between the target words and the linguistic contexts that are deemed relevant, then the co-occurrences are weighted via associations measures @xcite @xcite . a second family of models emerged in the early 2010s and became known as word embeddings or prediction-based models @xcite . in such models, the learning of word vectors is generally framed as a supervised task: a neural network is trained to predict words given other context words, and the vectors are learned as parameters. words that tend to cooccur will have similar vector representations. however, a common feature of both families is that they produce static vector representations, in the sense that each word gets represented as a single vector, which makes it difficult to handle cases of ambiguity and polysemy. the most recent generation of distributional vectors is said instead to be contextualized, because word representations are generated in context on the basis of the activation states of a neural language model @xcite . one of the advantages of models like bert @xcite ) is that they allow generating a specific word embedding for each context in which target words occur, making them an interesting option for modeling contextual phenomena such as nominal coercion. concerning the modeling work on nominal coercion in distributional semantics, @xcite were the first, to our knowledge, to use dsms to investigate the phenomenon. they considered pluralisation as a proxy of count usage, and built a traditional count model with separate vector representations for the singular and the plural of a list of candidate mass and count nouns. consistently with their initial hypothesis, they found that the vector similarity between singular and plural is higher for count nouns than for mass nouns, since the latter undergo a meaning shift when they are pluralized (cf. example 2 in section 1). @xcite later analyzed the factors affecting the similarity scores in the data by @xcite , reporting that abstract and highly polysemous nouns undergo greater semantic shifts as a consequence of pluralization. both these works are close in spirit to our research: in our first experiment, we will use several types of word embedding models to compare the distances between singular and plural forms of mass and count nouns; in our second experiment, we will use the contextualized vectors of bert to observe how coercion changes the semantic representations of the nouns in mass and count contexts, which we automatically extract from the british national corpus @xcite . to our knowledge, this is the first study specifically on mass-count nominal coercion including both static and contextualized embedding models, although other types of coercion have previously been investigated in the literature on dsms, e.g. complement coercion @xcite @xcite @xcite or classical metonymies (container-for-content, producer for product etc.) @xcite . ## experiment 1: comparing the singular-plural similarity in static embedding spaces in our first experiment, we follow @xcite in considering pluralisation as a reliable proxy of count usage and we compare the distributional representations of singular and plural forms of candidate mass and count nouns across the most popular word embedding spaces in the literature. if a model is able to detect the coercion meaning shift, then we expect to see that the average semantic similarity between singular and plural forms is lower for the mass nouns (see example 2 in section 1). we use a list of predominantly count and predominantly mass nouns introduced in the same paper, identified via the selection of syntactic contexts: the vectors of the target nouns and their corresponding plural forms are firstly extracted from different word embedding spaces. our pool of models includes the following vector spaces: cbow vectors @xcite , one model with window size 2 (cbow-w2) and one with window size 10 (cbow-w10); skip-gram vectors @xcite , one model with window size 2 (sgns-w2) and one with window size 10 (sgns-w10); glove vectors @xcite , one model with window size 2 (glove-w2) and one with window size 10 (glove-w10); fasttext vectors @xcite , one model with window size 2 (fasttext-w2) and one with window size 10 (fasttext-w10). finally, we also include two variants of the skip gram where the contexts are selected via syntactic dependency with the target word @xcite , one with untyped dependencies (sgns-synf) and one with typed dependencies (sgns-synt) (e.g. in the first case, given the target dog and the context big dog, the model will just use the syntactic neighbor big as a context, while the second will also include the type of syntactic relation linking the two words, i.e. adjectival modifier). all models have been trained with default hyperparameters on a concatenation of the ukwac @xcite , of the british national corpus and of a 2018 dump of wikipedia foot_0 , and the semantic similarity is estimated via the classical cosine metric. we report the average similarity scores between singular and plural forms for both mass and count nouns in in our second experiment, we extract sentences in which our mass and count nouns occur from the british national corpus, and we use the patterns described in @xcite to divide them into mass contexts and count contexts, and then we use the bert model to compare their contextualized representations. using bert allows us to take into account a wider variety of contexts rather than just using a pair of vectors for the singular and plural forms. in this case, we expect that both types of nouns, when they occur in different context types, will have a lower semantic similarity, because both of them will be undergoing semantic shifts (mass to count or count to mass). the selected patterns are the following: in both contexts, the nouns are excluded if followed by another noun, adjective, or participle to avoid selecting noun-noun compounds. in mass contexts, we also exclude the cases where the target nouns directly follow a participle to prevent misclassification of the participle noun phrases, e.g. the baked cake. to increase the reliability of the sentences for the experiment, we manually filter the sentences containing cases of idiomatic usages, e.g. day by day. as a result, we extract a total of 614512 sentences. only the candidate mass or count nouns occurring at least 20 times in both mass and count contexts are considered. generally, count nouns have a higher average frequency in both contexts, and both count and mass nouns have a higher average frequency in count contexts. the frequency of count nouns in count context ranges from 49218 (time) to 6148 (role), with a mean of 27787.47, while the frequency in mass context ranges from 8211(time) to 20 (role), with a mean of 1245.53. for mass nouns, the width of frequency in mass context is from 9259 (part) to 45 (recipe) and the average frequency is 2110.43, whereas the mass nouns in count contexts have an average frequency of 6414.19, a maximum of 49598 (time) and a minimum of 34 (information). notice that nouns can, in principle, occur both as count and as mass nouns, and their frequencies are computed separately as they have been extracted with different patterns. among the target nouns, time and opportunity appear as both candidate count noun and candidate mass noun. although the nouns may be argued as ambiguous, the syntactic patterns used to extract them are unambiguous and can correctly reflect their usage in the count contexts and mass contexts. foot_2 therefore, they could still be included to compare the meaning shift a noun undergoes in the transition from the 'standard' context to coerced context. then we use the bert-base-uncased model and the minicons python library @xcite 3 to generate semantic representations of the target nouns in context: the idea is to measure the similarity scores of each (mass or count) noun to itself for randomly sampled sentences. we carry out the sampling either i) by selecting context pairs where the target noun occurs in both cases in its mass, or in its count contexts (within the same context type, which could be either count or mass); or ii) by selecting context pairs where the target noun occurs once in a mass context and once in a count context (between context types). this means that each noun type will have its occurrences sampled in three different ways: the similarity comparison between 1) and 3) is the most relevant one for our study: we expect that similarities in 3) to be much lower than in 1), to an extent proportional to the meaning shift that the noun is undergoing. for each noun, we repeat the sampling 10 times from each group, and for each time we randomly extract 10 different context pairs to generate the vectors. notice that, differently from a big part of the literature, we use spearman's rank correlation and not the cosine as a similarity metric for bert vectors. our choice is motivated by recent findings about the anisotropy of contextualized vector spaces, where a small number of 'rogue' dimensions dominate the cosine similarity scores @xcite . @xcite showed that using postprocessing techniques like normalization or rank-based metrics such as spearman's rank led to much better correlations with human similarity judgments. moreover, rank-based metrics have been previously proven to be more robust than cosine in several similarity-related tasks @xcite @xcite . the results of spearman's rank correlation experiment are reported in let us illustrate the statement with an example for the predominantly count noun problem and an example for the predominantly mass noun water, respectively. the noun problems in s1 and s2 refer in both cases to a specific issue that needs to be resolved, while problem in s3 seems to be more generic and more similar to trouble. accordingly, the correlation of s1 and s2 is 0.55, and the correlation of s1 and s3 is 0.33. the correlation difference between the s1-s2 pair and the s1-s3 pair should reflect the meaning shift that problem undergoes, changing from its standard "count" meaning to its coerced interpretation in a mass context. the noun water in s4 and s5 is used in mass contexts and it has the standard meaning of i) water as a liquid that can be drunk (s4); and ii) the amount of water usage in a hydraulic system (s5). on the other hand, in the last context (s6) waters is rather referring to a specific geographical/territorial unit. the correlation of the s4-s5 pair is 0.59, while the correlation of the s4-s6 pair is 0.41, and the correlation difference between the two pairs should reflect the meaning shift from mass to count usage. it is immediately evident from table that 3 count nouns generally have a lower average spearman correlation score than mass nouns in either count or mass contexts, suggesting that the cluster of count nouns is less compact and their meanings are more varied and scattered across the semantic space. it should be noticed that many of the count nouns are highly frequent (e.g. child, thing, way, man, one, place, time, day all have more than 10k occurrences, more than any mass noun in our data), therefore they might display much more contextual variation in their usage, which could explain the relatively low similarity value. indeed, the similarity of the count nouns when the contexts are sampled from the count or the mass groups does not differ significantly, with the latter being even slightly higher. for both count and mass nouns, as we expected, we observe a lower similarity when the contexts are sampled from both groups (see also figure 2 ): the average difference with the sampling within the same group is, respectively, of 0.1 and 0.16 correlation points, confirming the finding that mass nouns typically undergo a greater meaning shift @xcite even when patterns beyond pluralization are taken into account. it is also noticeable that mass nouns have a relatively high similarity when they are sampled in count contexts, which could be explained by the fact that many of these nouns have systematic secondary meanings that are more compatible with a count usage (nouns denoting drinkable liquids are typically undergoing a shift from the liquid to the container, e.g. @xmath1, or from the liquid to the variety, e.g. @xmath2. in sum, the results of our experiment provide further support to the view that the mass/count distinction should be seen as a continuum, and that the syntactic context is the strongest cue to the type of denotation @xcite . moreover, even with more varied mass and count contexts than the ones used in previous studies @xcite , we also find that coercion makes mass nouns undergo a greater semantic shift than count nouns. we also analyze some of the factors mentioned as relevant by @xcite to predict the meaning shift of the nouns: frequency, polysemy, and concreteness. for polysemy, we simply use the wordnet synsets @xcite of a noun as an indicator of the number of word senses, while for concreteness we use the values from the english norms by @xcite . unfortunately, we do not find any significant correlation between the average differences in the spearman correlations that we computed with bert and the above-mentioned factors, probably because of the small size of our set of nouns. highest average correlation difference). despite the lack of significance of the correlation scores, it can still be noticed that: i) regarding polysemy, the most shifting nouns tend to have a relatively high number of word senses; ii) as for concreteness, the most shifting count nouns have relatively high values, while the most shifting mass nouns tend to denote more abstract entities. more studies with a larger set of predominantly mass and count nouns are needed to confirm the finding. ## conclusion in this paper, we have presented two experiments on modeling nominal coercion of mass and count nouns with two different typologies of distributional semantic models. in the first experiment, we compared the vector representations of singular and plural mass/count nouns across several popular word embedding models. perhaps surprisingly, we found that i) the count-based glove models and ii) the word2vec-like models with larger contextual windows were the most successful in identifying significant differences between singular and plural representations of mass nouns, whose meanings shifted more when they were pluralized, while the most of the other models did not detect any shift. we hypothesized, therefore, that such semantic shifts are better captured by semantic spaces that focus on modeling similarities of topic/domain, rather than similarity of co-occurrence in the same local contexts. in the second experiment, we compared the vectors generated by bert in different context types. we found that the self-similarity of the nouns sharply decreased when contexts of different types were sampled to generate the contextualized representations and that the shifts of predominantly mass nouns were more pronounced. our qualitative analyses suggested that factors such as polysemy and concreteness of the nouns might play a role in predicting semantic shifts, although more studies with a larger set of nouns are necessary. another promising direction for future research would be using dsms to model the effects of nominal coercion in human sentence processing, since psycholinguistic studies proved that, in several languages (e.g. english, german, mandarin chinese), coerced nouns lead to increased reading times and longer eye fixations @xcite @xcite @xcite . in this sense, integrating dsms-derived similarity metrics in the current computational models could lead to better estimation of reading difficulties induced by coercion operations.