When Polysemy Matters: Modeling Semantic Categorization with Word Embeddings (2022)

Abstract: Recent work using word embeddings to model semantic categorization have indicated that static models outperform the more recent contextual class of models (Majewska et al, 2021). In this paper, we consider polysemy as a possible confounding factor, comparing sense-level embeddings with previously studied static embeddings on both coarse- and fine-grained categorization tasks. We find that the effect of polysemy depends on how one defines semantic categorization; while sense-level embeddings dramatically outperform static embeddings in predicting coarse-grained categories derived from a word sorting task, they perform approximately equally in predicting fine-grained categories derived from context-free similarity judgments. Our findings highlight the different processes underlying human behavior on different types of semantic tasks.

## introduction a great deal of work has been devoted in recent years to creating computational models of meaning @xcite @xcite @xcite . such models have been evaluated on a variety of semantic tasks, from word pair similarity judgments to document classification. one task that has received relatively little attention is semantic categorization. besides making pair-wise judgments about the similarity between two words, humans can also reason about higher-order structures; we can tell not only that robin and sparrow are similar to each other, for example, but also that they belong in a group with other birds (e.g. ostrich and pigeon). based on the impressive performance of embedding models on other semantic tasks, we expect such models to excel at identifying semantic categories as well. our particular interest is on the role of polysemy in semantic categorization. because words generally have multiple distinct senses, categorization decisions will depend on which sense of a word is being considered. representing the distinct senses of polysemous words, then, should be important to modeling how humans categorize words. for this reason, we expect contextual embeddings, which represent each instance of a word in context as a unique embedding, to model semantic categorization better than static models, which conflate every use of a word into a single representation. but, in fact, recent work evaluating different word embedding models on verb categorization suggests just the opposite; @xcite found that contextual models perform poorly compared to older static models. in the following paper, we challenge this result. first, we extend the evaluation from @xcite , who compare word embedding clusters to coarse-grained semantic categories generated by humans in a word sorting task, by evaluating sensespecific embeddings in addition to the static embeddings previously reported. we find that retaining sense-level information from contextual bert embeddings more than doubles its f1 score, outperforming static embeddings by a large margin. this result suggests that the reported under-performance of bert in @xcite was due not to the irrelevance of context to categorization or an inherent weakness of contextual embedding models, but rather to the fact that information about polysemy was thrown away in generating static embeddings from contextual models. next, we evaluate the same set of models on fine-grained categorization, using categories derived from human similarity judgments. contrary to the coarse-grained setting, we find that static and contextual models perform about the same in predicting fine-grained categories. we surmise that humans use different cognitive processes to perform word sorting vs similarity judgment tasks. choosing the best word embeddings thus depends on the type of behavior one is trying to model. since both static and contextual embeddings have been shown to model pairwise similarity between words well @xcite , and since similarity is a primary criterion for categorization, it seems intuitive that word embeddings should perform well at categorization tasks. some previous work supports this intuition; word embeddings have excelled at word sense disambiguation @xcite @xcite and topic modeling @xcite when cast as categorization problems. in the present paper, we are interested in semantic category induction. instead of grouping instances of a word into distinct senses, or documents into topics, the goal of semantic categorization is to group unique words into semantically related clusters. this more abstract type of categorization has received less attention in the word embedding literature; a few probing studies have tested whether different models encode a pre-defined set of categories @xcite @xcite , but in all cases these categories were stipulated by the researchers and had not been experimentally validated. @xcite recently published a more empirical categorization dataset, based on judgments from non-expert native speakers, rather than stipulated by trained researchers. the dataset, spa-verb foot_0 , contains data from two tasks. the first is a sorting task, where participants grouped a set of verbs into broad semantic classes. the second task involves spatial multi-arrangement, which provides finer-grained judgments about the similarity between words within a single semantic domain. spa-verb is valuable as an evaluation resource for modeling categorization because it allows for a more direct comparison between human categorization behavior and model behavior than previous datasets. also, spa-verb contains 825 verbs in 17 semantic classes, which is much more comprehensive than other available category datasets. most of the verbs in spa-verb are polysemous. while many words belong to more than one class (corresponding to distinct senses of those words), the dataset has so far only been used to evaluate static word embeddings (either from static models or extracted static representations from contextual models). our goal with the following study is to find out when polysemy matters in modeling natural language semantics, in particular, whether sense-specific representations are better predictors of human behavior on some semantic tasks, but not others. ## models below we describe the word embedding models we evaluate on spa-verb: ## evaluation to evaluate the performance of each model on the ground truth classes, k-means clustering is used to group verbs into predicted classes. we use the same metrics as @xcite : modified purity and weighted class accuracy are combined in an f1 score, calculated as their balanced harmonic mean. modified purity is the mean precision of predicted clusters, while weighted class accuracy targets recall (see appendix b). because mpro bert has multiple representations for a single word, the same word form may show up more than once within a single cluster. to prevent artificially inflating the recall in evaluating mpro bert, we eliminate duplicates within each cluster before evaluation. ## coarse-grained categorization next we describe our evaluation of each model on coarse-grained categorization. ## fine-grained categorization next, we examine how word embeddings fare on finer-grained categories. we speculated that given a smaller, more focused set of words, there is less ambiguity about the relevant criteria for categorizing words, and so evaluating word embeddings on fine-grained categorization may be a better test of model quality than coarse-grained categorization. this section describes how we created a benchmark for fine-grained categorization from the spa-verb phase 2 data, and evaluated the same models on this new benchmark.