Exploring Category Structure with Contextual Language Models and Lexical Semantic Networks (2023)

Abstract: The psychological plausibility of word embeddings has been studied through different tasks such as word similarity, semantic priming, and lexical entailment. Recent work on predicting category structure with word embeddings report low correlations with human ratings. (Heyman and Heyman, 2019) showed that static word embeddings fail at predicting typicality using cosine similarity between category and exemplar words, while (Misra et al., 2021)obtain equally modest results for various contextual language models (CLMs) using a Cloze task formulation over hand-crafted taxonomic sentences. In this work, we test a wider array of methods for probing CLMs for predicting typicality scores. Our experiments, using BERT (Devlin et al., 2018), show the importance of using the right type of CLM probes, as our best BERT-based typicality prediction methods improve on previous works. Second, our results highlight the importance of polysemy in this task, as our best results are obtained when contextualization is paired with a disambiguation mechanism as in (Chronis and Erk, 2020). Finally, additional experiments and analyses reveal that Information Content-based WordNet (Miller, 1995) similarities with disambiguation match the performance of the best BERT-based method, and in fact capture complementary information, and when combined with BERT allow for enhanced typicality predictions.

## introduction the empirical success of contextual language models (clms) @xcite @xcite has led to much research analyzing their functionality @xcite @xcite and how they acquire semantic and world knowledge @xcite . it also raises the question of their plausibility as models of human semantic memory @xcite @xcite . the study of categorical knowledge, in particular typicality, provides a window into this question @xcite . as initially observed by @xcite , native speakers of english consider that certain exemplars (e.g., robin, crow) are more representative than others (e.g., penguin, ostrich) of a conceptual category (birds). that is, categorical knowledge is organized along a graded structure. according to prototype theory @xcite , the structure follows from the fact that properties frequently shared among category members tend to be integrated into its prototype. the main question we raise in this paper is whether distributional models, and clms in particular, are indeed aware of category structure, as captured in existing human typicality norms for english. the broader question for cognitive science is whether or not this type of knowledge can be learned through text-based exposure alone, thus contributing to the larger debate on the role of language in learning semantic knowledge @xcite . but there are also more practical motivations for this work, as many nlp tasks (such as information retrieval, question answering, natural language inference) can arguably benefit from typicality relations by understanding which exemplars are more relevant to particular concepts. previous work on this topic provide mostly negative results. @xcite use classical static embeddings and represent typicality scores between categories and their exemplars as the cosine distance between their corresponding word embeddings. they conclude that static embeddings poorly account for human typicality scores, as obtained from @xcite . more recently, @xcite use various clms to predict typicality. while clms have a larger, more expressive parameter space and are tuned over larger corpora with arguably better learning objectives than static word embeddings, @xcite report only slightly better, and still modest, correlations with human typicality judgements (in this case, @xcite ). their probing approach uses a cloze test formulation over taxonomic sentences (e.g., a robin is a bird). we revisit this question by first introducing an expanded suite of methods to extract typicality scores from clms, some of which improve correlations with human norms, showing the importance of using the right probe for typicality. how to reliably and efficiently probe clms is still an open question, including supervised and unsupervised approaches @xcite @xcite . the problem is exacerbated by the fact that typicality is a relation between pairs of concepts, abstracted away from contexts, while clms produce representations for contextualized word tokens (not types). we consider a wider array of approaches, all unsupervised, for predicting typicality from clms, using bert as representative test case. our probing approaches fall into two main classes: (i) bert as a probabilistic language model, and (ii) generating "static" word embeddings from contextual bert embeddings. our second contribution is showing the importance of polysemy (e.g., orange can be a fruit, a color, or a company) for typicality, an issue that has been largely overlooked in previous works. when judging category-exemplar pair typicality, humans arguably don't consider all the senses of the category and exemplar words. instead they consider the senses of the category and exemplar that are most compatible with one another, in effect performing a joint disambiguation. polysemy is problematic for static word embeddings as they collapse word senses into a single vector. clms, on the other hand, provide some form of sense selection through contextualization @xcite , but there are many possible ways to use clms and to provide contexts to these models (such as sampled or taxonomic sentences). we find that using a simple disambiguation mechanism, such as the method of deriving multi-prototype embeddings introduced in @xcite , leads to typicality predictions more closely correlated with human rankings. finally, another research question we address in this paper is whether categorical structure is present in lexical semantic networks, such as wordnet @xcite . the hierarchical concept organization in wordnet was indeed initially inspired by theories of human semantic memory @xcite . note that polysemy is also an issue in using wordnet, as we need to map category and exemplars to specific synsets. however, using the structure of wordnet, a simple disambiguation heuristic, and taking into account word frequency information leads to competitive results. further-more, we find that wordnet-based and bert-based typicality predictions contain complementary information, and creating a simple ensemble method of the two further increases performance. in summary, our main contributions are: ## related work recent years have seen an ever growing body of work on assessing whether distributional models constitute realistic models of human semantic memory, within both nlp @xcite and cognitive science @xcite @xcite @xcite . in this context, the study of categorical knowledge, and specifically graded typicality, plays a crucial role, given it is one of the most reliable findings in the study of human categorical knowledge @xcite . the first work on using distributional models to predict typicality scores is @xcite , who estimate these scores, in english and dutch, using cosine distances between the static embeddings of the exemplar and category words. these show that static embeddings, whether trained in a counted-or prediction-based fashion, yield poor correlations with human typicality norms. on the related task of lexical entailment (le), @xcite evaluated different unsupervised and supervised methods that use static word embeddings. they also found a significant gap between static word embeddings and human norms. while le is closely related to typicality, it differs in that le predicts the relationship between two words, while typicality compares the relationship between two concepts (in which words are used as a proxy, in our case). also, le ratings are not restricted to pairs of words belonging to the same category. we differ from these earlier work on typicality by considering more expressive clms instead of static embeddings. the most closely related work is @xcite who probe a wide range of clms (i.e., different variants of bert and gpt) for predicting typicality scores: these are extracted using a cloze task formulation over hand crafted taxonomic sentences (e.g. a robin is a bird.). they report better correlation scores with human norms, although still modest, which they take as indication that text exposure is not sufficient to learn category structure. our focus in this work is different and somewhat broader in that we consider a wider array of probing methods of clms, though restricted to unsupervised ones. supervised probing approaches, based on classifiers taking clm representations as inputs, are problematic as they provide a less direct probing approach, possibly adding additional confounds @xcite . another distinctive aspect of our work is that we compare clms to lexical networks like wordnet @xcite . furthermore, we study the impact of polysemy in typicality, which was not controlled in @xcite or in @xcite . the work of @xcite shares some similarities with @xcite : they also use bert under a cloze task setting to study typicality, but they focus on identifying whether the model has access to prototypical properties of concepts (e.g., a ball is round). ## problem setting and framework this section presents our general framework for assessing whether a lexical representation model (i.e., a clm or semantic network) is aware of category structure; the specific probing approaches will be described in sec. 4. let us assume a generic conceptual space consisting of a set c of categories as well as a set e of exemplars of these categories. for each category c in c, we assume a set of n c exemplars e c i in e for @xmath0. following the findings of @xcite , we posit that this conceptual space has a graded structure, in that certain exemplars (e.g., robin, crow) are more closely representative of their category (e.g., birds) than others (e.g., penguin, ostrich). this can be expressed through a typicality score t c @xmath1, often an ordinal scale, for each e c i , where t c @xmath2. these typicality scores can be obtained from human subjects through various kinds of stimuli, such as direct scoring of category-exemplar word pairs (e.g., robin-bird) and scoring of a taxonomic sentence (e.g., a robin is a bird.)(see sec. 5.1). to assess whether a lexical representation model is aware of such a structure, three main components are needed. as the conceptual space is latent, we first need a concept-to-word mapping function from category and exemplar concepts to words. as is done in human studies and previous work, we assume that categories and exemplars are reliably accessed through their corresponding singular words: the concepts robin and bird are accessed via the words robin and bird, respectively. this is a simplification, as the same concept can be realized by plural forms, as well as by different (synonymous) words. given this functional mapping, we will conflate a category c with its category word c and an exemplar e c i with its exemplar word e c i . second, we define a typicality scoring function over exemplar-category word pairs, which captures how typical the exemplar is of the category. each model will assign a typicality score s c i to each category c and exemplar e c i pair. some of our methods will make use of a text corpus, denoted s, taking the form of a pre-extracted set of sentences containing exemplar and/or category words. this corpus s can be viewed as an extra parameter of the typicality scoring function, as it will directly impact the predicted typicality values s c i . such a corpus is required to be able to fully leverage clms, whether they are used as probabilistic language models or to produce word vector representations. it is also through this corpus that one hopes to capture contextual word realizations that approximate the different senses associated with the exemplar and category words. indeed, we hypothesize that one important aspect of assessing typicality between category and exemplar words is to be able to deal with the polysemy of these words, predicting that the best methods should be able to disambiguate these words in such a way that the exemplar word is interpreted in a category-compatible sense and the category word in an exemplar-compatible sense. using wordnet synsets as a proxy for senses, we find that on average, category words are linked to 4.2 and 4.9 senses, while exemplars are linked to 3.3 and 4.7 senses on the typicality datasets of @xcite and @xcite , respectively foot_0 (see sec. 5.1 for dataset details). this shows the depth of the polysemy problem for typicality, as each word on average has multiple meanings. finally, we need an evaluation metric for measuring how well the predicted typicality scores s c i are able to mirror the human judgements t c i . the most obvious choice for this metric is to use spearman rank correlation @xcite , as this correlation measure is non-parametric and makes no strong (e.g., linear) assumption on the data distribution or the underlying scoring functions. ## methods we distinguish two classes of methods, depending on whether they use clms or word-net. for clm-based methods, we use bert (bert-base-uncased specifically) as a prototypical clm to assess whether clms are aware of category structure. the framework and methods easily generalize to other clms. while the concept-toword mapping is the same for these methods, the scoring function and the probing corpus are different. the bert methods fall into two categories: (i) those that use bert as a language model (bert-mlm, bert-sentemb, bert-mlm-taxo), and (ii) those that extract word vectors from bert (bert-avg, bert-mpro). note that these are all unsupervised probing methods: when an additional corpus s is used, it is only used to probe bert, not to fine-tune it. these methods embody different linguistic hypotheses and adopt different ways of dealing with polysemy. note that for embedding methods, we test all layers. secondly, we consider lexical semantic networks to compute typicality scores, using wordnet @xcite specifically. for this, we use the shortest-path and the lin @xcite similarity measures computed on wordnet @xcite . we compute the information content (ic) values @xcite used in the lin measure using the additional corpus s. we deal with polysemy by using the maximum similarity between exemplar and category synset pairs to compute typicality scores. ## combining clms and lexical networks given that wnic and bert-mpro rankings achieve the highest correlations, questions arise: how similar are the rankings produced by the two methods? if sufficiently different, how can we combine them? to answer these questions, we perform a series of analyses. we constrict this depth-first analysis to just the younger adults m&d dataset. ## conclusion we show that bert and wordnet-based methods are able to outperform previous methods in typicality prediction, but only with a disambiguation mechanism, either implicit through the selection of contexts or explicit via the estimation of distinct word senses. these results emphasize the importance of polysemy in assessing typicality, an issue that had been overlooked in previous work. we also show that bert and wordnet provide complementary information that are relevant to modeling category structure: a simple ensemble of the two leads to the best correlations with human judgements. we plan to further analyze the differences between clms and wordnet in typicality, and find more sophisticated ways to inject semantic knowledge into clms. also, we want to use datasets in other languages, to find if the results generalize.