Systematic word meta-sense extension (2023)

Abstract: The meaning of polysemous words often varies in a highly productive yet predictable way. Generalizing the regularity between conventional senses to derive novel word meaning is crucial for automated processing of non-literal language uses such as figurative expressions. We introduce a novel task called systematic word meta-sense extension (SWORME) to test and improve language models’ ability to extend word meaning to denote new semantic domains (also called meta-senses) that bear regular semantic relations with existing senses. We found that language models prefer incremental lexical semantic change toward conceptually similar meta-senses such as logical metonymy, and are much worse at predicting highly non-literal meaning extensions such as metaphors. We propose a novel analogy-based method of word meaning extension, and show that it effectively improves language model systematicity in making both gradual and radical types of meta-sense extension. We further demonstrate that learning systematic meta-sense extensions benefits language models on multiple benchmarks of figurative language understanding.

## introduction many words in the lexicon are polysemous in that the same word form can express multiple distinct yet related senses: for instance, some english verbs describing our interactions with physical objects such as get, grasp can also denote the acquisition or distribution of abstract knowledge (e.g. to grasp/get someone's idea); as a result, human speakers are able to extend the meaning of other interaction verbs like steal to form metaphorical expressions such as "to steal information". on the other hand, although recent work suggests that distributed semantic models such as word embeddings and contextualized language models can be applied given two conceptually related semantic domains (e.g. item and information) and usages of polysemous words describing both domains (e.g. the verbs get, grasp, sell that can take both item class and information class nouns as objects), we wish to extend the meaning of another word (e.g. steal with its literal sense only) from denoting one of the two domains to denoting both. to disambiguate related word senses @xcite @xcite and recognize regular relations between lexical items @xcite @xcite , few has investigated whether machines can also productively leverage the detected regularity to generate and understand novel language use in a human-like way. linguists and cognitive scientists have suggested that the extensional processes of many polysemous words from conventional to novel senses are governed by the same set of generative lexical rules @xcite @xcite @xcite and are therefore intrinsically related to each other -that is, word meaning extensions exhibit systematicity, as suggested by both theoretical studies of human cognition @xcite and empirical investigations of word meaning change @xcite @xcite . here we show that neural language models often fail to generate plausible novel word meaning that bears predictable system-atic relations with existing senses, a pattern that is consistent with their poor systematicity in nlp @xcite @xcite and similar failures observed in other domains of machine learning @xcite @xcite . the lack of systematicity in word meaning extension also explains recent findings that language models tend to struggle at processing under-represented figurative expressions including metaphor @xcite , simile @xcite and slang @xcite . a recent line of work has proposed to predict word meaning extension based on the cognitive theory of chaining @xcite , where novel meaning is linked to existing ones due to their proximity in semantic space @xcite @xcite @xcite . however, existing chaining models prefer extensions across literally similar domains with high overlapping in semantic features, while ignoring the relational similarity between word senses that is essential to understanding conceptual and linguistic metaphors @xcite . as a result, chaining models often fail to predict many figurative word senses that share few similar semantic features with literal meaning. we propose a novel task called systematic word meta-sense extension (sworme) to evaluate a language model's ability to predict regular types of word meaning extension in naturalistic context. as illustrated in figure 1 , given two semantic domains that are conceptually related via general cognitive processes such as analogy, we wish to simulate the scenario where a person, after learning usages of polysemous words describing both domains, can leverage the regular relation between them to extend the meaning of a new target word from one domain to the other. inspired by research in analogical inference @xcite @xcite , we introduce a new model that infers novel word meta-sense based on the relational similarity between systematically alternating word meta-senses, which predicts both incrementally and radically novel usages for over 7,300 polysemous english words. ## computational framework in this section, we first introduce the concept of word meta-sense, and formulate regular polysemy as systematic types of meta-sense alternation. next, we introduce the process of partitioning a polysemous word type into multiple hypothetical tokens signifying its different meta-senses to operationalize the scenario of meaning extension toward novel domains. we then define sworme as a task of inferring partitioned token pairs denoting systematically related meta-senses to substitute each other in naturalistic context. we finally introduce methods of learning systematicity in meta-sense extension. ## application to figurative language understanding we finally demonstrate that learning sworme can benefit transformer language models on the task of figurative language understanding (flu). data. we evaluate models on two publicly available datasets of natural language inference (nli) with figurative expressions: the impli dataset by @xcite contains 25,860 figurativeliteral expression pairs, where each literal expression can be either entailed or non-entailed by its paired figurative expression that comes from one of the two classes: metaphors or idioms. the fig-qa dataset by @xcite consists of 10,256 winograd-style questions @xcite , where a model is asked to identify a literal entailment among two candidates for a pair of superficially similar figurative expressions with opposite meaning. the questions in fig -qa can be categorized into four classes based on the type of knowledge required to answer them: objective knowledge (obj), visual metaphors (vis), social understanding (soc), and cultural metaphors (cul). models. we test three off-the-shelf pretrained transformer language models on flu: 1) bertbase-uncased (with 0.11b parameters, pretrained on 40 gb of text) implemented by huggingface @xcite , 2) gpt2-xl (with 1.5b parameters, pretrained on 800gb of text) implemented also by huggingface, and 3) llama (with 7b parameters, pretrained on 1tb of text) implemented by meta @xcite . before flu evaluation, each language model is fine-tuned on the the training set of sworme with α = 0.8 using either associative or analogical chaining objective (usage sentences containing the other 20% word types are left out as the validation set to decide model convergence). for auto-regressive models (gpt2-xl and llama), the contextualized embeddings of a target word is computed only results. ## conclusion we have presented a framework of systematic word meta-sense extension (sworme) that supports lexical items to express new semantic domains in a productive yet predictable way. our results show that the feature associative similarity only predicts incrementally novel meaning, while analogical similarity provides a general account for both gradual and radical types of word meaning extension. we also show that learning analogical chainingbased meta-sense extension improves transformer language model performance on figurative natural language inference.