GPT Perdetry Test: Generating new meanings for new words (2021)

Abstract: Human innovation in language, such as inventing new words, is a challenge for pretrained language models. We assess the ability of one large model, GPT-3, to process new words and decide on their meaning. We create a set of nonce words and prompt GPT-3 to generate their dictionary definitions. We find GPT-3 produces plausible definitions that align with human judgments. Moreover, GPT-3’s definitions are sometimes preferred to those invented by humans, signaling its intriguing ability not just to adapt, but to add to the evolving vocabulary of the English language.

## introduction humans are constantly expanding languages with new words. how are artificial language models, which are increasingly deployed 'in the wild', to handle the stream of neologisms that are appearing in slang or on social media @xcite )? today's most advanced language models, including @xcite , use a subword tokenization of input text, rather than consuming it word by word. this allows them to process words never seen in their training data. for example, the word 'perdetry', which has never been used in english, is treated by gpt-3 as a sequence of two tokens (fig. 1 ). the subword tokenization algorithm is designed for text compression and does not respect the natural morpheme boundaries. we explore gpt-3's understanding of english at the subword level by prompting it to give definitions of nonce words 1 (fig. 1 ). we find in human 1 we use the term 'nonce word' for a new word not used in english. it becomes a 'neologism' once it acquires a meaning. studies that not only does gpt-3 generate realistic, original meanings for new words, but its definitions are sometimes preferred to those invented by humans. this finding sheds light on gpt-3's ability to adapt to and even extend a changing vocabulary. while we cannot ascertain gpt-3's exact reasons for assigning meanings to nonce words, our results prove that these reasons are not limited to morphology: many neologisms have no clear roots or derivational origin. the meanings of words may be imported by their phonological qualities -more precisely, their orthographic realizations -or by clues to their membership in certain lexical strata. thus, at a high level, our findings suggest that gpt-3 has learned not only its world knowledge and capacity for long-range reasoning in text @xcite , but also the nuances of etymology and the correspondences of sound and meaning that lie at the very base of language understanding. below are some pairs of words together with their definitions. the goal is to guess, for each pair, which word goes with which definition. we will show you two options, and you will decide which of them is a better match. the words you'll get are rare, and we do not expect you to know many, or indeed any, of them. make your best guess. for some pairs, there is no correct answer. we'll show you the expected answers at the end. do not look up the words while doing the task: we are really interested in your gut feeling, right or wrong. a. recommor : a female dwarf caraber : a male witch; a wizard; a warlock b. recommor : a male witch; a wizard; a warlock caraber : a female dwarf ## related work the notion that some subword elements (phonesthemes) carry meaning, but, unlike morphemes, do not play a part in word formation has caused controversy in linguistics for over a century @xcite . in his seminal work, de saussure (1916) rejected this notion. yet, later work identified a large set of english phonesthemes, such as the cluster /gl/ in 'glow', 'glitter', 'gloss', etc. meaning "light"; a notable list was compiled by @xcite . recent studies found phonosemantic patterns that are common to many languages @xcite . in practice, words are even engineered for subconscious reactions: certain sounds in brand names are correlated with associations such as size (of a gadget) or speed (of a courier) @xcite . our study suggests that gpt-3 may understand such patterns as well. there is a body of work on joint modeling of (orthographic or phonological) word forms and grammatical classes such as noun gender and inflection pattern. in a recent study, @xcite used neural models to measure mutual information between meanings and inflection classes of czech and german nouns, which, for borrowed words, often depend on the language of origin. it is plausible that gpt-3 implicitly uses likely source languages of nonce words to generate meanings associated with some lexical strata, e.g., abstract nouns from norman french, concrete nouns from the germanic substrate, and artificially constructed terms with greek or latinate elements. (we direct the interested reader to the lexicon in appendix c.) work on neologisms in nlp includes tracking their emergence and spread on the internet @xcite , mapping them into embedding spaces @xcite @xcite , and codifying and predicting etymologies @xcite . others have studied definition generation @xcite and the reverse task of mapping definitions to words @xcite , albeit with pretrained embeddings. limited examples of a pretrained model's use of nonce words appear in @xcite . in this work, we study gpt-3's ability to define words never seen in context. ## creation of new words and meanings we trained a lstm model @xcite on a corpus of english words foot_0 with a standard character-level objective, then sampled strings from the lstm to create nonce words. the words were lemmatized and assigned parts of speech (pos): noun (n.), verb (v.), or adjective (adj.). to produce definitions for these words, we generated text from gpt-3, primed with input in the format "word (pos.) -". usually, gpt-3's outputs had the style of a dictionary definition (fig. 1 ). the definitions were filtered by common-sense criteria and lightly edited for consistency, as explained in appendix a. by this procedure, we obtained 146 word-definition pairs (67 n., 47 v., 32 adj.). for comparison in our study, we also sampled a set of real but rare english words from a corpus. 2 definitions for these words were taken from a dictionary. 3 this resulted in a combined set of 220 words (102 n., 70 v., 48 adj.), with a 2:1 ratio of fake to rare words in each pos. see appen-n. v. adj. human fake-fake 70.7% 59.8% 64.3% fake-rare 72.6 60.9 65.3 rare-rare 79.6 65.3 69.8 ## conclusion a character-level model of english words composed with gpt-3 is a complete scheme for generating new words and innovative meanings. gpt-3 invents definitions for words it has not seen in training that are seen as reasonable by humans. these results have implications for language models' ability to adapt and even add to an evolving vocabulary. they can inspire future work on machine understanding of new slang, optimization of words and acronyms, creation of fictitious entries, and automatically generating word games.