Prompting C hat GPT to Draw Morphological Connections for New Word Comprehension (2023)

Abstract: Though more powerful, Large Language Models need to be periodically retrained for updated information, consuming resources and energy. In this respect, prompt engineering can prove a possible solution to re-training. To explore this line of research, this paper uses a case study, namely, finding the best prompting strategy for asking ChatGPT to define new words based on morphological connections. To determine the best prompting strategy, each definition provided by the prompt was ranked in terms of plausibility and humanlikeness criteria. The findings of this paper show that adding contextual information, operationalised as the keywords ‘new’ and ‘morpheme’, significantly improve the performance of the model for any prompt. While no single prompt significantly outperformed all others, there were differences between performances on the two criteria for most prompts. ChatGPT also provided the most correct definitions with a persona-type prompt.

## introduction in recent years, large language models (henceforth llms) have become increasingly more powerful @xcite . however, in terms of training, llms have large power consumption @xcite . this results in large expenditures of energy and resources if the models are periodically re-trained to have updated information, such as an updated vocabulary. a potential alternative to re-training is prompt engineering, namely designing better input prompts to get heightened performance @xcite . through prompt engineering, considered even a programming method by @xcite , a model applies previous acquired knowledge to new tasks, thereby increasing its performance. to research the advantages of prompt engineering in terms of updating vocabulary, the current study looks for the best prompting strategy in a case study, namely employing morphological connections for new word comprehension. we define morphological connections as extrapolating the definition of unfamiliar words by use of morphological knowledge, i.e. morphemes. thus, our research question is "what prompting strategy best allows an llm to make correct morphological connections in a context of zero-shot learning?". this study is not relevant only for novel word comprehension, but also for the task of morphological decomposition. for example, @xcite showed evidence that bert can abstract over grammatical categorisations, while @xcite showed some models, such as gpt-2, use derivational morphemes in the generation of novel words, being able to generalise and use morphemes in a compositional manner. however, though llms might hold knowledge on abstract morphological categories, they might not always exploit this knowledge @xcite , given such models are not task-specific like decompositional models (for a systematic review on unsupervised learning in morphology, see @xcite . thus, the current article specifically concerns prompting agnostic-task models to exploit morphological knowledge, i.e. individual morphemes and their functional roles, for morphological connection. the chosen model was chatgpt (openai, 2021), a gpt-3.5 fine-tuned model. we compared statistically how the model performed given different types of prompts, and varying amounts of contextual information. here, 'context' was operationalised in terms of keywords (i.e. 'new' or 'morpheme') or information about the task the model needed to perform. given the model's training to avoid hallucinations, we expected it to classify new words as non-existent in the absence of context. we also used english as our test case, primarily due to being one of the languages for which chatgpt performs the best in various tasks @xcite . we show that depending on the criteria considered (i.e. plausibility or humanlikeness) different prompting strategies were better. additionally, in line with our expectations, regardless of the criteria used, adding more contextual information increases the model's chance of offering a correct word definition. our results prove useful specifically for the users of chatgpt, as despite the spread in its use, limited studies have been conducted on chatgpt's neologism comprehension @xcite . even more, our results also prove important for research on novelty and llms, especially given this is overall a neglected research topic according to @xcite . moreover, this study presents a systematic methodology to test an llm's ability to identify morphological connections or decompositions, which can be applied in the future to investigate other languages and language models as well. the article is organized as follows: section 2 outlines out methods, including the materials, procedure, scoring, and analysis, section 3 describes our results, and sections 4 and 5 include our discussion and conclusions. ## results the descriptive statistics for the number of plausible and humanlike definitions provided by the model for any given prompt are shown in ## discussion while our analysis did not identify one prompt that was significantly better than all others in humanlikenss or plausibility, we found that the model always performed better when the prompt provided additional contextual information in the form of the words new and morpheme. as expected, the more specific the task is for the language model, the better it performs. thus, our results also reinforce ekin's (2023) suggestions on giving specific details about the task. better performance on plausibility for words with a productive affix suggests that chatgpt may be better at identifying form-meaning mappings for productive suffixes than for unproductive ones. this finding is in line with what authors like @xcite found for language acquisition in children, and might be, similarly, caused by frequency of the affix. for example, the more productive the morpheme, the more often it may occur in the training data, and the more information the model could have regarding the contexts it appears in. eventually this might result in a more accurate representation of the meaning of the morpheme. in contrast, productivity had no significant effect on humanlike outputs of the models, which might indicate that high probabilities counts of productive morphemes do not encompass characteristics of human definitions. this might be because human definitions rely often on aspects that cannot be inferred from probabilities counts, such as personal experience or world knowledge when defining new words, e.g. defining musksome as having a certain kind of smell. however, though productivity had no effect on humanlike definitions, there was a significant interaction between productivity and the word new. more exactly, adding the word new to the prompt always improves it. however, the impact of the word 'new' on the humanlikeness of the generated definition is less strong if the morpheme is productive. thus, for words with unproductive morphemes, the model may improve more because it may be more likely to "look for" a morphological decomposition as a heuristic when given the additional context. this interaction effect between productivity and the word 'new' can also be influenced by the root morphemes of the words. this might be because the most salient word meaning differs significantly for humans and chatgpt: given the model was trained on data for a numerous specialised tasks, it sometimes regards specialised definitions very likely. this was clearly shown for the word assemblyless. we found that while almost all human participants defined the word in relation to an assembly as a 'group or gathering of people' or 'the process of assembling something', chatgpt most frequently drew the connection to a programming language called 'assembly'. this resulted in less humanlike definitions for the model, showcasing humanlike definitions were always plausible, but not the other way around, as shown in the example below: even more, productive morphemes are defined in @xcite in terms of appearance in hapax legomena, which might indicate they form more specialised words. thus, the probability of generating specialised definitions can increase because our definition of productive suffixes is defined, to some extent, in relation to specialised words. this would overall increase chance of generating specialised word definitions if the root morpheme can also have a specialised meaning, like assembly. even for words without a root morpheme with specialised meanings, productive derivational morphemes might often be associated with various root morphemes with diverse meanings, which can lead to bigger perplexity and, consequently, less specific or humanlike output. comparatively, unproductive morphemes have less chances to be associated with more words, which would make the model stick to a few definitions that might also be more specific. thus, depending if the user intends to obtain more humanlike or just plausible definitions, the prompt that scored highest in one of the criteria can be used. the statistical results can also be used to further create new prompts by addition of the two contextual words, i.e. 'new' and 'morpheme', depending on the tasks morphological connection will be used in. note that the both conditions of the best humanlike prompt either have no context, or both context words. this indicates some successful prompts follow an overall opposite effect, i.e. they lack context words but perform well. lastly, if the reader wants to achieve a balance in the chosen prompt, i.e. to be as humanlike and as plausible as possible, the 'persona' word generator prompt, condition + new, + morpheme may be the best choice, given its high scores in both criteria, averaged at 9.5 correct definitions for morphological connection. ## conclusion our study aimed to identify an optimal prompting strategy for chatgpt to draw morphological connections while producing word definitions for unfamiliar words. we conducted a word definition task using different prompts, and compared the model's performance on plausibility and humanlikeness for definitions that it provided for a set of morphologically complex nonce words. we found that irrespective of the prompt template, adding the words new and morpheme to a prompt, significantly improved the performance of chatgpt. thus, the users looking to obtain definitions of unfamiliar morphologically complex words from the model can apply the current findings by including such keywords in their prompts. in terms of best prompting strategies, our results found two prompts that had the maximal performance on plausibility, i.e. 'persona' prompt word generator, condition + new, + morpheme and condition -new, + morpheme, and 'context manager' prompt, condition + new, + morpheme. in terms of humanlikeness, our results show one prompt that had the best performance, i.e. 'persona' prompt word generator, condition -new, -morpheme, and condition + new, + morpheme. these results also indicate the 'persona' prompt word generator, condition + new, + morpheme scores, on average, the highest. however, note that the statistical analysis did not point to an overall best prompt. we found evidence that chatgpt's response is however also always modulated by factors such as the nature of its training data and world knowledge, which can lead it to produce definitions which while plausible, may not be humanlike. this is something that may be of interest to researchers in the future.