Exploring Automated Keyword Mnemonics Generation with Large Language Models via Overgenerate-and-Rank (2024)

Abstract: In this paper, we study an under-explored area of language and vocabulary learning: keyword mnemonics, a technique for memorizing vocabulary through memorable associations with a target word via a verbal cue. Typically, creating verbal cues requires extensive human effort and is quite time-consuming, necessitating an automated method that is more scalable. We propose a novel overgenerate-and-rank method via prompting large language models (LLMs) to generate verbal cues and then ranking them according to psycholinguistic measures and takeaways from a pilot user study. To assess cue quality, we conduct both an automated evaluation of imageability and coherence, as well as a human evaluation involving English teachers and learners. Results show that LLM-generated mnemonics are comparable to human-generated ones in terms of imageability, coherence, and perceived usefulness, but there remains plenty of room for improvement due to the diversity in background and preference among language learners.

## introduction recent advances in natural language processing have expanded the exploration of developing automated methods for applications in language learning, including second language acquisition @xcite @xcite , linguistic skill modeling @xcite , language assessment and correction @xcite , and language practice through conversational chatbots @xcite . in this work, we investigate an intriguing but relatively under-explored vocabulary learning application: keyword mnemonics (km) @xcite . km is a widely utilized technique employed by language learners to memorize vocabulary effectively, through the creation of memorable associations with keywords that they already know, aided target word (keywords) alleviate (a, leaf, he, ate) verbal cue on his plate, there was a leaf he ate to alleviate his hunger. visual cue figure 1 : an example of human-authored verbal cue @xcite for the target word "alleviate" and visual cue generated by stable diffusion xl. we study automated verbal cue generation in this work and leave visual cue generation for future work. by verbal and visual cues. this technique is used in many language learning resources, from traditional books @xcite @xcite to modern community-based learning platforms (mnemonic dictionary, 2024; koohii @xcite , where users contribute mnemonics and the community vote on their effectiveness. km involves a two-step process that creates an acoustic and imagery link for the learner. for instance, consider a learner learning the target word "alleviate", as shown in figure 1 . first, an acoustic link is established to an already-known set of keywords with similar pronunciation, e.g., "a, leaf, he, ate." second, a imagery link is established to these keywords with a verbal cue that evokes a mental image, e.g., "on his plate, there was a leaf to alleviate his hunger." these links can be further reinforced using a visual cue. despite its effectiveness, km generation is challenging since it demands significant human effort to meticulously find keywords that resemble the target word and create memorable verbal cues. for instance, the word "alleviate" has a large number of possible keyword combinations, such as "a, leaf, he, ate" or "a, levy, it"; sorting through them takes considerable effort from teachers and learners. moreover, the task of creating verbal cues using the keywords that evoke vivid mental images adds an extra layer of complexity to the process. existing work on automated km generation remains limited in both technique and evaluation. for example, earlier methods are restricted to generating a single keyword in the context of second language acquisition. the work in @xcite uses a cognitive psychology-inspired method to automatically generate a keyword in the first language that has high phonetic, orthographic, and semantic similarity with the target word in the second language. more recently, the emergence of large language models (llms) provides a new and potentially more scalable solution to automated km generation @xcite , expanding automated km generation to generating verbal cues using the keywords generated by @xcite . the work in @xcite fine-tunes llama-2 with verbal cues from an online platform and aligns it using student feedback, highlighting how incorporating student preferences can improve llmgenerated verbal cues. however, evaluating kms remains challenging due to two main reasons: the lack of automated metrics to evaluate the quality of verbal cues, and the subjective nature of mnemonics to language learners. we need automated ways to evaluate important aspects of kms such as imageability and coherence. the work in @xcite explored using text-to-image models for the automatic assessment of sentence imageability, although it has not been investigated in the context of language learning. ## problem statement we now formally define the verbal cue generation task. to do so, we first need to generate a set of syllabic keywords given a target word, then a verbal cue that contains them. given a target word t, we denote its syllables as s @xmath0, ..., s l ) where l is the total number of syllables in t. our goal is to generate a set of m syllabic keywords as @xmath1, . . . , k m }, where k m is phonetically similar to one (or more) consecutive syllables in t, beginning at index l m and ending at l ′ m , which we denote as s lm:l ′ m . the set of syllabic keywords also has to cover all syllables, i.e., ∪ m {l m , . . . , l ′ m } = {1, . . . , l}. this task is challenging, since that there are 2 l-1 total possible ways to split the set of syllables of t, and that we need to choose the right keywords that preserve the phonetic properties of the target word. then, our task is to generate a verbal cue @xmath2, ..., w n ), where w i denotes the i-th word in the cue. the constraint we put on the verbal cue is that it must contain both the specific target word and all syllabic keywords, which we formally define as ## methodology we propose a two-step overgenerate-and-rank method for km generation via llm prompting to navigate the large space of possible keywords and verbal cues. first, we overgenerate multiple sets of candidate syllabic keywords by prompting an llm, rank them according to a series of measures, and select the top set. second, we use these keywords to overgenerate multiple verbal cues, rank them according to insights obtained from a pilot study with english teachers. see supplementary material a for the exact prompts we used. ## automated evaluation in the following sections, we aim to explore three specific aspects of kms that can vary between individuals: imageability, coherence, and usefulness. in this section, we introduce proxy metrics for automatically evaluating imageability and coherence. later, in section 5, we conduct a human evaluation to additionally measure the perceived usefulness of verbal cues by both english teachers and learners, in addition to imageability and coherence. ## case study we now qualitatively analyze cues generated by our method based on feedback collected from learners through a post-experiment survey. we asked learners to identify and explain their top five most useful and least useful cues. from their responses, we discuss four representative examples of the most and least useful verbal cues, as shown in instead, the human-authored cue "an intimate date tends to intimidate her." is rated higher by learners across three aspects (imageability: 3.7 vs. 3.0, coherence: 3.8 vs. 3.0, usefulness: 3.9 vs. 2.4), evoking strong emotions to make it memorable. it is worth noting that the text-to-image model somewhat failed to describe the scenario in the cue for "retract," leading to a lower imr score. this example suggests that certain properties of verbal cues, such as creativity or emotion, may overcome other problems such as a lack of imageability. ## related work imageability is defined as "the ease with which a word arouses sensory images" @xcite . to quantify this intangible aspect of language, psycholinguists and psychologists have compiled human imageability ratings databases like the mrc psycholinguistic database @xcite and glasgow norms @xcite . however, collecting this data through interviews is costly and time-consuming, making it difficult to scale psycholinguistic databases to the size of modern nlp corpora like the corpus of contemporary american english, which includes over 60,000 lemmas with frequency and parts of speech data @xcite . recent studies have addressed the challenge of scalability in assessing imageability by applying machine learning techniques to automate the collection of imageability ratings. these efforts include predicting imageability using supervised learning @xcite , and employing im-age data mining to estimate word imageability by analyzing a various visual features @xcite . moreover, advancements have extended these predictive models to the sentence level. for instance, researchers have developed methods to evaluate the visual descriptiveness of captions and introduced ways to calculate a sentence's imageability score based on the imageability scores of its constituent words @xcite . additionally, computational techniques have been proposed that utilize text-to-image models to generate images and measure sentence imageability by calculating the cosine similarity between the image and word embeddings @xcite . ## conclusions and future work in this paper, we explored using large language models for the task of automated keyword mnemonics generation for vocabulary learning, via a novel overgenerate-and-rank method. through both automated and human evaluation with both english teachers and learners, we found that our generated verbal cues are comparable to or better than humanauthored ones, in terms of imageability, coherence, and usefulness. we also studied the intrinsic subjectivity among learners in our evaluation through qualitative feedback, which led to relatively lower inter-rater agreement. there are many avenues for future work. first, due to this intrinsic subjectivity, we need to develop methods to generate personalized cues that each individual learner will find useful, by adapting to their personal language knowledge and cultural background. second, we need to study automated visual cue generation, in the form of images or even videos, and evaluate their quality separately from the verbal cues. third, we plan to conduct an experiment with real language learners in classrooms, to evaluate whether our automatically generated verbal and even visual cues can indeed enhance vocabulary recall over a long period of time. fourth, for each language, there may be better ways of generating richer mnemonics by leveraging linguistic and cultural nuances. for instance, in mandarin, the character "休" ("to rest") can be linked to the keyword "shoe" based on sound "xiū", but a more meaningful mnemonic could involve the character's components-"人(person)" and "木(tree)"-that combine to form its meaning. developing such language-specific verbal cues is a promising direction for future work.