Generative Dictionary: Improving Language Learner Understanding with Contextual Definitions (2024)

Abstract: We introduce GenerativeDictionary, a novel dictionary system that generates word sense interpretations based on the given context. Our approach involves transforming context sentences to highlight the meaning of target words within their specific context. The method involves automatically transforming context sentences into sequences of low-dimensional vector token representations, automatically processing the input embeddings through multiple layers of transformers, and automatically generate the word senses based on the latent representations derived from the context. At runtime, context sentences with target words are processed through a transformer model that outputs the relevant word senses.Blind evaluations on a combined set of dictionary example sentences and generated sentences based on given word senses demonstrate that our method is comparable to traditional word sense disambiguation (WSD) methods. By framing WSD as a generative problem, GenerativeDictionary delivers more precise and contextually appropriate word senses, enhancing the effectiveness of language learning tools.

## introduction the need for effective language mastery grows more critical as the world becomes increasingly interconnected. reference resources like dictionaries are pivotal in language learning and vocabulary acquisition. traditional dictionaries such as word-net, cambridge learner dictionary, and macmillan dictionary, curated by professional lexicographers, provide a solid foundation for understanding language. they organize word senses into related synsets or lists of meanings that help learners grasp the nuances of vocabulary, enhancing their communication skills. traditional lexical resources typically list the most common senses of a word, sometimes overlooking figurative or less frequent usages * corresponding author (e.g., figure 1 shows the list of word senses for "baton" as presented by wordnet). users might struggle to identify the intended word sense in specific contexts, especially when the context extends beyond the provided sense inventory. this limitation underscores the need for more context-sensitive tools. consider the word "baton" in the sentence: "the successful passing of the [ baton ] demonstrated the team's ability to work collaboratively and manage responsibilities efficiently." in this context, "baton" does not refer to "a hollow cylinder passed from runner to runner in a relay race" but rather symbolizes "the responsibility for a person or a position." traditional dictionaries like wordnet may not capture this symbolic meaning, making it difficult for learners to grasp the metaphorical connection. we present a new system, generativedictionary foot_0 , that addresses this gap by interpreting words within their specific contexts. for example, in the sentence "the successful passing of the [ baton ] demonstrated the team's ability to work collaboratively and manage responsibilities efficiently," generativedictionary identifies "baton" as "short staff symbolizing authority" by leveraging advanced generative ai technology, this system fine-tunes pre-trained transformer models to analyze context and generate accurate word sense descriptions. the generativedictionary system enhances the usefulness of traditional dictionaries by providing concise, context-sensitive meanings. it bridges the gap between conventional word sense inventories and the nuanced interpretations required for effective communication. this innovative tool offers a more dynamic and practical approach to understanding language, making it an invaluable resource for learners and professionals alike. 2 related works dictionaries play a crucial role in language learning, particularly in the area of vocabulary expansion. studies had emphasize that a well-developed vocabulary is fundamental to language proficiency and effective communication, and dictionaries provide the necessary support for learners to access this vocabulary. @xcite ) furthermore, @xcite suggests that engaging with dictionaries encourages autonomous learning and helps learners to become more effective at decoding unfamiliar words independently. additionally, research by @xcite illustrates that dictionaries aid in vocabulary learning by enabling learners to confirm their understanding of words and explore various meanings and contexts. these studies collectively highlight the pivotal role of dictionaries in enriching a learner's vocabulary and enhancing their overall language competence. in our research, we represent word senses as concise, simple english descriptions (e.g., "stealmove quietly and secretly"), rather than an entry id in the sense inventory (e.g., "steal.v.2" in wordnet). more specifically, we focus on generating simple glosses for a target word in a given sentence for the purpose of assisted reading in language learning. the body of the wsd research most closely related to our work focuses on automatically classifying the target word in a given sentence into one of sense in pre-determined inventory, using information in the given sentence. the advent of word embeddings has revolutionized wsd by providing low-dimensional dense vector representations that capture semantic relationships between words. notably, word2vec @xcite utilize skip-gram and continuous bag-of-words (cbow) to enable models to capture syntactic and semantic properties of words from large corpora. the lack of annotated data for wsd became increasingly evident as the capability of embedding and model architecture became increasingly sophisticated, prompting researchers to explore innovative solutions to enhance performance. one significant approach involved incorporating additional contextual data, exemplified by models like glossbert @xcite , ewiser @xcite , and ares @xcite . glossbert leveraged gloss definitions to enrich the context used by bert, thus providing more comprehensive information about word senses. ewiser integrated synset embeddings from wordnet into its model, allowing it to utilize the rich semantic information encoded in these synsets. ares utilized large-scale multilingual data to train sense embeddings, enhancing the model's ability to disambiguate words in various languages and contexts. another approach aimed at reducing the complexity of sense inventories by compressing them, such as combining wordnet hypernyms. the esr (extended synset representation) @xcite model effectively merged similar senses into broader categories, thereby simplifying the inventory space. additionally, researchers sought to generate annotated data from unannotated corpora. techniques like distant supervision and semi-supervised learning enabled the automatic labeling of large text corpora, providing a substantial increase in training data without the need for extensive manual annotation. recent advances in pre-trained large language models (llms) have opened new avenues for reformatting the word sense disambiguation problem into a generative problem, where the context is provided as input and the sense is generated as output. the emergence of pre-trained transformerbased architectures @xcite , such as t5 @xcite , has revolutionized natural language processing by enabling models to understand and generate human-like text through extensive pre-training on large corpora. specifically, t5 (text-to-text transfer transformer) has demonstrated the potential of framing various nlp tasks, such as translation, and summarization, as a textto-text problem, thereby simplifying the modeling process. by leveraging these pre-trained models, we can alleviate the issue of limited annotated data, as these models possess a rich understanding of language nuances. this paradigm shift not only enhances the accuracy of wsd but also offers a scalable and adaptable solution, paving the way for more robust applications across different languages and domains. in contrast to previous researches on word sense disambiguation, we present a system that automatically learns to generate a short gloss for a target word in a given sentence, by curating a collection of data to fine-tune a pretrained text-to-text model. we exploit the inherent regularity in dictionary definitions and examples to build a model for effective word sense interpretation. ## methodology our method can be summarized in a series of streamlined steps. first, we collect a comprehensive sense inventory from various lexicographic resources. next, we simplify definitions that are excessively long or cumbersome, making them more accessible and easier to understand. to further enhance our dataset, we use generative ai to create additional example sentences based on the definitions. finally, we train our model using this enriched dataset, which now includes both the simplified definitions and the newly generated example sentences. this systematic approach ensures a com- ## system generativedictionary features a simple and userfriendly interface. users can write or copy-paste the desired text into the textbox on the left. by pressing the "enter" key, the text is submitted to the system. users can then hover over any word in the text and click on it to see the word's part of speech and definition, which will be displayed in the text box on the right. behind the scenes, our system identifies the target word and places brackets around it to identify it. the augmented sentence is then sent to the model, which generates the definition based on the context. this process allows generativedictionary to provide precise and context-sensitive definitions, enhancing the user's understanding of the text. ## evaluations to ensure our system produces results that are applicable in real life, we asked five english teachers to rate the generated definitions of a sample of dsd and ex-dsd. additionally, to benchmark our model's performance against the latest advancements in natural language processing, we compared our results with those generated by gpt-4-turbo. this comparison allowed us to gauge the relative effectiveness of our approach. the test set consists of a total of 500 sentences, each containing a marked target word. of these, 300 sentences are drawn from dictionary example sentences (dsd), while the remaining 200 sentences are generated using gpt-4 (ex-dsd), based on the definitions of the target words. we asked evaluators to assess the quality of the generated definitions for each sentence, using a grading scale from 0 to 2. this scale measures the degree of correctness, taking into account the precision of the definition and its suitability for the given context. a score of 0 is given when the generated definition is entirely incorrect or fails to capture the intended meaning of the target word. a score of 1 is assigned when the definition is partially correct. in these cases, the definition may still convey the general sense of the target word, but could include issues such as incorrect part of speech, definitions that are overly broad or narrow. a score of 2 reflects an accurate and appropriate definition that aligns well with both the meaning of the target word and its usage in the sentence. definitions awarded this score are not only correct but also contextually appropriate, effectively conveying the intended meaning without significant omissions or errors. the results are shown in moreover, the results indicated a slight decline in the number of definitions receiving a perfect score of 2 when sentences were generated rather than taken directly from dictionary examples. this suggests that the additional variables involved in generating sentences-such as variability in sentence structure, word usage, and contextual nuances-may introduce complexities that make it more challenging for the model to produce fully accurate and contextually appropriate definitions. in our comparisons with the latest large language models (llms), specifically gpt-4-turbo, we found that our models are not yet on par with the performance of chat-gpt. however, this disparity is understandable given the differences in model architecture and data size. gpt-4-turbo benefits from extensive training on vast datasets, which is not feasible for smaller models like ours. despite this, our models performed satisfactorily within their operational constraints. additionally, we observed that chat-gpt rarely utilized the same wording as the reference definitions. as a result, incorporating sequence similarity metrics could provide a more accurate assessment of semantic similarity between the generated definitions and the reference texts. this shift would allow us to better capture the essence of the definitions, even when the exact phrasing differs. ## conclusion and futureworks in this study, we introduced generative dictionary, a novel context-based dictionary system that enhances the user experience by providing contextsensitive definitions. by reframing word sense disambiguation (wsd) from a traditional classification problem to a text-to-text generation task, we harnessed the power of pre-trained transformer models. these models, embedded with extensive lexical knowledge, generate definitions for ambiguous words, simplifying the wsd task and leveraging advancements in natural language processing for improved accuracy and applicability. our evaluation results demonstrate that our models perform well, especially when generating concise definitions. this improvement underscores the effectiveness of short text generation in addressing the challenges associated with longer outputs for t5-based models. however, a performance gap remains between our models and state-of-the-art large language models like gpt-4, primarily due to differences in model size and training data volume. future work should aim to bridge this gap by refining our evaluation metrics and exploring more sophisticated methods for sequence similarity, which could provide a more accurate measure of the semantic quality of the generated definitions. enhancing our models' ability to produce precise and contextually appropriate definitions will be crucial for advancing wsd. overall, generative dictionary marks a significant step forward in wsd research, offering a usercentric approach that improves the dictionary experience by delivering relevant and context-aware definitions. by focusing on improved evaluation techniques and advanced similarity measures, future research can build on this foundation to achieve even greater performance and applicability in realworld scenarios.