Automatically Generated Definitions and their utility for Modeling Word Meaning (2024)

Abstract: Modeling lexical semantics is a challenging task, often suffering from interpretability pitfalls. In this paper, we delve into the generation of dictionary-like sense definitions and explore their utility for modeling word meaning. We fine-tuned two Llama models and include an existing T5-based model in our evaluation. Firstly, we evaluate the quality of the generated definitions on existing English benchmarks, setting new state-of-the-art results for the Definition Generation task. Next, we explore the use of definitions generated by our models as intermediate representations subsequently encoded as sentence embeddings. We evaluate this approach on lexical semantics tasks such as the Word-in-Context, Word Sense Induction, and Lexical Semantic Change, setting new state-of-the-art results in all three tasks when compared to unsupervised baselines.

## introduction modeling lexical semantics using unstructured text has a longstanding history in natural language processing due to its crucial role in both natural language understanding and natural language generation @xcite . over the past decades, there have been many relevant technological developments: from count-based @xcite to static @xcite and contextualized @xcite language models, and most recently, generative models @xcite . each of these advancements has contributed significantly to the goal of modeling the meaning of words. modern language models are based on the transformer @xcite architecture. given a word, these models generate semantic representations for each occurrence of the word based on its surrounding context @xcite . ideally, these representations should be similar for semantically related word usages and different for semantically distinct ones. typically, contextualized vectors (i.e., embeddings, pilehvar and camacho-collados, 2021) or lexical substitutes (i.e., bag-ofwords, @xcite ) are employed to represent word usages. however, recent advancements in text generation are shifting the attention towards representing word usages through generated sense definitions @xcite . automatically generated sense definitions provide a dual advantage. firstly, they distill the information stored in a sentence by abstracting away from the context. their use potentially condenses various word usage representations pertaining to the same underlying meaning. secondly, generated definitions provide a means to directly interpret word meaning from unstructured text, thereby enabling language models to serve as surrogate for dictionaries when encountering unfamiliar words @xcite , or known words in unfamiliar settings @xcite . in this work, we automatically generate definitions for words in-context by relying on two fine-tuned variants of the llama chat models @xcite refined through instruction tuning @xcite on lexicographic resources. we call the models llamadictionary and assess their performance in definition generation, achieving new state-of-the-art results on multiple datasets. we further extend our evaluation by using llamadictionary and the flan-t5-definition model fine-tuned by @xcite for large scale modeling of word meaning. specifically, we employ the generated sense definitions as intermediate sense representations. these representations are encoded using a pretrained sequence embedding model rather than using standard token embeddings. we evaluate our approach on three popular natural language processing tasks, namely word-in-context, word sense induction, and lexical semantic change, achieving new stateof-the-art results on all three tasks. ## automatic definition generation in this work, we fine-tuned two popular opensource generative models through instruction tuning, namely llama2chat foot_0 and llama3instruct foot_1 . we specifically chose to fine-tune chat models because they were already optimized to generate responses adhering to specific instruction prompts. we call the models resulting from finetuning llamadictionary. in the following, we refer to llama2dictionary and llama3dictionary for the fine-tuned versions of llama2chat and llama3instruct, respectively. using llama2dictionary and llama3-dictionary, we complement the existing flan-t5-definition 3b model by @xcite with two larger llama 7b and 8b, chat-based versions. ## evaluation setup our evaluation is structured into two parts. first, we assess the quality of definitions generated by llamadictionary and flan-t5-definition through the definition generation (dg) task. for this evaluation, we directly utilize the generated sense definitions. next, we explore their utility in three popular natural language processing tasks, namely 3 we have also experimented with quantization combined with lora (qlora, @xcite obtaining very similar evaluation results (see figure 4 ). these are omitted due to space restriction but are available in our github repository together with all our code, data, and results. our final llama2dictionary and llama3dictionary models are available on hugging face at francescoperiti/llama2dictionary and francescoperiti/llama3dictionary. word-in-context (wic), lexical semantic change (lsc), and word sense induction (wsi). specifically, instead of using standard token embeddings, we view sense definitions as intermediate sense representations and encode these as embeddings through a pretrained sequence embedding model. formally, this means that: given an occurrence of a word w, we employ a generative model g (i.e., llamadictionary or flan-t5-definition) to generate a definition d, which we subsequently encode as a vector v using a sentence embedding model m, i.e., @xmath0. following @xcite , we used the all-distilroberta-v1 sentence sbert model @xcite to encode definitions as contextualized sentence embeddings foot_2 . to validate our results, we also evaluate twelve other sbert models which show comparable results. we present these results for the base wic tasks and proceed with a single model (i.e., all-distilroberta-v1) for further evaluation on wsi and lsc. full results for wsi and lsc are available online and are summarized in figure 5 in the appendix. furthermore, we extend our evaluation by also considering generated definitions by the flan-t5-definition model recently finetuned by @xcite foot_3 as this model has not been evaluated on the wic, wsi, and lsc tasks previously. ## evaluation results in our evaluation, we used llama2dictionary and llama3dictionary with the parameters reported in ## conclusion inspired by recent advancements in text generation, in this work, we investigated the potential of fine-tuned large language models to generate sense definitions for words in-context. specifically, we fine-tuned two new llama chat based models, called llamadictionary, and assessed their performance along with an existing flan-t5-definition model on the definition generation task. next, we explored their utility for modeling word meaning by addressing lexical semantic tasks such as word-in-context, word sense induction, and lexical semantic change. in our experiments, we considered the generated definitions as intermediate representations, passed through a sentence embedding model. our results consistently show that we can use generated definitions to explicitly model the meaning of word usages through interpretable definitions. in all tasks, the use of sentence embeddings for generated definitions outperformed the use of standard token embeddings for word occurrences, setting new state-of-the-art results. across tasks, we find that the use of the larger 7b and 8b llamadictionary models compared to the smaller 3b t5-based model leads to slightly higher results in the definition generation task, while being equally strong on the lexical semantics tasks. an extension of the llamadictionary models is to fine-tune them on all the benchmarks that have been used for the flan-t5-definition model, as well to fine-tune the models further on generated usage sentences @xcite . our evaluation using automatically generated sense definitions in this paper paves the way for future advancements in modeling lexical semantics. for example, by offering an automatic labeling of senses, we can support the creation of lexicographic resources for all languages, including lowresource languages @xcite , providing a way to better know what change our words have experienced over time.