Explaining novel senses using definition generation with open language models (2025)

Abstract: We apply definition generators based on open-weights large language models to the task of creating explanations of novel senses, taking target word usages as an input. To this end, we employ the datasets from the AXOLOTL’24 shared task on explainable semantic change modeling, which features Finnish, Russian and German languages. We fine-tune and provide publicly the open-source models performing higher than the best submissions of the aforementioned shared task, which employed closed proprietary LLMs. In addition, we find that encoder-decoder definition generators perform on par with their decoder-only counterparts.

## introduction and related work recent nlp advancements have sparked interest in the computational modeling of semantic change. thus far, the research community has primarily focused on identifying words that have changed their meaning over time. existing approaches are primarily based on vector token representations (embeddings) and thus often do not enable the interpretation of the novel senses a word has gained @xcite . only recently, with the advent of new generative language models, the research community has begun to turn its attention to the interpretation of detected semantic change. one step in this direction was the axolotl'24 shared task on explainable semantic change modeling @xcite . the shared task was focused on the analysis of diachronic semantic shifts between two time periods, a challenge typical for historical linguists and lexicographers. it consisted of two separate subtasks, given a set of target word x usages (examples): in this paper, we apply llm-based definition generators @xcite @xcite to the second subtask of axolotl'24, where the participants were asked to create descriptions or definitions for novel word senses. 'novel' here means a sense which is present in a corpus from the newer ('second') time period, but is not mentioned in a dictionary covering the older ('first') time period. in the simplest form, the task is to provide a correct definition of a novel sense y of a target word x, given a bunch of x usages belonging to y. the performance of the systems was evaluated with bleu @xcite and bertscore @xcite , comparing the generated definitions to manually annotated gold definitions. an example instance from the axolotl'24 shared task is given in the axolotl'24 shared task was offered in three languages: finnish, russian and german, with the latter as a 'surprise language' without training and validation datasets. three teams participated in subtask 2, achieving promising performance, but still leaving ample room for improvements. one of the teams used glossbert @xcite fine-tuned with adapters to match usage examples to senses and definitions retrieved from wiktionary. two other teams prompted the gpt 3.5 language model to generate senses and definitions. thus, the best-performing systems relied on a closed proprietary llm and a lexical database, respectively. however, the use of a closed model is not ideal as it lacks transparency, and limits accessibility for researchers. similarly, the use of existing lexical resources such as wordnet or wiktionary at test time contradicts the goal of identifying novel senses, as genuinely new meanings are, by definition, absent from established ontologies. for this reason, in this paper we concentrate on the generative approach only and evaluate defini-tion generators built on three different instructiontuned open-weights llms: mt0, aya-101 and towerinstruct. our contributions are as follows: our code is publicly released on github foot_0 and model adapters are available on huggingface foot_1 . ## data in order to adapt an existing generative llm for the task of generating definitions, one needs to finetune it on a corresponding dataset. in this work, we utilize two resources: the axolotl'24 training data, and definitions and examples from dbnary @xcite , a lexicographic resource derived from wiktionary. for finnish and russian, we fine-tune two versions of each model: some of the axolotl'24 test set senses and definitions ultimately come from wiktionary. thus, to avoid data contamination, we removed from our dbnary datasets all the words (along with their senses, definitions and usages) also present in the axolotl'24 test and development sets for all languages. ## definition generators this section motivates the choice of models to finetune and describes the training setup. ## results the fine-tuned definition generators were used to create definitions for the usages with novel senses from the axolotl'24 subtask 2 test sets, which were then aggregated as described above. in this setup, we assumed that subtask 1 (finding these usages) is already solved. quoting the shared task organizers, "the evaluation of subtask 2 therefore limits itself to evaluating the validity of provided glosses" @xcite . the resulting 'target-sense-definition' triplets were evaluated by the official axolotl'24 scoring code. 8 the winner definition generators fine-tuned on open-weights llms outperform the best ax-olotl'24 subtask 2 generative submissions for all three languages under analysis. while towerdictionary and aya-101-defgen perform better than mt0defgen, it is not possible to define a winner based only on bleu and bertscore, since differences between towerdictionary and aya-101-defgen are not statistically significant ## conclusion in this work, we use large language models finetuned on definition modeling to generate labels for novel senses. one can think of this task as updating a dictionary based on a set of new texts. the participants of the axolotl'24 shared task mostly tackled the problem by using the proprietary gpt 3.5 model. we show that one can instead fine-tune open-weights llms and achieve a better performance in this subtask with contextualized definitions generated by them. we also publicly share the models. our comparison of different base models showed that instruction-tuned encoder-decoder (t5-like) models perform on par with their decoder-only (llama-like) counterparts. according to automatic metrics, larger models always outperform smaller ones, as well as larger fine-tuning datasets often do. however, not only data quantity, but also its relevance to the domain matters. our experiments also show that few thousands of fine-tuning data instances may not be enough for a complex semantic task such as definition generation.