Multilingual Definition Modeling (2025)

Abstract: In this paper, we propose the first multilingual study on definition modeling. We use monolingual dictionary data for four new languages (Spanish, French, Portuguese, and German) and perform an in-depth empirical study to test the performance of pre-trained multilingual language models on definition modeling of monosemic words when finetuned on this data. Furthermore, we use a zero-shot approach to test the multilingual capabilities of two popular chat-based Large Language Models (LLMs) in the task. Results show that multilingual language models can perform on-pair with English but cannot leverage potential cross-lingual synergies, with LLMs generally offering better performance overall. A comprehensive human evaluation of the LLM-generated definition highlights the zero and few-shot capabilities of these models in this new task, also showing their shortcomings. Finally, we show that performance on our task via BERTScore strongly correlates to the performance on multilingual LLM benchmarks, suggesting that our task offers a viable compute-constrained, stable and natural alternative to these.

## introduction definitions play a key role in effective communication and language understanding. however, creating high-quality and precise definitions requires time and effort given the complexity of languages and how they are constantly evolving. aiming to suffice this necessity, the definition modeling task was first proposed by @xcite . the goal is to estimate the probability of a textual definition given a word being defined. this task can be applied to generate definitions for known and new words, assisting language learners, as well as serving as a tool for language preservation and description @xcite . since its conception, several approaches have tackled this task, showing that it gives an arguably more transparent view of how much a model captures syntax and semantics. so far, existing approaches for this task are trained on a corpus of word-definition pairs, to be later tested on how well they generate definitions for words not seen during training. current approaches @xcite @xcite are mainly encoder-decoder based, in which a contextual representation for a word/phrase, encoded using a variety of features, is used to generate the definition. despite the progress, previous work has predominantly focused on definition modeling in english. concretely, we find just a handful of papers that work on definition modeling for languages other than english, namely @xcite for @xcite for portuguese and @xcite for chinese. we also find that dictionary data for french and italian have been released in the past few years, but that there are no approaches that leverage them so far @xcite @xcite . finally, while @xcite has covered nine different languages in their study, they did not explore multilingual capabilities of models, focusing on performance of monolingual models on the polysemy aspect of words. moreover, we note that these datasets are mostly based on crowdsourced dictionaries, where quality can be a concern. in light of these issues, in this paper, we present the first truly multilingual study on definition modeling, introducing datasets obtained from notcrowdsourced dictionaries in french, spanish, portuguese, and german, with a total of approximately 460k new terms entries and 730k definitions, thus substantially extending language coverage for the task. we combine our collected data with existing corpora to train models and lay out the first set of experiments on multilingual definition modeling. as our main focus is to analyze the potential synergy between different languages, we focus our experiments on monosemic words, hoping to explore polysemy in future work. furthermore, we also test a selection of large language models (llms) on our collected data, probing their multilingual abilities in this task through zero-shot and few-shot approaches @xcite . we evaluate performance based on three different automatic metrics, i.e., bleu, bertscore and comet, and through human evaluation. our results show that current multilingual language models such as mbart @xcite and mt5 @xcite can perform on-pair with english when fine-tuned on single-language dictionaries. they can also deliver consistent performance when trained in multiple languages, but are unable to leverage cross-lingual synergies to consistently improve performance. as for llms, we show that prompting techniques, either in the zero-shot or few-shot scenarios, offer generally better performance than multilingual language model fine-tuning, even in the case of llms. however, we also observe that the output language can be difficult to control and that techniques such as in-context learning do not lead to significantly better quality output. finally, we show empirical evidence suggesting that per-language performance in our task measured via bertscore is strongly correlated with performance on multilingual benchmarks. opposite to these, which are often translations of datasets in english, our task can offer a computeconstrained, static and natural alternative to assess the proficiency of llms on a given language. we release our code and data on github foot_0 . ## related work our paper is primarily related to the seminal work by @xcite ; @xcite , in which a model is tasked with generating a definition for a word given its respective embedding or with mapping dictionary definitions to lexical representations of words, respectively. later work has proposed improvements and extensions, introducing techniques and datasets to address shortcomings. for example, @xcite address polysemy and presents a dataset from oxford dic-tionaries, where each definition is supplemented with context sentences. @xcite proposed an approach for automatically explaining slang english terms in a sentence and introduced another dataset extracted from the urban dictionary. @xcite proposed to further rely on local and global contexts to help the model disambiguate and generate better definitions. different from our approach, their work focused on unfamiliar words and phrases (many of which are polysemous words), rarely used idioms, or emerging entities, relying on wikipedia as a source. more recently, @xcite proposed to study the problem of definition specificity, creating a method for tuning a model to account for hyperfocused (over-specific) or highly general (underspecific) definitions. finally, @xcite proposed to unify the seminal ideas of reverse dictionary and definition modeling in a single model, with the goal of helping better understand word sense and embeddings. @xcite proposed to directly finetune an llm specifically for definition modeling. previous work discussed so far has mostly focused on definitions in the english language. there are a few exceptions, such as the work of @xcite , who presented the first study on definition modeling for the french language with the release of a dataset collected from le petit robert, and the work of dimas furtado et al. (2024), whose main goal was to collect a dataset for portuguese definition generation, leveraging several models. their approach is, however, still monolingual. we also note that over the past few years, dictionary datasets in several languages derived from wiktionary have been released, including english @xcite , french @xcite , italian @xcite . @xcite collected data for all available wiktionaries. these datasets, were not accompanied by models that leverage them, and we were unable to find other works using them to train or evaluate models on definition modeling. we think one critical point here is that wiktionaries are built on the base of crowdsourcing, where quality could be a concern. finally, we find previous works aiming to generate definitions in specialized fields, e.g., scientific and medical terms @xcite , biomedical terminology definition @xcite , or financial terminology @xcite . ## data we are interested in assessing the capabilities of multilingual models to describe words in multiple languages. therefore, we aim to collect highquality data for "traditional" dictionary definitions, where most common words in a given language are contained. this approach deviates from recent previous work in definition modeling, where the focus has been on domains of "uncommon" words like slang terms @xcite or rarely used idioms and emerging entities @xcite and specific domains such as scientific or financial terms @xcite @xcite , and only in english. to support our study, we utilize data for three of the indo-european languages with the largest number of speakers, according to wikipedia: spanish (485 m native speakers), portuguese (236 m native speakers), and french (80 m native speakers). as we are particularly interested in studying potential performance gains when training on multiple languages, we focus on these in particular, as they all derive from the romance branch and thus share many similarities. finally, we also consider german, another indo-european language but from the germanic family with 75 m native speakers. we deem german as the ideal candidate to experiment together with existing datasets for the english language. ## experimental setup data an important distinction between our corpus and recent prior work is that our collected data does not contain examples of word usage. while we acknowledge the importance of such context to disambiguate a specific meaning for a given term in the case of polysemy, we opt to explore monosemic words as a first step into the study of multilinguality for definition modeling. we thus select the subset of terms that exhibit only a single meaning, which in practice we achieve by simply selecting the terms with a single definition entry. the final word lists are split into the 80/10/10 ratio. ## conclusions in this paper, we presented the first multilingual study on definition modeling, introducing monolingual dictionary data for german, spanish, french and portuguese. results show that llms can provide generally better results compared to smaller multilingual lms trained on monosemic dictionary data, but they can be unstable and generate outputs outside the target language. furthermore, techniques such as icl and finetuning did not lead to significantly better quality output. finally, we show that our task can serve as a natural, stable alternative to assess the proficiency of llms in a language. for future work, we would like to incorporate more languages and perform further human evaluation to better understand the relationship between human assesments and automatic metrics. we would also like to tackle the disambiguation problem by collecting usage examples to tackle polysemy. our supplementary material offers preliminary evidence suggesting that using llms could be a viable alternative for this ( §f). funded by project jpnp20006, commissioned by the new energy and industrial technology development organization (nedo). the computational resources of the ai bridging cloud infrastructure (abci), provided by aist, were used for most of the experiments in this paper. we are also grateful to the nvidia corporation, which donated one of the gpus we used. finally, we would also like to thank our volunteer annotators daniela serien, carolina dias alexiou, mauricio cordova, lucie kunitomo-jacquin, antoine jacquin and denis higashi imoto, for their williness to help and feedback.