Definition Modeling : To model definitions. Generating Definitions With Little to No Semantics (2023)

Abstract: Definition Modeling, the task of generating definitions, was first proposed as a means to evaluate the semantic quality of word embeddings—a coherent lexical semantic representations of a word in context should contain all the information necessary to generate its definition. The relative novelty of this task entails that we do not know which factors are actually relied upon by a Definition Modeling system. In this paper, we present evidence that the task may not involve as much semantics as one might expect: we show how an earlier model from the literature is both rather insensitive to semantic aspects such as explicit polysemy, as well as reliant on formal similarities between headwords and words occurring in its glosses, casting doubt on the validity of the task as a means to evaluate embeddings.

## introduction definition modeling @xcite is a recently introduced nlp task that focuses on generating a definition gloss given a term to be defined; most implementations rely on an example of usage as auxiliary input @xcite @xcite . in the last few years, it has been the focus of more than a few research works: datasets have been proposed for languages ranging from japanese @xcite to @xcite , and defmod has even been the subject of a recent semeval shared task @xcite . practical applications for defmod abound, from the generation of lexicographic data for lowresource languages @xcite , to computer-assisted language learning @xcite , creating learners' dictionaries @xcite , and from explaining slang @xcite to clarifying scientific terminology (august * equal contribution. @xcite ). yet, it was initially conceived by @xcite as an evaluation task for word embeddings. if a word embedding is a coherent lexical semantic representation, then it ought to contain all the information necessary to produce a coherent gloss. researchers have kept this semantic aspect firmly in mind: for instance, @xcite argue that defmod provides a means to dispense word-sense disambiguation (wsd) applications from fixed, rigid sense inventories. more broadly, dictionaries in nlp are often used to capture some aspect of semantics. this point bears closer inquiry. one may expect that writing definitions requires some knowledge of the meaning of the headword, but little has been done to confirm this expectation. here, we focus on empirically verifying what impacts a model's ability to generate valid definitions. as such, our interest lies mostly in examining what factors in the performance of a successful definition modeling system, rather than in the engineering aspects of defmod implementations. we therefore re-purpose the fine-tuning protocol of @xcite to train a bart model @xcite to generate definitions, which we subsequently evaluate on infrequent words: as bevilacqua et al. have extensively demonstrated the quality of their model on english data, it is suitable for our own endeavor. our findings suggest that it is possible to generate definition with little semantic knowledge: our defmod system, far from manipulating semantic information, mostly relies on identifying morphological exponents and tying them to lexicographic patterns. semantic aspects of the headword-e.g., its polysemy or frequency-do not appear to weigh on model performances as captured through automatic metrics. ## related works there is a broad domain of research that focuses on nlp solutions to lexicography problems and assessing how suitable they are (e.g., @xcite @xcite . conversely, many nlp works have used dictionaries to address semantic tasks, such as hypernym or synonym detection @xcite word-sense-disambiguation @xcite @xcite , compositional semantics @xcite @xcite , interpretability (chang and chen, 2019), representation learning @xcite or word retrieval (siddique and sufyan beg, 2019, a.k.a. reverse dictionaries). we more narrowly concerned ourselves with definition modeling @xcite , formulated as a sequence-to-sequence task @xcite @xcite . our fine-tuning approach is borrowed from @xcite ; note that @xcite also employed a plm (viz. t5, @xcite . we refer readers to @xcite for a more thorough introduction. ## model & dataset datasets we retrieve data from dbnary @xcite , foot_0 an rdf-formatted dump of wiktionary projects. 2 this source of data has previously been used to build defmod datasets @xcite , and is available in multiple languages-a desirable trait for future replication studies. more details are provided in appendix b. for each term to be defined, we also tabulate its number of occurrences by tallying the number of string matches in a random subset of 5m documents from the deduplicated english oscar corpus @xcite . headword frequency is worth focusing on, for at least two reasons. first, lexicographers are more likely to cover frequent words: dictionarymakers often espouse a data-driven approach to determine whether words should be included in general or specialized dictionaries @xcite ); foot_2 second, dictio-nary users should also be less familiar with rarer words-and likely require definitions. hence, we set aside definitions where the headword has five or fewer occurrences in our oscar subset for test purposes only, and further distinguish low-frequency headwords depending on whether they are attested in our oscar sample. remaining headwords are then split 80-10-10 between train, validation, and a second held out test set, so as to also measure models on identically distributed items. as such, we have three test sets, distinguished by the frequency of the headword in our oscar sample: we note as # = 0 the test set comprised of forms unattested in the sample; # ≤ 5 corresponds to headwords with five or fewer occurrences; # > 5 matches with train set and validation set conditions. model the core of our methodology is borrowed from bevilacqua et al. (2020): we fine-tune a generative pretrained language model, namely @xcite , to produce an output gloss given an input example of usage, where the term to be defined is highlighted by means of special @xmath0. we justify our adoption of their methodology by the fact that they report high results, through extensive nlg and wsd evaluation: as such, the approach they propose is representative of successful modern approaches to defmod, and is suitable for a study such as ours. we refer the reader to their paper and appendix a for details. we expect defmod systems to be sensitive to the variety of examples of usages and number of target glosses: more examples of usage should lead to higher performances, whereas not exposing the model to polysemy should be detrimental. this can be tested by down-sampling the training set, so as to select one gloss per headword (1g or ∀g) and/or one example of usage per gloss (1e or ∀e). this leads us to defining four related models: ∀g∀e, ∀g1e, 1g∀e, and 1g1e. seen headwords e.g. in bevilacqua et al. (2020). 5 they also highlight a strikingly consistent behavior across all four configurations: mann-whitney u tests stress that we do not observe lower performances for rarer words, as one would naively expect, except in few cases (∀g∀e, ∀g1e and 1g1e models, when comparing unattested and rare headwords) with relatively high p-values given the sample sizes (@xmath1.01 always). another way to stress the lack of effect related to explicit polysemy or contextual diversity consists in correlating bleu scores across models: comparing the bleu scores obtained by one model (say the ∀g∀e) to those of another model (e.g., the 1g1e model) indicates whether they behave differently or whether bleu scores are distributed in roughly the same fashion. we systematically observe very high pearson coefficients (0.82 < @xmath2.90). in other words, definitions that are poorly handled in any model will in all likelihood be poorly handled in all other models, and definitions that are easy for any single model will be easy for all other models. we provide a breakdown per split and per model in appendix c, ## digging further: manual evaluation to better understand model behavior, we sample 50 outputs of the ∀g∀e model, per bleu quartile, for the validation split and our three test splits. we then annotate these 800 items as follows. ## conclusions in this work, we have presented how an earlier definition modeling system was able to achieve reasonable performances and produce fluent outputs, although the factual validity leave much to be desired. this behavior is almost entirely due to morphologically complex headwords, for which the model is often able to derive reasonable glosses by decomposing the headword into a base and an exponent, and mapping the exponent to one of a limited set of lexicographic patterns. the model we studied seems more sensitive to formal traits than to explicit accounts of polysemy. there are numerous limitations to this work: we focused on one specific fine-tuning approach for one specific english plm. nonetheless, we have shown that models can achieve reasonable performances on defmod without relying on semantics, casting doubt on the task's usefulness for word embedding evaluation, as initially suggested by @xcite in other words: using lexicographic data as inputs for an nlp model does not ensure that it will pick up on the semantic aspects contained therein. this work is part of the fotran project, funded by the european research council (erc) under the eu's horizon 2020 research and innovation program (agreement № 771113). we also thank the csc-it center for science ltd., for computational resources. references tal august, katharina reinecke, and noah a. smith. 2022. generating scientific definitions with controllable complexity. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 8298-8317, dublin, ireland. association for computational linguistics. satanjeev banerjee and alon lavie. 2005. meteor: an automatic metric for mt evaluation with improved correlation with human judgments. in proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72, ann arbor, michigan. association for computational linguistics. diego bear and paul cook. 2021. cross-lingual wolastoqey-english definition modelling. in proceedings of the international conference on recent advances in natural language processing (ranlp 2021), pages 138-146, held online. incoma ltd. michele bevilacqua, marco maru, and roberto navigli. 2020. generationary or "how we went beyond word sense inventories and learned to gloss". in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 7207-7221, online. association for computational linguistics. tom bosc and pascal vincent. 2018. auto-encoding dictionary definitions into consistent word embeddings. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 1522-1532, brussels, belgium. association for computational linguistics. ting-yun chang and yun-nung chen. 2019. what does this word mean? explaining contextualized embeddings with natural language definition. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 6064-6070, hong kong, china. association for computational linguistics. definition generation. in proceedings of the 21st chinese national conference on computational linguistics, pages 921-931, nanchang, china. chinese information processing society of china. adam kilgarriff, miloš husák, katy mcadam, michael rundell, and pavel rychlý. 2008. gdex: automatically finding good dictionary examples in a corpus. in proceedings of the 13th euralex international congress, pages 425-432, barcelona, spain. institut universitari de linguistica aplicada, universitat pompeu fabra. cunliang kong, yun chen, hengyuan zhang, liner yang, and erhong yang. 2022. multitasking framework for unsupervised simple definition generation. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 5934-5943, dublin, ireland. association for computational linguistics. michael lesk. 1986. automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone. in proceedings of the 5th annual international conference on systems documentation, sigdoc '86, page 24-26, new york, ny, usa. association for computing machinery. mike lewis, yinhan liu, naman goyal, marjan ghazvininejad, abdelrahman mohamed, omer levy, veselin stoyanov, and luke zettlemoyer. 2020. bart: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 7871-7880, online. association for computational linguistics. chin-yew lin. 2004. rouge: a package for automatic evaluation of summaries. in text summarization branches out, pages 74-81, barcelona, spain. association for computational linguistics. timothee mickus, timothée bernard, and denis paperno. 2020. what meaning-form correlation has to compose with: a study of mfc on artificial and natural language. in proceedings of the 28th international conference on computational linguistics, pages 3737-3749, barcelona, spain (online). international committee on computational linguistics. timothee mickus, denis paperno, and matthieu constant. 2019. mark my word: a sequence-to-sequence approach to definition modeling. in proceedings of the first nlpl workshop on deep learning for natural language processing, pages 1-11, turku, finland. linköping university electronic press. timothee mickus, kees van deemter, mathieu constant, and denis paperno. 2022. semeval-2022 task 1: codwoe -comparing dictionaries and word embeddings. in proceedings of the 16th international workshop on semantic evaluation (semeval-2022), pages 1-14, seattle, united states. association for computational linguistics.