Putting W ord N et’s Dictionary Examples in the Context of Definition Modelling: An Empirical Analysis (2022)

Abstract: Definition modeling is the task to generate a valid definition for a given input term. This relatively novel task has been approached either with no context (i.e., given a word embedding alone) and, more recently, as word-in-context modeling. Despite their success, most works make little to no distinction between resources and their specific features (e.g., type and style of definitions, or quality of examples) when used for training. Given the high diversity lexicographic resources exhibit in terms of topic coverage, style and formal structure, it is desirable for downstream definition modeling to better understand which of them are better suited for the task. In this paper, we propose an empirical evaluation of the well-known lexical database WordNet, and specifically, its dictionary examples. We evaluate them both directly, by matching them against criteria for good dictionary writing, and indirectly, in the task of definition modeling. Our results suggest that WordNet’s dictionary examples could be improved by extending them in length, and incorporating prototypicality.

## introduction definition modeling (dm), as introduced by @xcite , is the task of generating a dictionary definition for a given word. this task was made possible by the adoption in nlp of sequenceto-sequence architectures based on rnns @xcite . recently, dm systems have shown impressive performance in several intrinsic and downstream tasks, mostly thanks to being able to go from context-less @xcite only used the definiendum foot_0 as a conditioning token at all timesteps) to a contextually richer setting, e.g., by conditioning the generated definition to an example of usage of the target word @xcite @xcite @xcite . recently, a notable leap in dm was achieved in @xcite , who fine-tuned bart @xcite on example-definition pairs, and reported high results in intrinsic benchmarks and, more importantly, used their dm system for downstream nlp, specifically word sense disambiguation (wsd) and word-in-context classification. dm has also been explored from other perspectives, e.g., generating definitions with appropriate specificity using re-ranking mechanisms @xcite , or extending the generation cross-entropy loss with a reconstruction objective @xcite (reminiscent of works that used dictionary definitions for improving word embeddings via autoencoders @xcite or lstms @xcite ). moreover, @xcite explore a bart-based model for performing the reverse task to dm, i.e., exemplification modeling, or generating a dictionary example given a term and its definition. other applications of dm range from the aforementioned lexical semantics tasks to reverse dictionary (predict a word given a definition), interpretability, or for clarifying technical and medical terminology @xcite , whereas recent applications of bart to tasks not originally designed to be solved generatively are semantic role labeling @xcite , relation extraction (cabot and navigli, 2021) or entity linking @xcite . despite the above successes, little attention has been paid so far to the quality of the dictionary examples (or contexts) used for fine-tuning these models. in fact, most existing dm systems train on wordnet (wn) @xcite , which is the defacto lexical database for english. however, we are not aware of previous work that has explored the quality (and hence, suitability for dm) of wn examples. therefore, in this paper, we first inves-tigate the quality of wn examples by evaluating against the gdex (good dictionary examples) set of criteria @xcite , and use as a point of comparison a widely adopted open dataset used in dm, which is primarily based on the oxford dictionary @xcite ) (cha). it is worth noting, however, that these two resources were built for different objectives, as the initial purpose behind creating wn was to explain how lexical meaning is stored in the mind @xcite , and its primary use may be as a sense inventory @xcite . however, with this caveat in mind, and given how lexicographic resources are currently converging into useful pretraining and fine-tuning datasets for lexical semantics, we also propose to extrinsically test these two resources in the dm task. specifically, in our second set of experiments we fine-tune a bart-based model on wn and cha, and show that generally speaking, results of models fine-tuned on wn perform slightly worse than if fine-tuned on cha. our preliminary results suggest that wn's examples sometimes do not provide enough context, making it difficult to learn a good representation for the word being contextualized. we also report an experiment comparing dm modeling results on wn nouns vs. wn verbs; which suggests that a dm model trained on wn nouns performs slightly better. ## data wordnet (wn) is an electronic lexical dictionary for english that describes words (11,7097 nouns, 11,488 verbs, 22,141 adjectives, and 4,601 adverbs) organized in groups of synonyms called "synsets" @xcite . each synset is described by its definition, lemmas, examples of usage (for some but not all words), and the relations between synsets, e.g., hypernymy (is-a), meronymy (is-part) or troponymy (manner-of). wn has typically been used in lexicographic and language learning settings @xcite , but more importantly, also in nlp, e.g., as a natural language interface for optimizing the precision of search engines, wsd or query expansion @xcite . moreover, relations in wn have been used extensively, for example for improving word embeddings via retrofitting @xcite @xcite . cha @xcite , the other resource we consider in this paper, is based on oxford dictionaries. it was released with two splits, namely seen, where definitions in the training set also exist in the test set, and unseen, which contains a set of words not available in the training set @xcite . this is similar to the lexical splits (as opposed to random splits) present in other analogous tasks such as graded lexical entailment @xcite . in this paper, we are concerned with the quality of examples in wn (and how they compare with cha), i.e., sentences where a target word appears, and which should be informative enough to convey the necessary contextual information to clarify fully or partially the word's meaning (encoded in a natural language definition or gloss, instead of e.g., a word embedding). we show in data lemma definition example wn people (plural) any group of human beings (men or women or children) collectively old people cha people human beings in general or considered collectively each day he has looked at a key issue facing us as a nation as a people as frail human beings wn sheet any broad thin expanse or surface a sheet of ice cha sheet a large rectangular piece of cotton or other fabric used on a bed to cover the mattress and as a layer beneath blankets when these are used mary quietly got off the bed and covered him with the sheet and blanket wn tall great in vertical dimension; high in stature tall people cha tall of great or more than average height especially with reference to an object relative to width the elevator came to a stop and the doors slid open revealing the sixth floor of the tall building ## experiments in this section, we introduce the two sets of experiments we perform. first, the descriptive comparison between wn and cha examples using gdex as a proxy (section 3.1). second, we describe the setting for the dm experiment, where we test wn as supervision signal (section 3.2). ## analysis in this section, we discuss the results of our two experiments, namely gdex-wise comparison between wn and cha, and wn's stress test in the dm task. 2 we implemented our experiments using the simpletransformers ( http:// simpletransformers.ai/ ) library, a wrapper on top of transformers @xcite . ## conclusion definition modeling is the task to generate a dictionary definition given an input word and, optionally, some context. while different lexicographic resources are used as supervision for dm systems, there is little work analyzing their intrinsic quality. our evaluation is focused on the examples available in wordnet and the oxford dictionary, where we train a sequence-to-sequence definition modeling architecture based on bart using these two dictionaries. we found that wn's dictionary examples are written in a style that may make them hard to learn (especially verbs), and that they are, generally, (perhaps too) short. for the future, we would like to explore extrinsic evaluations and perform additional experiments with other datasets and language models.