Generationary or “How We Went beyond Word Sense Inventories and Learned to Gloss” (2020)

Abstract: Mainstream computational lexical semantics embraces the assumption that word senses can be represented as discrete items of a predefined inventory. In this paper we show this needs not be the case, and propose a unified model that is able to produce contextually appropriate definitions. In our model, Generationary, we employ a novel span-based encoding scheme which we use to fine-tune an English pre-trained Encoder-Decoder system to generate glosses. We show that, even though we drop the need of choosing from a predefined sense inventory, our model can be employed effectively: not only does Generationary outperform previous approaches in the generative task of Definition Modeling in many settings, but it also matches or surpasses the state of the art in discriminative tasks such as Word Sense Disambiguation and Word-in-Context. Finally, we show that Generationary benefits from training on data from multiple inventories, with strong gains on various zero-shot benchmarks, including a novel dataset of definitions for free adjective-noun phrases. The software and reproduction materials are available at http://generationary.org .

## introduction virtually all modern approaches to word sense disambiguation (wsd), i.e. the task of automatically mapping a word in context to its meaning @xcite , use predetermined word senses from a machine lexicon, both in supervised @xcite @xcite and in knowledge-based settings @xcite @xcite . nevertheless, researchers in natural language processing (nlp), lexical semantics, and lexicography, have long been warning the community about the cognitively inaccurate * these authors contributed equally. nature of discrete sense boundaries @xcite @xcite . as @xcite argued, different language users have different understandings of words. this fact explains why inter-annotator agreement (ita) estimates on wsd annotation tasks have never exceeded the figure of 80% @xcite @xcite . moreover, this casts doubt upon the reliability of human-made inventories and "gold standard" evaluation datasets @xcite . having no indisputable way of determining where one sense of a word ends and another begins, together with the fact that little consensus about how to represent word meaning has hitherto existed @xcite @xcite , are issues lying at the core of what makes wsd hard @xcite . moreover, while english inventories of senses and corpora are widely available the same cannot be said for other languages @xcite @xcite , and this limits the scalability of natural language understanding tasks to multiple languages @xcite . in this paper we overcome these limitations by proposing a unified approach to computational lexical semantics that has as its central focus definition modeling (dm), i.e. the task of generating a gloss foot_0 from static or contextual embeddings @xcite . generating a meaning description (definiens) to define a given term in context (definiendum) avoids many of the concerns highlighted above, in that we are not limited to a preexisting list of meanings. we show that we can use a single generation model, i.e. generationary, not just to compete on the dm benchmarks, but also to achieve strong results on fully-discriminative tasks such as wsd and the recently-proposed word-in-context @xcite . this, in turn, results in a more solid assessment of the generation quality, a notorious problem in natural language generation (nlg) evaluation @xcite . in contrast to previous approaches in dm @xcite , we dispense with the requirement of having the definiendum represented by a single vector, and we condition gloss generation on a context of which the definiendum is an arbitrary span. this allows for the generation of contextual definitions for items that are rarely covered by sense inventories, such as free word combinations (e.g. clumsy apology or nutty complexion). finally, the generative formulation makes it possible to train on several lexicographic resources at once, resulting in a versatile model that performs well across inventories, datasets, and tasks. the main contributions of our approach are as follows: we envision many possible applications for generationary, such as aiding text comprehension, especially for second-language learners, or extending the coverage of existing dictionaries. ## related work recent years have witnessed the blossoming of research in definition modeling (dm), whose original aim was to make static word embeddings intepretable by producing a natural language definition @xcite . 2 while subsequently released datasets have included usage examples to account for polysemy @xcite , many of the approaches to "contextual" dm have nevertheless exploited the context merely in order to select a static sense embedding from which to generate the definition @xcite @xcite . such embeddings, however, are non-contextual. other works have made a fuller use of the sentence surrounding the target, with the goal of explaining the meaning of a word or phrase as embedded in its local context @xcite @xcite . however, these approaches have never explicitly dealt with wsd, and have shown limits regarding the marking of the target in the context encoder, preventing an effective exploitation of the context and making dm overly reliant on static embeddings or surface form information. for example, in the model of @xcite , the encoder is unaware of the contextual target, whereas @xcite use a marker embedding to represent targets limited to single tokens. finally, @xcite replace the target with a placeholder, and the burden of representing it is left to a character-level encoder and to static embeddings. this latter approach is interesting, in that it is the only one that can handle multi-word targets; however, it combines token embeddings via order-invariant sum, and thus it is suboptimal for differentiating instances such as pet house and house pet. recent approaches have explored the use of large-scale pre-trained models to score definitions with respect to a usage context. for example, @xcite proposed to recast dm as a definition ranking problem. a similar idea was applied in wsd by @xcite , leading to state-of-the-art results. however, both of these approaches fall back on the assumption of discrete sense boundaries, and are therefore unable to define targets outside a predefined inventory. with generationary, by contrast, we are the first to use a single encoder-decoder model to perform diverse lexical-semantic tasks such as dm, wsd and wic. moreover, we address the issue of encoding the target in context by using a simple, yet effective, encoding scheme which makes use of special tokens to mark the target span, producing a complete and joint encoding of the context without the need for other components. this allows the effective usage of a pre-trained model, which we fine-tune to generate a gloss given the context. ## generationary with this work we present a new approach to computational lexical semantics, by means of which we generate glosses for arbitrary-sized phrases in context. our work has a wider scope than its predecessors, in that we put forward a unified method that overcomes the limits of both dm and wsd. with respect to dm, our full sequence-to-sequence framing of the task enables us to deal with units having different compositional complexity, from single words to compounds and phrases. thus, generationary can gloss a definiendum that is not found in dictionaries, such as starry sky, with the appropriate definiens, e.g.: 'the sky as it appears at night, especially when lit by stars'. as regards wsd, instead, we are no longer bound by the long-standing limits of predefined sense inventories. thus, it is possible to give (i) a meaningful answer for words that are not in the inventory, and (ii) one that fits the meaning and the granularity required by a given context better than any sense in the inventory. consider the following: the target word in (1 a) is associated foot_2 with the gold gloss (1 b) from wordnet @xcite , the most used sense inventory in wsd. however, generationary arguably provides a better gloss (1 c). in what follows, we detail our approach. ## quantitative experiments we first perform a threefold automatic evaluation to test the strengths of generationary in different settings. on the one hand, we assess its ability to produce suitable definitions by testing the generation quality on the dm setting (section 5.1). on the other, we aim to further appraise how well the generated outputs describe the contextual meaning, by evaluating the performance they bring about on the discriminative benchmarks of wsd (section 5.2) and wic (section 5.3). foot_4 ## qualitative experiment given that the ability of generationary to produce fluent and meaningful definitions is its key asset, in addition to the automatic evaluation reported in section 5 we devised a qualitative experiment on two distinct datasets we constructed. while our previous experiments shed light upon the quality of generationary in comparison with other automatic systems, here we employ human annotators to compare definitions produced with our approach against glosses written by human lexicographers. the datasets that we use in this experiment are (i) our hei++ dataset of definitions for adjectivenouns phrases (section 4.2) and (ii) sampleval, i.e. a sample of 1,000 random instances made up of 200 items foot_7 for each of the five wsd datasets included in all (see section 5.2), with at most one total instance per sense. with hei++ we assess the ability of generationary to accurately gloss complex expressions, such as free phrases (e.g. wrong medicine or hot forehead), that are rarely covered by traditional dictionaries. with sampleval, instead, we test whether generated glosses can improve over gold definitions associated with gold senses in wordnet. ## generation examples in ## error analysis to have a broader picture of the quality of the outputs produced by means of generationary, we perform behavioural testing for our gen-uni model, in the spirit of @xcite . as a result, we can identify two main trends behind failures to generate an appropriate contextual definition, which we refer to as disambiguation errors and hallucinations, respectively. disambiguation errors when the model predicts a perfectly good definition for the target, but one that fits another common context of occurrence, a disambiguation error arises. for instance, given the c, t pair in (2 a), with the word pupil as the target, the model fails to identify the "aperture in the iris of the eye" sense, and instead produces an output gloss which is compatible with the meaning of the homograph (2 b): hallucinations other errors stem from the fact that the model can only rely on the knowledge about possible definienda that it is able to store in the parameters during the pre-training and training stages. thus, if the contextual knowledge is not sufficient to extrapolate a definition, the modelwhich is required to always generate an outputwill hallucinate an answer on the basis of contextual clues, incurring the risk of introducing nonfactualities. this particularly concerns named entities and domain-specific concepts, but the clearest examples can be seen with targets that do not correspond to any existing, fictional or non-fictional entity. for example, given the input sentence (3): in this case, the model picked the cue of the cartoonish squeaky mcduck character, and hallucinated the name of a plausible cartoon series for it. note that neither squeaky mcduck nor the cartoon series actually exist. ## conclusion with this work, we showed that generating a definition can be a viable, suitable alternative to the traditional use of sense inventories in computational lexical semantics, and one that better reflects the non-discrete nature of word meaning. we introduced generationary, an approach to automatic definition generation which, thanks to a flexible encoding scheme, can (i) encode targets of arbitrary length (including unseen multi-word expressions), and (ii) exploit the vast amount of knowledge encoded in the bart pre-trained encoder-decoder, through fine-tuning. from two points of view, generationary represents a unified approach: first, it exploits multiple inventories simultaneously, hence going beyond the quirks of each one; second, it is able to tackle both generative (definition modeling) and discriminative tasks (word sense disambiguation and wordin-context), obtaining competitive to state-of-theart results, with particularly strong performances on zero-shot settings. finally, human evaluation showed that generationary is often able to provide a definition that is on a par with or better than one written by a lexicographer. we make the software and reproduction materials, along with a new evaluation dataset of definitions for adjective-noun phrases (hei++), available at http://generationary.org . training is performed for at most 50 epochs. we employ a single nvidia geforce rtx 2080 ti gpu to perform all the reported experiments, with average runtimes per epoch of bart fine-tuning ranging from ca. 50 minutes (gen-sem) @xmath0. on the dm task, we evaluate on the best epoch, i.e. the one with the lowest cross-entropy loss on the dev set, with no hyperparameter tuning. on the wsd task, instead, we perform minimal hyperparameter tuning, with search trials just on beam size (testing with values of 1, 10, 25, and 50), choosing as the best configuration the one with the highest f1 on our dev set, semeval-2007; with simple similarity scoring, the best gen-sem has a beam size of 10, while, with mbrr similarity scoring, the best gen-sem has a beam size of 25. we use only mbrr with gen-uni, with a beam size of 10, resulting in the best performance on the development set. on the wic task we only perform tuning of the threshold on the dev set, by trying every value in range between the lowest and the highest z score, with a minimum step of 0.025. we compute similarities in batches of 125 pairs. for training and prediction of the models of ishiwatari et al. (2019) , we use the code provided by the authors. 16 we use the same hyperparameters, except that we increase the vocabulary size to 39,000, which results in much improved performances on our benchmarks.