VCDM : L everaging V ariational Bi-encoding and D eep Contextualized W ord R epresentations for I mproved D efinition M odeling (2020)

Abstract: In this paper, we tackle the task of definition modeling, where the goal is to learn to generate definitions of words and phrases. Existing approaches for this task are discriminative, combining distributional and lexical semantics in an implicit rather than direct way. To tackle this issue we propose a generative model for the task, introducing a continuous latent variable to explicitly model the underlying relationship between a phrase used within a context and its definition. We rely on variational inference for estimation and leverage contextualized word embeddings for improved performance. Our approach is evaluated on four existing challenging benchmarks with the addition of two new datasets, “Cambridge” and the first non-English corpus “Robert”, which we release to complement our empirical study. Our Variational Contextual Definition Modeler (VCDM) achieves state-of-the-art performance in terms of automatic and human evaluation metrics, demonstrating the effectiveness of our approach.

## introduction in most current nlp tasks, fixed-length vector representations of words, word embeddings, are used to represent some form of the meaning of the word. in the case of humans, however, oftentimes we will use a sequence of words known as a definition -a statement of the meaning for a term-to express meanings of terms (words, phrases, or symbols). it is with this in mind that the question of "can machines define?" is aimed to be answered with the task of definition modeling @xcite . definition modeling can be framed as a task of conditional generation, in which the definition d of the word or phrase is generated given a conditioning variable w such as a word's associated word embedding or other representations of context. current approaches for this task @xcite @xcite are mainly encoderdecoder based, in which one encodes a contextual representation for a word/phrase w using a variety of features such as context or character composition, and uses the contextual representation(s) to generate the definition d. despite the relative success of existing approaches for definition modelling, their discriminative nature -where distributional-derived information is at one end of the model and lexical information is at the other-limits their power as the underlying semantic representations of the distributional and lexical information are learned in an implicit rather than direct way. for example, although @xcite successfully showed that both local and global contexts are useful to disambiguate meanings of phrases in certain cases, their approach heavily relies on an attention mechanism to identify semantic alignments between the input phrase and the output definition, which may introduce noise and ultimately be insufficient to capture the entire meaning of each phrase-definition pair. to tackle this issue, we propose to explicitly model the underlying semantics of phrase-definition pairs by introducing a continuous latent variable z over a definition space, which is used in conjunction with w to guide the generation of definition d. the introduction of this latent representation enables us to treat it as a global defining signal during the generation process, complementing existing alignment mechanisms such as the attention. although the latent definition variable enables us to explicitly model underlying semantics of contextdefinition pairs, the incorporation of it into the task renders the posterior intractable. in this paper we recur to variational inference to estimate this in-tractable posterior, effectively making our model a conditional variational autoencoder and evolving the generation process from p(d|w) to p(d|w, z). we also note that existing approaches for definition modelling heavily rely on word embeddings, which due to their fixed nature can only capture so much of the semantics, being known to offer limited capabilities when dealing with polysemy. considering the success of pretrained deep contextualized word representations which by specifically addressing these limitations have been shown to improve performance on a variety of downstream nlp tasks @xcite , in this paper we propose a mechanism to integrate deep contextualized word representations in the definition modelling task. specifically, we successfully leverage bert @xcite as our contextual encoder and our definition encoder to produce representations for w and d respectively. finally, we develop two new datasets for this task, one derived from the cambridge dictionary foot_1 , and the other derived from le petit robert foot_2 . in summary, our contributions are: datasets and pre-trained models will be publicly released to the greater nlp community to help facilitate further advances on this task upon acceptance of this paper. our work is related to the seminal paper by @xcite , who proposed using the definitions found in everyday dictionaries as a means of bridging existing gaps between lexical and phrasal semantics. effectively, they train a language model to map dictionary definitions to lexical representations of words, presenting the task of reverse dictionaries, where the goal is to return the name of a concept given a definition. noraset et al. (2017) later introduced the task of definition modeling, in which a model is tasked with generating a definition for a given word, given its respective embedding. the authors argued that, compared to other related tasks such as word similarity or analogical relatedness, definition generation can be considered a more transparent view of the information captured by an embedding. however, this method does not incorporate contextual information, preventing it from generating appropriate definitions for polysemic words. addresing this, gadetsky et al. (2018) studied the problem of polysemy in definition modeling, introducing an attention-based model which uses contextual information determine components in the embedding which may refer to a relevant word meaning. ni and wang (2017) explore a different but related problem, proposing an approach for automatically explaining non-standard english expressions (i.e. slang) in a given sentence. they present a hybrid word-character sequence-to-sequence model that directly explains unseen non-standard expression, garnering reasonable definitions of expressions given their context. more recently, @xcite have tackled some of the limitations of previous works on definition modelling and non-standard english expression explanation. concretely, they note that whenever it is not possible to figure out the meaning of a given expression from its immediate local context, it is common to consult dictionaries for definitions or search documents or the web to find other global context to help in interpretation. in light of this, they introduce the task of describing a given phrase in natural language, based on its local and global contexts. to tackle this the authors introduce a model which consists of two context encoders (one for the local context, and one for the global context) as well as a description decoder. our proposed model, uses a more practical variational encoder-decoder framework, allowing us to take advantage of explicitly modeling the phrasedefinition relationship, while also leveraging deep contextualized word representations for more informative context representations. finally, our model is also related to @xcite , in which conditional variational autoencoders (cvaes) -an extension of the original variational autoencoder (vae) (kingma and welling, 2014)-were proposed for generating diverse structured output, mainly in the context of image generation, and visual object segmentation and labeling. our work is also related to cvae models that have been developed for the domain of natural language processing, specifically @xcite who proposed a cvae in the context of neural machine translation (nmt). as the usage of vaes has become relatively common, we will omit a detailed explanation of these models, referring readers to @xcite . ## proposed approach in a way analogous to previous work @xcite , our proposed approach is a generative probabilistic model for word definitions, in which the goal is to estimate the probability of generating a definition d, given an input w. concretely, we propose to directly capture the joint semantics of the (w, d) pairs by introducing a latent variable z to model the underlying definition space. our proposed generative process can be formulated as follows: since the introduction of our latent variable makes posterior inference intractable, in this paper we resort to variational inference to perform posterior approximation. effectively, this makes our proposed generative model a cvae @xcite such that the variational lower bound can be formulated as follows: in the following subsections we give details on how we specifically model each one of these components. with this in mind, we develop the following architecture comprised of 3 major components: notation for clarity when explaining the approach, we define the notation conventions we will follow, namely: d refers to dimensions, c refers to the context vectors produced by the attention mechanism, g refers to projections/activations, and h refers to sets of vectors. ## conclusion in this paper we have introduced a generative model that directly combines distributional and lexical semantics via a continuous latent variable for the task of definition modeling. empirical results on multiple corpora, including two new datasets released, show that our model is able to outperform previous work by a consistent margin, also successfully being able to leveraging contextualized word representations. for future work we are interested in exploring how definition modeling could be adapted to a multilingual or cross-lingual setting.