Embedding Structured Dictionary Entries (2020)

Abstract: Previous work has shown how to effectively use external resources such as dictionaries to improve English-language word embeddings, either by manipulating the training process or by applying post-hoc adjustments to the embedding space. We experiment with a multi-task learning approach for explicitly incorporating the structured elements of dictionary entries, such as user-assigned tags and usage examples, when learning embeddings for dictionary headwords. Our work generalizes several existing models for learning word embeddings from dictionaries. However, we find that the most effective representations overall are learned by simply training with a skip-gram objective over the concatenated text of all entries in the dictionary, giving no particular focus to the structure of the entries.

## introduction while word embedding models are typically trained using large text corpora with objectives based on distributional semantics, recent work has shown how to take advantage of external resources like wordnet @xcite and other manually created dictionaries in order to better capture wordlevel semantic relationships of interest. for example, previous work has used the graph structure of external resources to post-process pre-trained word embeddings, enforcing that the similarity between embeddings reflects the similarity inferred from the graph structure of lexicons like wordnet @xcite . following in a similar principle, others use known synonymy and antonymy relationships between words to adjust the distance between word embeddings @xcite . other work uses traditional dictionaries to improve the overall coverage of word embedding models by creating embeddings for rare words be leveraging information from their definitions @xcite . while dictionaries have been shown to be useful, most previous work has focused only on using the text of the definitions in order to learn word representations. however, many dictionaries include additional structural elements such as usage examples, quotations containing the headword, tags, labels, and more. for some online crowd-built dictionaries, information such as the contributing users and even upvotes and downvotes are available. we conjecture that such meta information may prove useful and, therefore, we seek to leverage all of this additional information to build improved representations of the words defined in a given dictionary. to do this, we generalize the consistency-penalized autoencoder (cpae) @xcite to allow for not only the reconstruction of dictionary definitions, but also for making predictions about the other structural elements available, such as usage examples and user-assigned tags. we make the following contributions in this paper: (1) we propose a flexible, multi-task learning extension to the cpae model that can be used to produce embeddings from structured dictionary entries, (2) we evaluate the applicability of this extended model to three english-language dictionary datasets, each with their own unique characteristics and sets of structural elements, and (3) we demonstrate the a simple baseline approach for learning word embeddings, based on the popular skip-gram with negative sampling framework, can often lead to representations that better capture word-level semantic similarity according to a range of commonly used evaluation tasks. ## auto-encoding structured entries with multi-task learning next, we present an approach for learning word embeddings that implicitly encode a wide range of the elements that are present in a dictionary entry. given a word defined in a dictionary, the objective of the model is to accurately recover as much structural information as possible, including the word's definition, usage examples, tags, and authors. we also leverage user provided votes as a means of sorting and filtering the dictionary entries. the model takes a word's definition as input, and learns a transformation from the words in the definition to an embedding that contains features that describe the structural elements of the dictionary entry for the word. we treat the prediction of each type of structural element as a separate task within a multi-task learning framework. ## evaluation and results we evaluate all produced embeddings foot_3 across a range of intrinsic evaluation tasks as used in @xcite . 6 for these word-level semantic similarity tasks, the machine generated scores (cosine similarity between the produced word embeddings) are compared against human-labeled similarity scores by computing the correlation between the two sets of scores. the tasks involved include the marco, elia and nam (men) annotated word pairs based on image captioning data @xcite , the simverb (sv) verb similarity dataset @xcite , both of which have standardized development and testing splits. we use the development splits of these datasets in order to tune our models. the wordsim-353 (ws) dataset contains both similarity (ws-s) and relatedness (ws-r) annotations for the same sets of words, allowing us to examine the ability of our models to capture each of these semantic relations. we also evaluate using the simlex-999 dataset and a subset of that data, simlex-333 (sl999 and sl333) @xcite . the sl333 subset contains only the 333 most related pairs according to the human annotations. stanford's contextual word similarities (scws) dataset @xcite , the 65 word pairs studied by rubenstein and goodenough (rg65) 1965, the mechanical turk (mt) dataset @xcite , and the rare words (rw) dataset @xcite round out the rest of our evaluatoin tasks. for models that use our proposed architecture, we initialize the input embeddings using the baseline pre-trained skip-gram embeddings. we train these embeddings ourselves in the case of wordnet and wikitionary, and use the ud-basic embeddings released by @xcite for urban dictionary. 7 ## conclusions we show that the extension of the cpae model to include additional structural elements can provide some gains in word-level semantic similarity tasks, however, the the extra complexity of this approach is unnecessary for learning useful word embeddings, and in many cases, leads to degradation in the scores across a range of standard word embedding evaluation metrics in comparison to simpler approaches. to build general purpose word embeddings from a sufficiently large dictionary (i.e., containing at least several hundred million tokens of text), our recommendation is to simply concatenate all of the structural elements together as a single text, inserting the entry headword between each element, and applying the widely popular skip-gram architecture to this text to learn traditional distribution embeddings. this approach requires only a single learning objective, trains in much less time, and achieves competitive results in many cases, making it an easier alternative to explicitly leveraging structural information from dictionary entries while still creating useful embeddings. future work should explore how these approaches would work when applied to more english dictionaries such as the oxford english dictionary 9 in order to better understand the effects of using a more standardized dictionary to learn embeddings. further, dictionaries in other languages, particularly lower-resource languages, should be considered, since our results suggest that the approaches described in this paper outperform the baseline approach mostly in settings where the total amount of text in the dictionary is small. formally, let @xmath0, w 1 in , . . . , w n in } be a sequence of tokens in the definition for w h . the elements of d belong to the vocabulary of all words that appear in definitions, v in , and w h belongs to the vocabulary of all headwords, v h . in the case of polysemous words which have more than one meaning, we concatenate the tokens from all definitions together into a single sequence, and separate them by a special sep token. in , e 1 in , . . . , e n in } is the set of d indimensional embeddings representing words in the definition. these embeddings can be learned during training, or pre-initialized and frozen, as discussed later in this section. the embeddings are passed into an encoder layer in order to produce a single d h -dimensional embedding @xmath1. the encoder can be any type of model that takes a variable-length sequence of embeddings as input and produces a single, fixedlength embedding as output. this embedding is then fed to any number of decoders, each with their own specific objective and loss function. the goal of each decoder's loss is to influence the weights of the encoder to produce an embedding h that is most useful for capturing a specific structural element of the dictionary entry for w h , or to retain some other important property of the embedding h. in the following subsections, we describe the decoders that we use and their associated loss, which become components in the overall loss function for our model: for up to n objectives, each with its own associated weight term. these weights can be used to control the overall influence of the objective in the final loss computation.