Lacking the Embedding of a Word? Look it up into a Traditional Dictionary (2022)

Abstract: Word embeddings are powerful dictionaries, which may easily capture language variations. However, these dictionaries fail to give sense to rare words, which are surprisingly often covered by traditional dictionaries. In this paper, we propose to use definitions retrieved in traditional dictionaries to produce word embeddings for rare words. For this purpose, we introduce two methods: Definition Neural Network (DefiNNet) and Define BERT (DefBERT). In our experiments, DefiNNet and DefBERT significantly outperform state-of-the-art as well as baseline methods devised for producing embeddings of unknown words. In fact, DefiNNet significantly outperforms FastText, which implements a method for the same task-based on n-grams, and DefBERT significantly outperforms the BERT method for OOV words. Then, definitions in traditional dictionaries are useful to build word embeddings for rare words.

## introduction words without meaning are like compasses without needles: pointless. indeed, meaningless words lead compositionally to meaningless sentences and, consequently, to meaningless texts and conversations. second language learners may grasp grammatical structures of sentences, but, if they are unaware of the meaning of single words in these sentences, they may fail to understand the whole sentences, especially when there is an insufficient context for unfamiliar words. this is why a large body of natural language processing research is devoted to devising ways to capture word meaning. as language is a living body, distributional methods @xcite @xcite are seen as the panacea to capture word meaning as opposed to more static models based on dictionaries @xcite and other lexical resources @xcite . distributional methods may easily capture new meaning of existing words and, eventually, can easily assign meaning to emerging words. in fact, the different methods can scan corpora and derive the meaning of these new words by observing them in context @xcite @xcite . words are then represented as vectors -now called word embeddingswhich are then used to feed neural networks to produce meaning for sentences @xcite @xcite and meaning for whole texts @xcite . distributional methods have a strong limitation: word meaning can be assigned only for words where sufficient contexts can be gathered. rare words are not covered and become the classical out-of-vocabulary words, which may hinder the understanding of specific yet important sentences. to overcome this problem, n-grams based distributional models have emerged @xcite where word meaning is obtained by composing "meaning" of character n-grams forming a word. these n-grams act as proto-morphemes and, hence, meaning of unknown words can be obtained by composing meaning of proto-morphemes. traditional dictionaries can offer a solution to find meaning of rare words. they have been put aside since they cannot easily adapt to language evolution and they cannot easily provide distributed representations for neural networks. in this paper, we propose to use definitions in dictionaries to compositionally produce distributional representations for out-of-vocabulary (oov) words. trying to reproduce in a distributional setting the compositional properties that hold between symbols is a debated task since compositional distributional models were proposed @xcite @xcite @xcite . definitions in dictionaries are intended to describe the meaning of a word to a human reader. then, we propose two models to exploit definitions to derive the meaning of oov words: (1) definition neural network (definnet), a simple neural network; (2) defbert, a model based on pre-trained bert. we experimented with different tests and datasets derived from wordnet @xcite . firstly, we determined if definnet and defbert can learn to derive word meaning from definitions. secondly, we aimed to establish whether definnet and def-bert can cover oov words, which are not covered by word2vec @xcite or by the bert pre-trained encoder, respectively. in our experiments, definnet and defbert significantly outperform state-of-the-art as well as baseline methods devised for producing embeddings of unknown words. in fact, definnet significantly outperforms fasttext @xcite , which implements a method for the same task-based on ngrams, and defbert significantly outperforms the bert method for oov words. then, definitions in traditional dictionaries are useful to build word embeddings for rare words. ## background and related work out-of-vocabulary (oov) words have been often a problem as these oov words may hinder the applicability of many nlp systems. for example, if words are not included in a lexicon of a probabilistic context-free grammar, interpretations for sentences containing these words may have a null probability. hence, solutions to this problem date back in time. in the context of word embeddings, three families of solutions have been proposed: (1) contextbased methods, (2) form-based methods, (3) combination of previous. the first family includes methods addressing the issue of learning new terms from tiny data either tuning existing models (herbelot and @xcite or performing a linear transformation on the average of all context word embedding @xcite . in form-based methods, the most common solution is to use word n-grams @xcite or word pieces of variable length @xcite as proxies to model morphemes. embeddings are learned for 3-grams as well as for word pieces. in @xcite these 3-grams are then combined to obtain the embedding for the entire word. for example, the word cheerlessness, which contains 3 morphemes (cheer, less and ness), is modeled by using embeddings for ⃗ che, ⃗ hee, ..., ⃗ ess in the 3-gram approach and by using embeddings for ⃗ cheer and ⃗ lessness in the word pieces approach. these embeddings are possibly capturing information about the related morphemes. in this way, oov word embeddings are correlated with meaningful bits of observed words. these models are our baselines. the last family includes methods taking into account both contextual and morphological information @xcite @xcite . deriving word embeddings for oov words from dictionary definitions is an alternative approach. this approach has shown to be competitive in low resource scenarios in @xcite where an lstm model was fed with the definition. dictionary definitions have been used in early attempts to train rudimentary compositional distributional semantic models @xcite , which aimed to build embeddings for sequences of two words. in the word embedding field, several algorithms using definitions were proposed to build new embeddings matrices @xcite @xcite . however, those methods are alternatives to the corpus-based distributional ones while our method is focused on tackling the oov words problem, complementing existing word embedding spaces. lexical resources have been also used exploiting their underlying semantic graph as an additional source of information @xcite . however, models based on those semantic graphs rely on a stronger assumption than models based on definitions only. universal sentence embedders (uses) @xcite can play an important role in this novel approach. in fact, definitions are particular sentences aiming to describe meaning of words. therefore, uses should obtain an embedding representing the meaning of a word by composing embeddings of words in the definition. moreover, deriving word embeddings from definitions can be seen as a semantic stress test of universal sentence embedders. generally, the ability of uses @xcite @xcite to semantically model sentences is tested with end-to-end downstream tasks, for example, natural language inference (nli) @xcite @xcite , question-answering @xcite as well as dialog systems @xcite . uses such as bert @xcite are encoding semantic features in hidden layers @xcite . however, uses' success in downstream tasks may be due to superficial heuristics (as supposed in @xcite and @xcite ) and not to deep modeling of semantic features. therefore, our study can contribute to this debate. in fact, to the best of our knowledge, it is the first study aiming to investigate if uses can model meaning by producing embedding for words starting from their definitions. ## model this section introduces our proposals to use definitions in generating embeddings for out-ofvocabulary words: definition neural network (definnet) and bert for definitions (defbert). section 3.1 describe the basic idea to process word-net definitions. section 3.2 describes the definition of the feed-forward neural network definnet. finally, section 3.3 describes how we used the universal sentence embedder bert in producing embeddings for definitions. ## experiments experiments aim to investigate three issues: ## conclusions and future work building word embedding for rare out-ofvocabulary words is essential in natural language processing systems based on neural networks. in this paper, we proposed to use definitions in dictionaries to solve this problem. our results show that this can be a viable solution to retrieve word embedding for oov rare words, which work better than existing methods and baseline systems. moreover, the use of dictionary definitions in word embedding may open also another possible line of research: a different semantic probe for universal sentence embedders (uses). indeed, definitions offer a definitely interesting equivalence between sentences and words. hence, unlike existing semantic probes, this approach can unveil if uses are really changing compositionally the meaning of sentences or are just aggregating pieces of sentences in a single representation. finally, this paper promotes responsible artificial intelligence as intended in human-in-the-loop artificial intelligence (zanzotto, 2019). in fact, it gives the possibility to track how human knowledge is used by learning algorithms.