Learning Embeddings for Rare Words Leveraging I nternet Search Engine and Spatial Location Relationships (2021)

Abstract: Word embedding techniques depend heavily on the frequencies of words in the corpus, and are negatively impacted by failures in providing reliable representations for low-frequency words or unseen words during training. To address this problem, we propose an algorithm to learn embeddings for rare words based on an Internet search engine and the spatial location relationships. Our algorithm proceeds in two steps. We firstly retrieve webpages corresponding to the rare word through the search engine and parse the returned results to extract a set of most related words. We average the vectors of the related words as the initial vector of the rare word. Then, the location of the rare word in the vector space is iteratively fine-tuned according to the order of its relevances to the related words. Compared to other approaches, our algorithm can learn more accurate representations for a wider range of vocabulary. We evaluate our learned rare-word embeddings on the word relatedness task, and the experimental results show that our algorithm achieves state-of-the-art performance.

## introduction since @xcite introduced the idea of learning continuous vectors for words using network-based language models, many word embedding techniques have been proposed such as word2vec @xcite , glove @xcite , etc. however, nearly all existing word embedding approaches need words that have a high frequency in the corpus and cannot learn good representations for rare words (including low-frequency words and unseen words). as words in a corpus follow a zipfian distribution, only a small proportion of the total tokens are frequent words, while most of them are rare words. therefore, how to learn qualified embeddings for rare words is an essential issue to be solved. from the human perspective, when encountering a new word, it is an instinct to take a look at its structure or to look up its definition in a dictionary. the essence of both behaviours is to transform a rare word to a set of familiar words expressing the same meaning to it. based on the above ideas, some proposed techniques have attempted to exploit subword information or lexical resources to predict the rare word representation. in the area of subword-based approaches, fast-text @xcite learns representations for character n-grams and represents words as the sum of the n-gram vectors. ngram2vec @xcite learns n-gram representations from n-gram co-occurrence statistics and incorporates this information into the word representations. @xcite proposed the mimick model to predict vectors for out-of-vocabulary words by learning a function from spellings to distributional embeddings. the attentive mimicking model (am) @xcite and the form context model (fcm) @xcite jointly use surface form and context information to improve representations of rare words. in another way, lexical resources are used to infer the representation for a rare word from the vectors of the words having a semantic association with it. semland @xcite infers the representations for rare words by exploiting the definitions and relationships in an external lexicon wordnet @xcite . @xcite proposed to use word relation knowledge found in semantic lexicons to retrofit word vectors. @xcite proposed to train a long short-term memory (lstm) network to predict the representations of rare words based on auxiliary data (e.g., a dictionary definition) from knowledge bases. @xcite embedded a knowledge base into a vector space by the node2vec @xcite graph embedding algorithm and then mapped the embedded words from this space to a corpus-based space. however, the performance of these approaches heavily depends on the coverage of external data sources. if a rare word is uncovered by the lexicon, the rare-word embedding will not be available. in addition to the approaches outlined above, a great concern has been raised over the pretrained language models for their outstanding performance in various natural language processing (nlp) tasks. among the pre-trained models, elmo @xcite and bert @xcite are two most typical ones. based on the pre-trained language model, we can use a function of the internal layers as the vector of a word. the pre-trained language models have strong coverage ability and can predict vectors for nearly all rare words. however, based on our experimental results (section 3.4), we found the semantics of rare words are not learned well when the context information is not provided or the rare words come from specific domains. as the largest source of information in the world, the internet consists of billions of pages of data in all fields. to figure out the meaning of a rare word, almost everyone's first priority has changed to retrieve it on the internet and extract the useful information from the associated webpages. inspired by this, we propose a two-step algorithm to learn rare-word embeddings using the internet search engine and the spatial location relationships. we firstly find the top-n most relevant words to a rare word from the webpages returned by the internet search engine and compute the initial embedding of the rare word by averaging the vectors of these extracted words. according to the order of the top-n most relevant words, we further iteratively fine-tune the location of the rare word in the vector space to make it satisfy the constraints of spatial location relationships. the constraints are that if a rare word is more relevant to a word than other words, the distance between the rare word and this word is closer than the distances between the rare word to others in the vector space. compared to the existing approaches, there are three advantages of our approach: (i) we can obtain a powerful coverage for rare words; (ii) we can provide more accurate vector representations for rare words; (iii) we can support representing multilingual rare words. this paper is organized as follows: section 2 describes our methodology in detail. section 3 presents the experimental results. the paper is concluded in section 4. ## methodology in this section, we will begin by introducing our motivation, then describe how we define the relevance metric and obtain the top-n related words to a rare word using the internet search engine, and finally present the fine-tuning process toward achieving the more precise embedding learning. ## experiments in this section, we present our experimental settings and results. we take the word relatedness task as the evaluation framework, and the spearman correlation coefficient (ρ × 100) is adopted to assess the quality of the learned embeddings. also, the percentage of missed pairs (pmp) is used to evaluate the vocabulary coverage of our model. baidu foot_0 search engine is used to retrieve the relevant webpages in our coarse-tuning process. all experiments use the same fine-tuning settings: @xmath0, ∆ = 0.1. we first report the performance of our algorithm in different hyper-parameters. then, we compare the quality of the rare-word embeddings before and after the fine-tuning process to verify the effectiveness of our two-step approach. next, we compare our algorithm with the cbow algorithm and six state-of-the-art english rare-word embedding learning algorithms. finally, we evaluate our algorithm on two chinese word datasets to investigate the scalability of our approach for a language other than english.