Improving Word Embeddings through Iterative Refinement of Word- and Character-level Models (2020)

Abstract: Embedding of rare and out-of-vocabulary (OOV) words is an important open NLP problem. A popular solution is to train a character-level neural network to reproduce the embeddings from a standard word embedding model. The trained network is then used to assign vectors to any input string, including OOV and rare words. We enhance this approach and introduce an algorithm that iteratively refines and improves both word- and character-level models. We demonstrate that our method outperforms the existing algorithms on 5 word similarity data sets, and that it can be successfully applied to job title normalization, an important problem in the e-recruitment domain that suffers from the OOV problem.

## introduction inability to represent rare and unseen words, which is referred to as the out-of-vocabulary (oov) problem, limits usefulness of the standard embedding approaches to real-world applications. for example, in the e-recruitment domain there is an extremely large number of ways one can write a job title for the same job type. linkedin users who are software engineers may describe themselves as software developer, python guru, sw engi., sw developper or sw and web application developer, which are caused by different variants, abbreviations, misspellings, or compoundings. in this case, standard embedding for a finite set of job titles is not helpful when recommending jobs to users with rare or unseen job titles. an embedding model that can effectively handle the oov problem should preserve both semantic and syntactic characteristics of words and phrases. recently proposed mimicking approach is a promising solution to this problem, with two representative algorithms being mimick @xcite and gwr @xcite . their main idea is to train a character-level embedding neural network (nn) that can reconstruct, or mimic, an embedding from word-level embedding model. the trained character-level model is then used to generate both semantically and syntactically relevant embeddings for arbitrary character sequences. while mimick and gwr used bilstm and cnn character-level models, respectively, similar approaches were recently proposed with several other architectures @xcite . in order to provide high-quality embeddings useful for downstream applications, it is crucial for the model to preserve semantic characteristics of words and phrases, while also being robust to syntactic and word-form changes. however, this may not always be the case for the vanilla mimicking method. to see why, let us first denote the word embedding model as w and the character embedding model as g in the mimicking mechanism. let us take a peek at model w produced by training on career profile corpus, described in detail in section 5. the top 5 neighbors for word java developer in the vector space induced by w are software developer, software engineer, web developer, sr java developer, and programmer. after fitting g to mimic embeddings from w , the top 5 neighbors of java developer in the vector space induced by g become java developeur, javadeveloper, javva developer, javapython developer, javas developpper, many of which are rare phrases that were not placed in the neighborhood by w . this is helpful when dealing with the oov and rare words such as javas developpper, but might negatively affect fully-connected layer (d-dim) c o d e r bilstm fully-connected layer (d-dim) c o d e r cnn highway layer (a) mimick (b) gwr figure 2: different architectures for character-level models g the quality of the embedding produced by g as the word-form similarities become too influential in the training of the model, resulting in g failing to preserve semantic similarities of words with the same meaning but different spelling. in the above example, software engineer and web developer no longer have similar embeddings with java developer, even though they refer to similar occupations. ideally, model g should retain ability to map semantically related words into the same neighborhood, something that w does well. in addition, g should also be able to differentiate between words with similar spelling that are disparate semantically (e.g., java cafe expert should not be in the same neighborhood with java developer or java expert). in order to better capture the syntactic and semantic similarities between words, we developed an iterative mimicking framework that strikes a good balance between word-level and character-level representations of words. in the proposed 2-mode framework, w training is re-initialized by g embeddings, then g is fine-tuned by mimicking w embeddings, and the process is repeated several times. as we demonstrate by experiments, the resulting procedure produces improved character-level embeddings on both common and rare and oov words. we refer to the resulting approach as iterative mimicking (im). our contributions are summarized below: in this section, we discuss existing approaches capable of generating representations for oov words. aggregation of n-grams or morphemes. fasttext @xcite algorithm directly learns embeddings of n-grams with a word2vec objective (cbow or skipgram). then, embedding for an oov word is simply aggregated as the average of the word's n-gram embeddings. in charagram @xcite , a nonlinear aggregation function is used instead of averaging n-gram embeddings. aggregation over morphemes is discussed in @xcite . character-aware modules in deep nns. stacking deep nns on top of character-level modules is a popular recent trend. for example, character bilstms or cnns are co-trained with a language model in elmo @xcite and charcnn @xcite . similar to mimicking, the character-aware modules in those methods are able to assign vectors to oov words. however, they are computationally expensive due to a need to train a heavyweight language model on a large corpus to be effective. there are also approaches that co-train the character-aware modules with auxiliary supervised tasks, such as part-of-speech tagging (santos and zadrozny, 2014), text classification @xcite and machine translation @xcite , but the resulting embeddings are not necessarily reusable for other downstream tasks. mimicking. this approach allows learning lightweight character embedding models in a two-step manner @xcite @xcite . it was shown that it yields superior performance to other baselines on multiple intrinsic and extrinsic tasks. our proposed method is generalizing this line of research, as discussed in the following section. ## methodology we first introduce the original mimicking mechanism used in mimick @xcite and gwr @xcite . we then describe our proposed iterative mimicking (im) mechanism, which is a generalization of the original approaches. ## word similarity we first perform intrinsic evaluation of embedding models on word similarity tasks. given word pairs and the word similarity judgements by human labelers, the quality of embedding models is measured as the correlation between the human judgment and the cosine similarity of word embeddings. ## job title normalization we dedicate the rest of the paper to the task of job title normalization, an important problem in online recruiting industry worth astounding $400b @xcite . the unstructured nature of job titles and occupations makes job search on these online platforms difficult, as potential matches can be missed due to large variability. for example, although sw eng. and programmer refer to the same occupation, nlp models may fail to automatically match these two very different strings. this problem is further exacerbated for rare titles, which are even more difficult to model due to their low counts. as seen in figure 4 , titles observed less than 10 times in our data set cover as much as 70% of the total corpus, making handling of rare titles a problem of very significant importance in the e-recruiting domain. to the best of our knowledge, this is the first work to utilize mimicking-based approaches to address the problem of job title normalization. we formally describe the problem of job title normalization as follows. we are given a data set @xmath0, d i )|@xmath1, ..., n }, where t i is job title and d i is job description. both are free-form text entered by a user of a professional network such as linkedin or indeed. let us denote the vocabulary of job titles as t , and the vocabulary of words in job descriptions as v. both t and v are infinite sets. let us also suppose there is a job title taxonomy @xmath2, . . . , o m }, which is a finite set denoting m distinct job categories. an example of such a taxonomy is the standard occupational classification (soc), whose o*net-soc 2010 release contains 1,100 job titles @xcite . then, the task is to match (or normalize) job title @xmath3. we note that this definition of job normalization corresponds to entity linking @xcite . in our experiments we will evaluate results according to this definition. we also note that an alternative information retrieval definition of job normalization is to find job titles that are most similar to the query job title. we do not show experiments according to this definition. to classify a job title into a category, we first learn how to embed an input string representing a job title into a vector space using the character-level model g. using the same model we can map all job categories into the same vector space by assigning each category a vector. 2 then, given a particular job title, we match it to the closest job category in the vector space. due to the large variety of ways one can write a job title, models that are able to generate high-quality embeddings for rare or oov titles are preferable. we note that this approach does not require manually labeled training data, unlike the method proposed in @xcite . all that is required is a corpus of (job titles, job description) pairs to train the embedding models. ## conclusion in this paper we introduced a lightweight framework that enhances the existing mimicking models by iteratively retrofitting a word-level and a character-level embedding neural networks. we showed that the proposed algorithm can be successfully applied to the nlp task of word similarity, as well as the task of job title normalization, a challenging and very important problem in the e-recruitment domain.