Improving Character-Aware Neural Language Model by Warming up Character Encoder under Skip-gram Architecture (2021)

Abstract: Character-aware neural language models can capture the relationship between words by exploiting character-level information and are particularly effective for languages with rich morphology. However, these models are usually biased towards information from surface forms. To alleviate this problem, we propose a simple and effective method to improve a character-aware neural language model by forcing a character encoder to produce word-based embeddings under Skip-gram architecture in a warm-up step without extra training data. We empirically show that the resulting character-aware neural language model achieves obvious improvements of perplexity scores on typologically diverse languages, that contain many low-frequency or unseen words.

## introduction neural language models (nlm) usually maintain a fixed vocabulary and map each word to a continuous representation. these models cannot handle new words and are not effective for languages with rich morphology. one solution is to use smaller units, such as bytes, characters, or word pieces learned from word tokens @xcite . however, this approach has to process longer sequences than word-level alternatives and may increase modeling and computational challenges @xcite . this paper focuses on another way based on word-level models in which a character encoder is used on top of characters of each word to calculate the word representation. they are often referred to as character-aware nlms (cnlms) @xcite @xcite @xcite . however, the character encoders in cnlms often show over-representation of orthography rather than semantic meaning in the re-sulting word embedding despite the fact that training word-based nlms usually helps learn such semantic meaning @xcite @xcite . for example, in cnlms, the nearest neighbors of the word 'his' with cosine similarity are 'hhs' and 'this' while 'my' is far from the nearest neighbors. to alleviate the over-representation issue in cnlms, we propose to directly force the character encoder to produce the word-based embedding in a warm-up step before the training of cnlms starts. specifically, the character encoder encodes an input word, and the encoded embedding will be forced to be close to the embeddings of its surrounding words and far from the word embeddings of negative samples. unlike the dynamically constructed embedding of the input word, the embeddings of surrounding words and negative samples are word-based, and thus these embeddings will not be biased to surface forms. the above method is similar to the architecture of the skip-gram model @xcite with the difference that we use a complex character encoder which is shown to be powerful for languages with rich morphology. in our experiments foot_0 , we choose the widely used long-short term memory (lstm) @xcite and the recent state-of-theart awd-lstm-lm @xcite for language modeling. for the character encoder, we experiment with bidirectional lstm (bilstm) over character trigrams as this variant has shown better performance than other character encoders on 10 languages @xcite . we evaluate our method in two types of datasets. one contains 14 typologically diverse languages with a large number of low-frequency words and unseen words in the test set. thus, we can test our method in a real lm setup. another one contains 5 lan-guages, where the new words in the test set have been replaced @xmath0. it is commonly used for evaluating cnlms in this field. our experiments empirically show our method can achieve obviously improved perplexity scores on a wide range of languages. finally, we analyze the learned word embedding with our character encoder on english word similarity tasks. ## related work a lot of work has tried to improve cnlms in recent years, such as analyzing performance of cnlms with different character encoders and character units @xcite , reusing subword embeddings in cnlms @xcite , injecting subword-level information into softmax @xcite , and combining word-and character-level information in cnlms @xcite @xcite . as for alleviating the over-representation issue mentioned above, @xcite used a highway network on top of their character encoder and found their highway network can encode semantic features that are not discernible from orthography alone. @xcite used syllables and morphemes in a word to construct word embeddings and showed syllable-or morpheme-based cnlms are less biased towards surface forms than a standard cnlm. however, this approach relies on extra toolkits to extract syllables or morphemes. to our knowledge, there is not much work particularly paying attention to this issue in cnlms. it is discussed only in a section in the above mentioned work, and the experiment is limited to several languages. furthermore, the analysis of character encoders is done by manually selecting several words with their nearest neighbors based on cosine similarity, while we formally verify that the character encoder captures more semantic features on 5 english word similarity tasks. our method is simple and different from the highway network used by @xcite . we do not need to change the existing architecture of cnlms. another related work to ours is word representation learning as we utilize the skip-gram architecture. one goal of this field is to learn word embeddings on large-scale corpus and use them on downstream tasks, which is different from ours. our method works without extra training data, and we do not aim at transfer learning with other train-ing data like a standard skip-gram model. ## model description the whole architecture is shown in figure 1 . we use bilstm over character trigrams as our character encoder since this variant performed best on most datasets @xcite . given a word w t , we denote its embedding as x @xmath1, where d is the embedding size. we compute the representation of w t in bilstm as follows: after warming up, we use the trained character encoder to initialize the one in our cnlm and then train our cnlm with a standard lm loss. ## conclusion in this paper, we proposed to warm up a character encoder of a character-aware neural language model under the skip-gram architecture to capture better semantic relationships between word pairs. our method is simple and effective. it was tested on a standard character-aware neural language model and a recent state-of-the-art model. the results showed that our method is effective on typologically diverse language datasets. for future work, we plan to extend our method to transformer-based language models and investigate how our model works for other tasks such as text generation.