Improving Low Compute Language Modeling with In-Domain Embedding Initialisation (2020)

Abstract: Many NLP applications, such as biomedical data and technical support, have 10-100 million tokens of in-domain data and limited computational resources for learning from it. How should we train a language model in this scenario? Most language modeling research considers either a small dataset with a closed vocabulary (like the standard 1 million token Penn Treebank), or the whole web with byte-pair encoding. We show that for our target setting in English, initialising and freezing input embeddings using in-domain data can improve language model performance by providing a useful representation of rare words, and this pattern holds across several different domains. In the process, we show that the standard convention of tying input and output embeddings does not improve perplexity when initializing with embeddings trained on in-domain data.

## introduction language modeling is an essential part of many nlp applications, including predictive keyboards, speech recognition, and translation. recent work has focused on (1) small constrained datasets, such as the penn treebank @xcite and wikitext-103 @xcite , and (2) vast resources with billions of words used to train enormous models with significant computational requirements @xcite . this leaves a gap: when a substantial amount of in-domain data is available, but computational power is limited. we explore how initialising word embeddings using in-domain data can improve language modeling in english. testing all valid configurations of weight tying, embedding freezing, and initialisation, we find that the standard configuration is not optimal when rare words are present. instead, the best approach is to initialise with in-domain data, untie the input and output, and freeze the input. to understand this difference, we run a series of experiments to measure the impact of changing (a) the threshold for replacing rare words with a special symbol; (b) the source of data for initialisation; (c) the amount of training data for the language model; and (d) the hyperparameters for both the baseline and our proposed approach. we find that the improvement comes from improved representation of rare words. these findings are confirmed through experiments on four additional domains, with similar trends. we also compare our approach to an n-gram language model and a large-scale transformer model. we find that if a large-scale transformer is inappropriate either for computational or modeling reasons, it is best to train an lstm-based language model with as much data as possible and initialise the embeddings on all available in-domain data. ## proposed approach we propose initialising the language model's word embeddings with vectors trained on additional indomain data. to make this most effective, we make two other key changes to training. first, we prevent embeddings from shifting during training. without this, the embedding space could become inconsistent as vectors for words seen in training shift while those for words seen only in the additional data stay the same. second, we do not tie the weights of the input embeddings and final output layer. to understand the impact of these factors, we train models with every valid combination of weight tying, freezing, and pretraining. 1 we experiment with @xcite 's awd-lstm -a high-performing model that can be trained in under a day on a single gpu (without fine-tuning). we train embeddings using glove on gigaword. 2 for evaluation, we consider two versions of the penn treebank. std is the standard version used in language modeling, with words of frequency less than five converted to unk, all words lowercase, numbers replaced with a special symbol, and punctuation removed. rare has the same pre-processing but without replacement of rare words. 3 there are also four clear sections of performance in the table: freezing the output embeddings consistently leads to poor performance, even with pretrained embeddings pretrained. in contrast, freezing with pretrained input embeddings leads to some of the best results. we expected freezing with random initialisation to perform poorly, but the drop is modest for input freezing and dramatic for output freezing. this suggests that the two embedding matrices are serving different purposes in the model. the results do support the practise of tying when the input embeddings are random, but the benefit is half as large when they are pretrained. for the dataset with rare words we see mostly the same trends. the exception is the bottom six rows. once rare words are present, random initialisation of the input embeddings is considerably worse than pretraining (third last row). again, there is an asymmetry between input and output, with the top five models all using pretrained input embeddings, but only three of them using pretrained output embeddings. tying is also no longer the best approach, with the top three models not tying. our proposed approach, using pretrained untied embeddings and freezing the input, has the best results. the only difference between std and rare is 3 the script to generate our rare data from the ldc release is available at: http://jkk.name/emnlp20lm/ . the lack of unks in rare. this impacts 5.1% of tokens in the validation set (33% of types). while our pretrained embeddings do not cover all of these rare words, they do cover most. the vocabulary from gigaword that we build vectors for covers 99.5% of the validation word tokens in std (98% of word types), and 98.8% of the validation word tokens in rare (84% of word types). ## when & why does pretraining help? to understand the strengths and limitations of this new approach, we consider a series of experiments, each probing a specific variable. to simulate our target scenario, we use 44 million words of wall street journal data from the north american news corpus (nanc, @xcite . this provides enough data for pretraining, training, validation, and test sets all in the exact same domain (not even varying the newspaper). we apply similar preprocessing as in the previous section, but break the data down into articles rather than sentences and keep rare words. we compare the six best configurations from standard approach. our approach, but with random output embeddings and without freezing. standard approach + pretraining. our approach, but without freezing. our approach. our approach, but with random output embeddings. other domains show the same pattern. first we consider varying the domain to make sure this is not an artifact of news data. ## conclusion initialising embeddings with vectors trained on indomain data can improve performance by providing better representations for rare words. this effect persists even as more in-domain data is used to train the language model. our work also suggests that standard model components like embedding tying should be retested as we continue to explore the space of language modeling.