Grounded Compositional Outputs for Adaptive Language Modeling (2020)

Abstract: Language models have emerged as a central component across NLP, and a great deal of progress depends on the ability to cheaply adapt them (e.g., through finetuning) to new domains and tasks. A language model’s vocabulary —typically selected before training and permanently fixed later—affects its size and is part of what makes it resistant to such adaptation. Prior work has used compositional input embeddings based on surface forms to ameliorate this issue. In this work, we go one step beyond and propose a fully compositional output embedding layer for language models, which is further grounded in information from a structured lexicon (WordNet), namely semantically related words and free-text definitions. To our knowledge, the result is the first word-level language model with a size that does not depend on the training vocabulary. We evaluate the model on conventional language modeling as well as challenging cross-domain settings with an open vocabulary, finding that it matches or outperforms previous state-of-the-art output embedding methods and adaptation approaches. Our analysis attributes the improvements to sample efficiency: our model is more accurate for low-frequency words.

## introduction language models (lms) are at the heart of natural language processing, especially following their recent success in the pretraining paradigm @xcite @xcite . continued advances in nlp rely on the adaptability of lms to domains beyond their training data and to new domains and tasks, e.g., through domain adaptive pretraining followed by finetuning @xcite . here, we focus on an important component of lms, namely the output vocabularyover which a lm's probability distribution over the "next word" (given the history) ranges-and investigate the impact of the type of its representation on the adaptability of neural lms. today, lms are typically trained with a closed output vocabulary derived from the training data; the vocabulary is not modified when the language model is adapted or deployed. this makes large pretrained language models struggle with rare words, despite being able to produce contextualized representations for them @xcite . more importantly, this means a generative lm can never give nonzero probability to a specific word it did not see in training. this is a longstanding challenge of language modeling @xcite , but it becomes especially important when we adapt to new domains and tasks. one way to "open up" the vocabulary is to model sequences of bytes, characters, or "wordpieces" rather than the conventional word tokens @xcite @xcite . while effective, this approach requires the lm to memorize subsequences if it is to treat them as words. these models appear to require greater network depth and show slower convergence than word-based alternatives @xcite ; the extra work comes at a cost. this is one of the reasons why the area of word-level language modeling is still very active @xcite @xcite . interpolations between word-and character-or morphology-based lms represent another class of solutions @xcite @xcite . these "hybrid" approaches combine benefits from both model types. however, they introduce complexity which makes them potentially more difficult to train, maintain, and analyze. notable for enabling adaptability are interpolated lms based on copy mechanisms @xcite , dynamic evaluation @xcite , and neural caches @xcite ; the last provides state-of-the-art adaptation performance and, unlike the rest, it does not require additional training. we propose a new word-level grounded compositional output lm (groc) that applies a compositional representation to the output vocabulary (section 3). each word's output embedding is built from its surface character sequence and (if available) those of semantically related words and a free-text definition of from wordnet @xcite . this parameterization offers two chief advantages. first, groc can assign probability to words not seen during training. this means that a vocabulary different from the training vocabularye.g., one associated with a different text domain, crucial in adaptive settings-can be considered at inference time. second, because there are no word type-specific parameters, the number of model parameters in groc does not depend on the training vocabulary or its size. we evaluate groc on language modeling with both fixed and open vocabularies in english. on standard language modeling (section 4) we observe that our model has superior perplexity and is more sample efficient than a variety of existing output embedding approaches, including the recent adaptive embedding of @xcite . the open-vocabulary settings include a cross-domain setting and finetuning (section 5). we find that groc also outperforms strong interpolated baselines, including the unbounded neural cache model of @xcite on "near" domains and performs competitively on "far" domains. our analysis shows that our approach has improved sharing across words in the output vocabulary. we show experimentally that the perplexity gains are strongest for low-frequency words, implying improved sample efficiency relative to baselines: compositional output representations allow us to predict words from fewer training examples. ## preliminaries on language modeling language models assign probability to sequences of tokens; the task is usually framed as learning the conditional probability distributions over individual tokens given their histories of tokens to the left @xcite . training requires a sequence of t tokens @xmath0, . . . , x t , each x t a member of a preselected vocabulary v. we let x @xmath1, 1} |v| denote the one-hot encoding of x t . the probability of the sequence x is factored using the chain rule of probability: p(x) = t @xmath2, . . . , x t-1 ). to approximate this joint distribution, researchers have fit parametric families based on relative frequencies @xcite @xcite and neural networks @xcite . here, we focus on the latter due to their established effectiveness @xcite . tokens in this work correspond to words but they can also correspond to individual characters @xcite or byte pairs @xcite . ## groc: grounded compositional output language models we present our grounded compositional output language model (figure 2 ). foot_0 following the decomposition of neural language models in section 2 (equations 2-3), we consider each part of the model in turn: input embeddings (section 3.1), output embeddings (section 3.2), and bias (section 3.3). r n e i u 3 5 9 f 5 c d 6 d j 0 v r w c l n j u e p n m 8 f z c o m 7 @xmath3 ## conventional language modeling we first establish the performance of groc in the conventional closed-vocabulary setting, considering two datasets. we consider out-of-sample generalization (measured by test-set perplexity) and also analyze fit across the vocabulary by frequency bin. ## cross-domain language modeling to demonstrate our model's ability to generalize beyond its training data, we evaluate it across domains with an open vocabulary, in two settings: zero-resource, where it is first trained on one domain and then tested on a new target domain, and low-resource, in which the model is further exposed to training data in the new domain. ## conclusion we proposed an adaptive language model based on grounded compositional outputs. we demonstrated that it reduces the number of parameters and increases sample efficiency, outperforming strong output embedding methods and adaptation baselines on both in-domain and open-vocabulary settings respectively. in principle, our results should be applicable to word-piece language models which are currently based on lookup tables to improve their sample efficiency and compactness. in future work, it would be interesting to investigate to what extent pretrained language models benefit from groc on such zero-resource or lowresource adaptation settings. this work indicates several other future directions for language modeling in low-resource domains: extension to other languages, scaling training to even larger vocabularies, and applying groc in a large pretraining setting to expand its zero-shot generalization. we report here the computer infrastructure, experimental details, including hyperparameter bounds, hyperparameter optimal values, training speed, development scores, for both of the experiments where applicable. we also provide a comparison with state of the art by taking into account the number of model parameters and guide the reader through the replication effort we did to reproduce the neural cache by @xcite .