Revisiting Simple Neural Probabilistic Language Models (2021)

Abstract: Recent progress in language modeling has been driven not only by advances in neural architectures, but also through hardware and optimization improvements. In this paper, we revisit the neural probabilistic language model (NPLM) of Bengio et al. (2003), which simply concatenates word embeddings within a fixed window and passes the result through a feed-forward network to predict the next word. When scaled up to modern hardware, this model (despite its many limitations) performs much better than expected on word-level language model benchmarks. Our analysis reveals that the NPLM achieves lower perplexity than a baseline Transformer with short input contexts but struggles to handle long-term dependencies. Inspired by this result, we modify the Transformer by replacing its first self-attention layer with the NPLM’s local concatenation layer, which results in small but consistent perplexity decreases across three word-level language modeling datasets.

## introduction over the past decade, state-of-the-art neural architectures for language modeling (lm) have transitioned from simple recurrent neural networks @xcite to lstms @xcite and finally to transformers @xcite . this progress is not due solely to lmspecific advances, however, as general-purpose upgrades such as residual connections @xcite and layer normalization @xcite have enabled scaling to huge datasets and model sizes @xcite on powerful gpus. in this paper, we revisit the neural probabilistic language model (nplm) of @xcite , the first (and simplest) neural architecture proposed for language modeling, through the lens of modern architecture design, hardware, and optimization. given an input sequence of tokens, the nplm first concatenates the previous n token embeddings and @xcite , which concatenates token embeddings within a fixed local window and feeds them to a stack of feed-forward layers to predict the next token. our modified version additionally concatenates representations of the distant context, which are computed by applying a weighted average to token representations outside the local window. to what extent are its limitations mitigated by modern design and optimization choices? to answer this question, we design an upgraded nplm featuring increased depth and window size n that incorporates residual connections, layer normalization, and dropout. we also include global context representations to the concatenation layer by applying simple aggregation functions to embeddings outside of the local context window. these modifications substantially improve the nplm: on the wikitext-103 benchmark dataset, the original nplm of @xcite reaches a validation perplexity of 216, compared to 31.7 for our implementation, and 25.0 for a transformer baseline. can we improve transformer language models by hybridizing them with nplms? interestingly, we discover that our nplm actually outperforms the transformer when given shorter input contexts (figure 2 ), although it is unable to take full advantage of longer contexts. inspired by this result, we create two simple variants of the transformer, one in which the first self-attention layer is replaced with the nplm's concatenation layer, and the other in which self-attention in the first layer is constrained to a small local window. foot_0 these adjustments result in small but consistent perplexity decreases compared to a baseline transformer across three word-level language modeling datasets (the first variant obtains 24.1 validation perplexity on wikitext-103). our qualitative analysis shows that the modified transformers are better at predicting rare tokens and named entities, especially those that have already appeared in the context. ## neural probabilistic language models modern neural language models (nlms) compute the conditional probability of a token w t given preceding (or prefix) tokens @xmath0. more concretely, a composition function g is applied to the sequence of token embeddings @xmath1, which results in a dense vector @xmath2. a softmax classifier then takes z as input and produces a distribution p (w t | @xmath3. transformers @xcite are currently the most popular choice for the composition function g. nplm definition: first introduced by @xcite , the nplm uses a simple composition function reminiscent of n-gram language modeling. it concatenates the last k prefix embeddings and passes the result through a feed-forward layer: the nplm has many intuitive limitations: (1) it ignores the global context provided by prefix tokens further than k tokens away; (2) it uses a different set of parameters for each position in the prefix window; and (3) it has a relatively small number of parameters, which limits its expressivity. ## using nplms to improve transformers while our upgraded nplm achieves a massive perplexity reduction compared to the original implementation, it is still ∼ 6 perplexity points short of the baseline transformer lm. are there any takeaways from our results that can be used to improve transformer lms? in this section, we begin with an analysis experiment on wikitext-103 that shows nplms outperform transformers when given shorter prefixes. inspired by this result, we propose two variants of a transformer lm that integrate elements of the nplm, and discover that both of them decrease perplexity across three word-level language modeling datasets ( ## related work the nplm model in this paper based entirely on the original formulation from @xcite . the variants in our analysis are based on the transformer model @xcite and transformer lms @xcite @xcite @xcite @xcite @xcite . the constrained local attention in transformer-c is adopted at all layers of models such as longformer @xcite and big bird @xcite due to its sparsity. our work conceptually resembles that of @xcite , who modernize hmm language models, as well as simple rnn-based language models @xcite . our linguistic analysis is inspired by experiments from @xcite . ## conclusion we discover that general-purpose advances in neural architecture design, hardware, and optimization significantly improve the nplm, a classic language model. an analysis of our upgraded nplm inspires us to hybridize it with a modern transformer lm and obtain perplexity decreases across three word-level lm datasets.