  Recent progress in language modeling has been driven not only by advances in
neural architectures, but also through hardware and optimization improvements.
In this paper, we revisit the neural probabilistic language model (NPLM)
of~\citet{Bengio2003ANP}, which simply concatenates word embeddings within a
fixed window and passes the result through a feed-forward network to predict
the next word. When scaled up to modern hardware, this model (despite its many
limitations) performs much better than expected on word-level language model
benchmarks. Our analysis reveals that the NPLM achieves lower perplexity than a
baseline Transformer with short input contexts but struggles to handle
long-term dependencies. Inspired by this result, we modify the Transformer by
replacing its first self-attention layer with the NPLM's local concatenation
layer, which results in small but consistent perplexity decreases across three
word-level language modeling datasets.
