We formulate language modeling as a matrix factorization problem, and show that the expressiveness of Softmax-based models (including the majority of neural language models) is limited by a Softmax bottleneck. Given that natural language is highly context-dependent, this further implies that in practice Softmax with distributed word embeddings does not have enough capacity to model natural language. We propose a simple and effective method to address this issue, and improve the state-of-the-art perplexities on Penn Treebank and WikiText-2 to 47.69 and 40.68 respectively. 1 * Equal contribution. Ordering determined by dice rolling. 1 Code is available at httpsUnder review as a conference paper at ICLR 2018 matrices that have much larger normalized singular values and thus much higher rank than Softmax and other baselines on real-world datasets. We evaluate our proposed approach on standard language modeling benchmarks. MoS substantially improves over the current state-of-the-art results on benchmarks, by up to 3.6 points in terms of perplexity, reaching perplexities 47.69 on Penn Treebank and 40.68 on WikiText-2. We further apply MoS to a dialog dataset and show improved performance over Softmax and other baselines. Our contribution is two-fold. First, we identify the Softmax bottleneck by formulating language modeling as a matrix factorization problem. Second, we propose a simple and effective method that substantially improves over the current state-of-the-art results.
REF show that this approach does not solve the softmax bottleneck problem.
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
ArXiv
ArXiv
Computer Science