Stolen Probability: A Structural Weakness of Neural Language Models (2020)

Abstract: Neural Network Language Models (NNLMs) generate probability distributions by applying a softmax function to a distance metric formed by taking the dot product of a prediction vector with all word vectors in a high-dimensional embedding space. The dot-product distance metric forms part of the inductive bias of NNLMs. Although NNLMs optimize well with this inductive bias, we show that this results in a sub-optimal ordering of the embedding space that structurally impoverishes some words at the expense of others when assigning probability. We present numerical, theoretical and empirical analyses which show that words on the interior of the convex hull in the embedding space have their probability bounded by the probabilities of the words on the hull.

## introduction neural network language models (nnlms) have evolved rapidly over the years from simple feed forward nets @xcite to include recurrent connections @xcite and lstm cells @xcite , and most recently transformer architectures @xcite . this has enabled ever-increasing performance on benchmark data sets. however, one thing has remained relatively constant: the softmax of a dot product as the output layer. nnlms generate probability distributions by applying a softmax function to a distance metric formed by taking the dot product of a prediction vector with all word vectors in a high-dimensional embedding space. we show that the dot product distance metric introduces a limitation that bounds the expressiveness of nnlms, enabling some words to "steal" probability from other words simply due to their relative placement in the embedding space. we call this limitation the stolen probability effect. while the net impact of this limitation is small in terms of the perplexity measure on which nnlms are evaluated, we show that the limitation results in significant errors in certain cases. as an example, consider a high probability word sequence like "the united states of america" that ends with a relatively infrequent word such as "america". infrequent words are often associated with smaller embedding norms, and may end up inside the convex hull of the embedding space. as we show, in such a case it is impossible for the nnlm to assign a high probability to the infrequent word that completes the high-probability sequence. numerical, theoretical and empirical analyses are presented to establish that the stolen probability effect exists. experiments with n-gram models, which lack this limitation, are performed to quantify the impact of the effect. ## background in a nnlm, words w i are represented as vectors x i in a high-dimensional embedding space. some combination of these vectors x @xmath0, which are fed into a a neural unit as features to generate a prediction vector h t . nnlms generate a probability distribution over a vocabulary of words w i to predict the next word in a sequence w t using a model of the form: we refer to this calculation of logits and transformation into a probability distribution as the dotproduct softmax. ## problem definition nnlms learn very different embeddings for different words. in this section we show that this can make it impossible for words with certain embeddings to ever be assigned high probability in any context. we start with a brief examination of the link between embedding norm and probability, which motivates our analysis of the stolen probability effect in terms of a word's position in the embedding space relative to the convex hull of the embedding space. ## experiments in this section we provide empirical evidence showing that words interior to the convex hull are probability-impoverished due to the stolen probability effect and analyze the impact of this phenomenon on different models. ## related work other work has explored alternative softmax configurations, including a mixture of softmaxes, adaptive softmax and a taylor series softmax @xcite @xcite . there is also a body of work that analyzes the properties of embedding spaces @xcite . we do not seek to modify the softmax. instead we present an analysis of how the structural bounds of an nnlm limit its expressiveness. ## conclusion we present numerical, theoretical and empirical analyses showing that the dot-product softmax limits a nnlm's expressiveness for words on the interior of a convex hull of the embedding space. this is structural weakness of nnlms with dot-product softmax output layers, which we call the stolen probability effect. our experiments show that the effect is relatively common in smaller neural language models. alternative architectures that can overcome the stolen probability effect are an item of future work.