On Randomized Classification Layers and Their Implications in Natural Language Generation (2021)

Abstract: In natural language generation tasks, a neural language model is used for generating a sequence of words forming a sentence. The topmost weight matrix of the language model, known as the classification layer, can be viewed as a set of vectors, each representing a target word from the target dictionary. The target word vectors, along with the rest of the model parameters, are learned and updated during training. In this paper, we analyze the properties encoded in the target vectors and question the necessity of learning these vectors. We suggest to randomly draw the target vectors and set them as fixed so that no weights updates are being made during training. We show that by excluding the vectors from the optimization, the number of parameters drastically decreases with a marginal effect on the performance. We demonstrate the effectiveness of our method in image-captioning and machine-translation.

## introduction deep neural networks enabled breakthroughs in natural language generation tasks such as machinetranslation @xcite , image captioning @xcite , and more. generating the text is done by employing a conditional language model as the decoder component, responsible for predicting the next word at each step during decoding, as depicted in fig1 . for predicting the next word, the language model first encodes into a vector @xmath0, denoted as context representation, both the previously predicted words, and the task's related input (such as source sentence in machinetranslation or input image in image-captioning). then, at the classification layer, the context representation is projected onto a set of weight vectors, resulting in a vector termed as logits vector. afterward, a softmax function is applied to output a distribution over the target vocabulary words. the set of weight vectors used for producing the logits, denoted here as matrix @xmath1, where v is the target vocabulary. matrix w can be viewed as |v | vectors, where each is a d-dimensional vector representing a specific word from the target vocabulary, we term these vectors through the paper as target word embeddings and target vectors, interchangeably. the target embeddings, along with the rest of the model parameters, are estimated so as to minimize the loss function. during training, an additional set of |v | vectors are being learned to represent the previously predicted words when given to the decoding step. these vectors are referred as input word embeddings. learning the input embeddings during training was shown to improve the performance of nlp classifers and allows achieving a level of generalization that is not possible with classical n-gram language models @xcite . their main advantages are capturing the relationship between words and allowing similar words to have embedding vectors close in space. while a large amount of work has done to understand the properties and demonstrate the effectiveness of learning the input word embeddings, the benefits of learning the target word embeddings remain unexplored. in this paper, we show that by randomly drawing the target word embeddings and excluding them from training, the number of trained parameters are drastically decreased with a marginal effect on the performance. by this, we show that the properties captured in the target word embeddings, such as word frequencies and relationships, are surprisingly redundant and may be ignored in low resource environments. ## background, notations and definitions consider a nlg task, such as machine-translation or image-captioning, where for a given input instance x i , a corresponding sentence s i should be generated. typically, an encoder-decoder based neural network is trained for solving the task. the encoder is responsible for encoding the input instance into a vector, while the decoder responsible for generating the next word given the input vector and previously generated words. commonly, an attention mechanism is incorporated during the decoding @xcite . denote by d @xmath2, s i )} n @xmath3, s i is the corresponding sentence, and n is the number of training examples. training the model is done by maximizing the following objective function: for generating the next word, a context vector f θ ∈ r d is learned and is responsible for encoding the given input along with the previously predicted words. then, the context vector is projected onto each of the target word vectors w @xmath4, ..., |v |, by calculating the dot-product between the vectors. afterward, a bias term @xmath5, and a softmax function is applied, resulting in a distribution over the target words. formally: notice that both the angles and magnitudes of the target word vectors are influencing the predicted probability in eq-1. the cosine of the angle between w j and f θ measures how well the word s j fits into the context. hence, interchangeable words have their corresponding target vectors directed at the same angle. the magnitudes of the target vectors control on the predicted probability, in a way that they have a stronger effect on words whose embeddings have direction similar to f θ , and less effect or even a negative effect on words in other directions. consider a case where the cosine of the angle between w j and f θ is close to 1, meaning that the angle between them is close to 0. in this case, increasing the magnitude ||w j || would result in an increased probability for the word s j . however, when w j is directed in an opposite direction to f , the cosine of the angle between them would be close to -1, and therefore, increasing the magnitude would result in a lower probability. by fixing the target vectors and the bias term, the model can maximize the probability in eq-1 only by optimizing the vector f θ . in recent work, press and wolf (2016) proposed tying the target and input embeddings by using the same vectors to represent both. the paper showed that the performance of weight tied models are on par with learning two separate vectors in machinetranslation. however, the method forces the target vectors to have the same dimension as the input embeddings and adds additional computational costs. more recently, several works @xcite @xcite explored the effects of fixing the classification layer in image classification models and demonstrated that the accuracy, number of parameters and out-ofdistribution detection ability improve. in this paper, we empirically show that randomly drawn, fixed target word embeddings allow models to achieve high performance in natural language generation tasks. from an efficiency perspective, fixing these vectors decrease the number of parameters in the classification layer by d • |v |. since the target vocabulary typically contains thousands of words, and the dimension of the context vector f θ is in hundreds or thousands, the reduction in parameters is significant. ## experiments in this section, we present our experimental results. we evaluate our approach on image-captioning and machine-translation tasks. we start by describing the experimental setup; then, we present the results and analyze the target embeddings.