Language Model Metrics and P rocrustes Analysis for Improved Vector Transformation of NLP Embeddings (2020)

Abstract: Artificial Neural networks are mathematical models at their core. This truism presents some fundamental difficulty when networks are tasked with Natural Language Processing. A key problem lies in measuring the similarity or distance among vectors in NLP embedding space, since the mathematical concept of distance does not always agree with the linguistic concept. We suggest that the best way to measure linguistic distance among vectors is by employing the Language Model (LM) that created them. We introduce Language Model Distance (LMD) for measuring accuracy of vector transformations based on the Distributional Hypothesis ( LMD Accuracy ). We show the efficacy of this metric by applying it to a simple neural network learning the Procrustes algorithm for bilingual word mapping.

## introduction the distributional hypothesis @xcite inspired the development of embeddings that capture the meaning of language based on how words co-occur with each other @xcite . natural language processing relies heavily on these high dimensional vectors to represent words, phrases, sentences or documents, in a form that can be processed by deep neural networks which were originally designed for tasks related to computer vision. input embeddings are transformed by network layers into output vectors which represent solutions to many nlp tasks @xcite . in order to learn these transformations, a network must be able to calculate the difference between predicted vectors and actual word vectors. this distance calculation is a crucial part of measuring loss, and performing back-propagation. these core functions of neural networks have primarily relied on mathematical processes without regard to linguistic principles. we demonstrate that nlp embedding transformation is better measured using linguistic similarity functions rooted in knowledge of languages rather than concepts such as euclidean or angular distance, which assumes vectors to be "physical" objects. ## language models and data as in raw data, nlp vectors dimensions typically share values that are treated similarly and are thus undifferentiated in a sense. this seems to contradict the assertion that each nlp embedding dimen-sion has a specific unique meaning like feature data. instead, the meaning of a dimension is more like probability, representing how often a word is used with a particular meaning, rather than the actual meaning of the word. vectors with dimensions that differ in meaning, as in nlp embeddings, cannot be used with typical spatial measurements such as l p norm and cosine distance. we contend that nlp vector distance can best be measured by the language models which represent the vectors. therefore, we seek to replace mathematic calculations with predictions from language models. we simply rely on the language model itself to provide a distance measurement for our custom metric. in this research, we use the word2vec model @xcite to produce a custom bilingual word mapping dataset. this dataset, combined with the gensim model of keyed vectors @xcite , provides a distributional distance measurement based on word movers distance @xcite . our neural network is a simple multilayer perceptron (mlp) which accepts spanish word vectors as input and predicts english word vectors. this simple model was chosen because it is analogous to any layer found in innumerable, more complex, neural networks. showing improved efficacy in this model should demonstrate improvement in any nlp task. ## language model distance an exact measurement of equality is not possible for high-dimensional nlp embeddings. embeddings of several hundred dimensions, and one-hot encoded vectors on the order of tens of thousands of dimensions, are particularly difficult to measure. lm d(p, t, m, k) = t rue, if @xmath0.set(p, k) f alse, otherwise instead, we suggest that the true measure of nlp vector distance is best provided by the model which defines the vectors. we present a family of metrics, language model distance (lmd), which calculates distance and equality among nlp vectors by using the language model itself. lm d is defined as in equation 1 where the distance between predicted vector p and known truth vector t, is provided by model m, given neighbor threshold k. the distance measure is binary because it is based on set inclusion, and not physical or euclidean distance. thus, lm d can be used as a measure of accuracy, and records a true positive when t is within the neighborhood of the predicted vector (@xmath1.set(p, k)). ## learning orthogonal procrustes analysis the orthogonal procrustes algorithm is a process for finding the optimal mapping of one set of vectors to another. typically, the vectors represent points in 2 or 3 dimensional space, for image processing, or they represent qualitative data measured in few dimensions @xcite . after resizing and repositioning of vectors, an optimal rotation matrix r is produced by a method similar to singular value decomposition. this classic approach to vector transformation has been explored as a solution for some nlp tasks @xcite . therefore we ask: can a neural network be trained to perform the same optimal transformation for nlp embeddings which occupy a much higher dimensional space? our task is to train a simple mlp to learn the optimal mapping r, between two disparate vector spaces representing a bilingual dataset. we measure the success of this task using lm d as the basis for accuracy as in figure 2 and equation 1. we create two separate language models from a parallel corpus of european parliament translations, the so called europarl dataset @xcite . we use the word2vec model in continuous bagof-words (cbow) mode @xcite to build two separate distributions. by using a bilingual corpus, and training language models sep-arately, we ensure that the models share a common domain, but the vector spaces remain separate. for training, we then map word vectors from one distribution to the other, using a set of 1000 most common words pairs, obtained from from a language learning website foot_0 . ## learning general procrustes analysis to further test, we try to learn general procrustes analysis; a much harder task because it requires the network to generalize. we have just shown that a simple neural network can learn to transform vectors. this is noncontroversial since all neural networks perform this task at every layer. however, not all networks are able to generalize. using the same network configuration as before, we now evaluate embeddings that we have not seen in training, as is common. this is equivalent to learning the generalized procrustes algorithm. results in figure 4 show that lmd accuracy is more like cosine distance when generalization is required. note that we use lmd accuracy only for metrics. this model uses cosine similarity for error calculation and back-propagation. we conclude that such l p norm measurements can only drive generalization as far as they are able to measure accuracy. the local variation in lmd accuracy, evident in figure 4 , may be significant as it may make determining the derivative of the function difficult. the derivative of lmd accuracy must be worked out before it can be incorporated into a loss function and be used in back-propagation. the overall shape of the curve, despite irregularities is encouraging as the slope may be computed using ordinary least squares in a calculation of rolling regression, or by other numerical methods. ## conclusion we suggest that language model metrics described here may be incorporated directly into activation and loss functions, and may be used as an error measurement for back-propagation. we suggest this basic enhancement would improve the generalized procrustes algorithm and other nlp processing in general. this is left for future work.