Locality Preserving Loss: Neighbors that Live together, Align together (2021)

Abstract: We present a locality preserving loss (LPL) that improves the alignment between vector space embeddings while separating uncorrelated representations. Given two pretrained embedding manifolds, LPL optimizes a model to project an embedding and maintain its local neighborhood while aligning one manifold to another. This reduces the overall size of the dataset required to align the two in tasks such as crosslingual word alignment. We show that the LPL-based alignment between input vector spaces acts as a regularizer, leading to better and consistent accuracy than the baseline, especially when the size of the training set is small. We demonstrate the effectiveness of LPL-optimized alignment on semantic text similarity (STS), natural language inference (SNLI), multi-genre language inference (MNLI) and cross-lingual word alignment (CLA) showing consistent improvements, finding up to 16% improvement over our baseline in lower resource settings.

## introduction over the last few years, vector space representations of words and sentences, extracted from encoders trained on a large text corpus, are primary components to model any natural language processing (nlp) task, especially while using neural or deep learning methods. neural nlp models can be initialized with pretrained word embeddings learned using word2vec @xcite or glove @xcite and fine-tuned sentence encoders like bert @xcite or roberta @xcite . they show state-ofthe-art performance in a number of tasks from partof-speech tagging, named entity recognition, and machine translation to measuring textual similarity 1 code source: https://github.com/codehacken/localitypreservation @xcite . bert's success has spawned research dedicated to understanding @xcite and reducing parameters in transformer architectures @xcite . despite these successes, supervised transfer learning is not a panacea. models based on pretrained word embeddings (for bilingual induction) or bert-based models require a large parallel corpus to train on. can we reduce the number of training samples even further? we approach this problem by proposing a framework that exploits the inherent relationship between word or sentence representations in their pretrained manifolds. these relationships help train the model with fewer samples, since each training sample represents a group of instances. we consider three types of tasks: (1) a regression task such as semantic text similarity; (2) a classification task (e.g., natural language inference (nli)); and (3) vector space alignment, where the purpose is to learn a mapping between two independently trained embeddings (e.g., crosslingual word alignment) learning bilingual word embedding models alleviates low resource problems by aligning embeddings from a source language that is rich in available text to a target language with a small corpus with limited vocabulary. 2 largely, recent work focuses on learning a linear mapping to align two embedding spaces by minimizing the mean squared error (mse) between embeddings of words projected from the source domain and their counterparts in the target domain @xcite . minimizing mse is useful when a large set of translated words (between source and target languages) is provided, but the mapping overfits when the parallel corpus is small or may require non-linear transformations the neighbors are distributed across the manifold due to overfitting. (b) shows alignment with a locality preserving loss (lpl) that reconstructs the original manifold in the target domain m2 maintaining its local structure. @xcite . in order to reduce overfitting and improve word alignment, we propose an auxiliary loss function called locality preserving loss (lpl) that trains the model to align two sets of word embeddings while maintaining the local neighborhood structure around words in the source domain. with classification and regression tasks where there are two inputs (e.g., nli and sts-b), we show how the alignment between the two input subspace acts as regularizer, improving the model's accuracy on the task with mse alone and when mse and lpl are combined together. lpl achieves this by augmenting existing text ↔ label pairs with pseudo-pairs constructed from their neighbors. specifically, our main contributions are: training with lpl shows up to 16.38% (90.1% relative), on snli up to 8.9% (19.3% relative) improvement when trained with just 1000 samples (0.002% of the dataset). we train a crosslingual word alignment model giving up to 4.1% (13.8% relative) improvement in comparison to a mse optimized mapping while reducing the size of the parallel corpus required to train the mapping by 40% (3k out of 5k pairs). in locally linear embedding (lle) @xcite , the datapoints are assumed to have a linear relation with their neighbors. to project each point, first a reconstruction loss is utilized to learn the linear relation between a point and their k neighbors. then, the linear relation is used to learn the embeddings in the reduced dimension. 3 wang et al. ( 2014 ) extend autoencoders by modifying the reconstruction loss to use nearest neighbors of data points, leveraging neighborhood relationships between datapoints from non-linear dimension reduction methods like lle and laplacian eigenmaps. ## incorporating locality preservation into task-based learning in this section, we describe locality preserving loss, the assumption underlying the loss function and objective functions (eq. ( 2 ), eq. ( 3 )) that are optimized. the cumulative loss function while training the model is defined in eq. ( 4 ). ## experiment results & analysis we demonstrate the effectiveness of the locality preserving alignment on three types of tasks: semantic text similarity (regression), natural language inference (text classification) and crosslingual word alignment (mapping / regression). in order to compute local neighborhoods, as needed for, e.g., (2), we build a standard kd-tree and find the nearest neighbors using euclidean distance. ## conclusion in this paper, we introduce a new locality preserving loss (lpl) function that learns a linear relation between the given word and its neighbors and then utilizes it to learn a mapping for the neighborhood words that are not a part of the word pairs (parallel corpus). also, we show how the results of the method are comparable to current supervised models while requiring a reduced set of word pairs to train on. the models are trained with sgd as compared to others that learn with svd. additionally, the same alignment loss is applied as a regularizer in a classification task (nli) and a regression task 7 check appendix section b for more details.