Why Overfitting Isn’t Always Bad: Retrofitting Cross-Lingual Word Embeddings to Dictionaries (2020)

Abstract: Cross-lingual word embeddings (CLWE) are often evaluated on bilingual lexicon induction (BLI). Recent CLWE methods use linear projections, which underfit the training dictionary, to generalize on BLI. However, underfitting can hinder generalization to other downstream tasks that rely on words from the training dictionary. We address this limitation by retrofitting CLWE to the training dictionary, which pulls training translation pairs closer in the embedding space and overfits the training dictionary. This simple post-processing step often improves accuracy on two downstream tasks, despite lowering BLI test accuracy. We also retrofit to both the training dictionary and a synthetic dictionary induced from CLWE, which sometimes generalizes even better on downstream tasks. Our results confirm the importance of fully exploiting training dictionary in downstream tasks and explains why BLI is a flawed CLWE evaluation.

## introduction cross-lingual word embeddings (clwe) map words across languages to a shared vector space. recent supervised clwe methods follow a projection-based pipeline @xcite . using a training dictionary, a linear projection maps pre-trained monolingual embeddings to a multilingual space. while clwe enable many multilingual tasks @xcite @xcite , most recent work only evaluates clwe on bilingual lexicon induction (bli). specifically, a set of test words are translated with a retrieval heuristic (e.g., nearest neighbor search) and compared against gold translations. bli accuracy is easy to compute and captures the desired property of clwe that translation pairs should be close. however, bli accuracy figure 1: to fully exploit the training dictionary, we retrofit projection-based clwe to the training dictionary as a post-processing step (pink parts). to preserve correctly aligned translations in the original clwe, we optionally retrofit to a synthetic dictionary induced from the original clwe (orange parts). let's think about why that might be. bli accuracy is only computed on test words. consequently, bli hides linear projection's inability to align all training translation pairs at once; i.e., projectionbased clwe underfit the training dictionary. underfitting does not hurt bli test accuracy, because test words are excluded from the training dictionary in bli benchmarks. however, words from the training dictionary may be nonetheless predictive in downstream tasks; e.g., if "good" is in the training dictionary, knowing its translation is useful for multilingual sentiment analysis. in contrast, overfitting the training dictionary hurts bli but can improve downstream models. we show this by adding a simple post-processing step to projection-based pipelines (figure 1 ). after training supervised clwe with a projection, we retrofit @xcite the clwe to the same training dictionary. this step pulls training translation pairs closer and overfits: the updated embeddings have perfect bli training accuracy, but bli test accuracy drops. empirically, retrofitting improves accuracy in two downstream tasks other than bli, confirming the importance of fully exploiting the training dictionary. unfortunately, retrofitting to the training dictionary may inadvertently push some translation pairs further away. to balance between fitting the training dictionary and generalizing on other words, we explore retrofitting to both the training dictionary and a synthetic dictionary induced from the clwe. adding the synthetic dictionary keeps some correctly aligned translations in the original clwe and can further improve downstream models by striking a balance between training and test bli accuracy. in summary, our contributions are two-fold. first, we explain why bli does not reflect downstream task accuracy. second, we introduce two post-processing methods to improve downstream models by fitting the training dictionary better. ## limitation of projection-based clwe this section reviews projection-based clwe. we then discuss how bli evaluation obscures the limitation of projection-based methods. let @xmath0, where each column x @xmath1, and let @xmath2. projection-based clwe maps x and z to a shared space. we focus on supervised methods that learn the projection from a training dictionary d with translation pairs (i, j). @xcite first propose projectionbased clwe. they learn a linear projection @xmath3: recent work improves this method with different optimization objectives @xcite , orthogonal constraints on w @xcite @xcite , pre-processing @xcite , and subword features @xcite @xcite . projection-based methods underfit-a linear projection has limited expressiveness and cannot perfectly align all training pairs. unfortunately, this weakness is not transparent when using bli as the standard evaluation for clwe, because bli test sets omit training dictionary words. however, when the training dictionary covers words that help downstream tasks, underfitting limits generalization to other tasks. some bli benchmarks use frequent words for training and infrequent words for testing @xcite . this mismatch often appears in real-world data, because frequent words are easier to find in digital dicitonaries @xcite . therefore, training dictionary words are often more important in downstream tasks than test words. ## retrofitting to dictionaries to fully exploit the training dictionary, we explore a simple post-processing step that overfits the dictionary: we first train projection-based clwe and then retrofit to the training dictionary (pink parts in figure 1 ). retrofitting was originally introduced for refining monolingual word embeddings with synonym constraints from a lexical ontology @xcite . for clwe, we retrofit using the training dictionary d as the ontology. intuitively, retrofitting pulls translation pairs closer while minimizing deviation from the original clwe. let x ′ and z ′ be clwe trained by a projection-based method, where x ′ = wx are the projected source embeddings and z ′ = z are the target embeddings. we learn new clwe x ˆand z ˆby minimizing we use the same α and β as @xcite to balance the two objectives. retrofitting tends to overfit. if α is zero, minimizing l b collapses each training pair to the same vector. thus, all training pairs are perfectly aligned. in practice, we use a non-zero α for regularization, but the updated clwe still have perfect training bli accuracy (figure 2 ). if the training dictionary covers predictive words, we expect retrofitting to improve downstream task accuracy. de es fr it ja ru zh 40 60 80 100 original (train) original (test) +retrofit (train) +retrofit (test) +synthetic (train) +synthetic (test) bli accuracy for proc de es fr it ja ru zh 40 60 80 100 original (train) original (test) +retrofit (train) +retrofit (test) +synthetic (train) +synthetic (test) bli accuracy for cca de es fr it ja ru zh 40 60 80 100 original (train) original (test) +retrofit (train) +retrofit (test) +synthetic (train) +synthetic (test) bli accuracy for rcsls ## experiments we retrofit three projection-based clwe to their training dictionaries and synthetic dictionaries. 1 we evaluate on bli and two downstream tasks. while retrofitting decreases test bli accuracy, it often improves downstream models. ## related work previous work proposes variants of retrofitting broadly called semantic specialization methods. our pilot experiments found similar trends when replacing retrofitting with counter-fitting @xcite and attract-repel @xcite , so we focus on retrofitting. recent work applies semantic specialization to clwe by using multilingual ontologies @xcite , transferring a monolingual ontology across languages @xcite , and asking bilingual speakers to annotate task-specific keywords @xcite . we instead re-use the training dictionary of the clwe. synthetic dictionaries are previously used to iteratively refine a linear projection @xcite . these methods still underfit because of the linear constraint. we instead retrofit to the synthetic dictionary to fit the training dictionary better while keeping some generalization power of projection-based clwe. recent work investigates cross-lingual contextualized embeddings as an alternative to clwe @xcite @xcite @xcite . our method may be applicable, as recent work also applies projections to contextualized embeddings @xcite @xcite . ## conclusion and discussion popular clwe methods are optimized for bli test accuracy. they underfit the training dictionary, which hurts downstream models. we use retrofitting to fully exploit the training dictionary. this post-processing step improves downstream task accuracy despite lowering bli test accuracy. we then add a synthetic dictionary to balance bli test and training accuracy, which further helps downstream models on average. bli test accuracy does not always correlate with downstream task accuracy because words from the training dictionary are ignored. an obvious fix is adding training words to the bli test set. however, it is unclear how to balance between training and test words. bli accuracy assumes that all test words are equally important, but the importance of a word depends on the downstream task; e.g., "the" is irrelevant in document classification but important in dependency parsing. therefore, future work should focus on downstream tasks instead of bli. we focus on retrofitting due to its simplicity. there are other ways to fit the dictionary better; e.g., using a non-linear projection such as a neural network. we leave the exploration of non-linear projections to future work.