Revisiting the Context Window for Cross-lingual Word Embeddings (2020)

Abstract: Existing approaches to mapping-based cross-lingual word embeddings are based on the assumption that the source and target embedding spaces are structurally similar. The structures of embedding spaces largely depend on the co-occurrence statistics of each word, which the choice of context window determines. Despite this obvious connection between the context window and mapping-based cross-lingual embeddings, their relationship has been underexplored in prior work. In this work, we provide a thorough evaluation, in various languages, domains, and tasks, of bilingual embeddings trained with different context windows. The highlight of our findings is that increasing the size of both the source and target window sizes improves the performance of bilingual lexicon induction, especially the performance on frequent nouns.

## introduction cross-lingual word embeddings can capture word semantics invariant among multiple languages, and facilitate cross-lingual transfer for lowresource languages @xcite . recent research has focused on mapping-based methods, which find a linear transformation from the source to target embedding spaces @xcite @xcite . learning a linear transformation is based on a strong assumption that the two embedding spaces are structurally similar or isometric. the structure of word embeddings heavily depends on the co-occurrence information of words @xcite , i.e., word embeddings are computed by counting other words that appear in a specific context window of each word. the choice of context window changes the co-occurrence statistics of words and thus is crucial to determine the structure of an embedding space. for example, it has been known that an embedding space trained with a smaller linear window captures functional similarities, while a larger window captures topical similarities @xcite . despite this important relationship between the choice of context window and the structure of embedding space, how the choice of context window affects the structural similarity of two embedding spaces has not been fully explored yet. in this paper, we attempt to deepen the understanding of cross-lingual word embeddings from the perspective of the choice of the context window through carefully designed experiments. we experiment with a variety of settings, with different domains and languages. we train monolingual word embeddings varying the context window sizes, align them with a mapping-based method, and then evaluate them with both intrinsic and downstream cross-lingual transfer tasks. our research questions and the summary of the findings are as follows: rq1: what kind of context windows produces a better alignment of two embedding spaces? our result shows that increasing the window sizes of both the source and target embeddings improves the accuracy of bilingual dictionary induction consistently regardless of the domains of the source and target corpora. our fine-grained analysis reveals that frequent nouns receive the most benefit from larger context sizes. rq2. in downstream cross-lingual transfer, do the context windows that perform well on the source language also perform well on the target languages? no. we find that even when some context window performs well on the source language task, that is often not the best choice for the target language. the general tendency is that broader context windows produce better performance for the target languages. ## bilingual lexicon induction we first evaluate the learned bilingual embeddings with bilingual lexicon induction (bli). the task is to retrieve the target translations with source words by searching for nearest neighbors with cosine similarity in the bilingual embedding space. the evaluation metric used in prior work is usually top-k precision, but here we use a more informative measure, mean reciprocal rank (mrr) as recommended by @xcite . fixed target context window settings. first, we consider the settings where the target context size is fixed, and the source context size is configurable. this setting assumes common situations where the embedding of the target language is available in the form of pre-trained embeddings. figure 1 shows the result of the four languages. firstly, we observe that too small windows (1 to 3) for source embeddings do not yield good performance, probably because the model failed to train accurate word embedding models with insufficient training word-context pairs that the small windows capture. at first, this result may seem to contradict with the result from @xcite . they trained english and spanish embeddings with fasttext @xcite and the window size of 2, and then aligned them with an unsupervised mapping algorithm @xcite . when they changed the window size of the spanish embedding to 10, they only observed a very slight drop on top-1 precision (from 81.89 to 81.28). we suspect that the discrepancy with our result is due to the different settings. first of all, fasttext adopts a dynamic window mechanism, which may obfuscate the difference in the context window. also, they trained embeddings with full wikipedia articles, which is an order of magnitude larger than ours; the fasttext algorithm, which takes into account the character n-gram information of words, can exploit a nontrivial amount of subword overlap between the quite similar languages. overall, we observe that the best context window size for the source embeddings increases as the target context size increases, and increasing the context sizes of both the source and target embedding seems beneficial to the bli performance. configurable source/target context window settings. hereafter, we present the results where both the source and target sizes are configurable and set to the same. figure 3 summarizes the result of the same domain setting. as we expected from the observation of the settings where the target window size is fixed, the performance consistently improves as the source and target context sizes increase. given that the larger context windows tend to capture topical similarities of words, we hypothesize that the more topical the embeddings are, the easier they are to be aligned. topics are invariant across different languages to some extent as long as the corpora are comparable. it is natural to think that topic-oriented embeddings capture languageagnostic semantics of words and thus are easier to be aligned among different languages. this hypothesis can be further supported by looking at the metrics of each part-of-speech (pos). intuitively, nouns tend to be more representative of topics than other pos, and thus are expected to show a high correlation with the window size. figure 2 shows the scores for each pos. 8 in all languages, nouns and adjectives show stronger (almost perfect) correlation than verbs and adverbs. different-domain settings. the results so far are obtained in the settings where the source and target corpora are comparable. when the corpora are comparable, it is natural that topical embeddings are easier to be aligned as comparable corpora share their topics. in order to see if the observations from the comparable settings hold true for different-domain settings, we also present the result from the different-domain (news) source corpora in figure 4 . firstly, compared to the same-domain settings (figure 3 ), the scores are lower by around 0.1 to 0.2 points across the languages and context windows, even with the same amount of training data. this result confirms previous findings showing that domain consistency is important to the isomorphism assumption @xcite . as to the relation between the bli performance and the context window, we observe a similar trend to the comparable settings: increasing the context window size basically improves the performance. figure 5 summarizes the results for each pos. the performance on nouns and adjectives still accounts for much of the correlation with the window size. this suggests that even when the source and target domains are different, some domain-invariant topics are captured by larger-context embeddings for nouns and adjectives. ## downstream tasks although bli is a common evaluation method for bilingual embeddings, good performance on bli does not necessarily generalize to downstream tasks @xcite . to further gain insight into the effect of the context size on bilingual embeddings, we evaluate the embeddings with three downstream tasks: 1) sentiment analysis; 2) document classification; 3) dependency parsing. here, we briefly describe the dataset and model used for each task. sentiment analysis (sa). we use the webis-cls-10 corpus foot_8 (prettenhofer and stein, 2010), which is comprised of amazon product reviews in the four languages: english, german, french, and japanese (no russian data available). we cast sentiment analysis as a binary classification task, where we label reviews with the scores of 1 or 2 as negative and reviews with 4 or 5 as positive. for the model, we employ a simple cnn encoder followed by a multi-layer perceptrons classifier. document classification (dc). mldoc 11 @xcite is compiled from the reuters corpus for eight languages including all the languages used in this paper. the task is a four-way classification of the news article topics: corporate/industrial, economics, government/social, and markets. we use the same model architecture as sentiment analysis. dependency parsing (dp). we train deep biaffine parsers @xcite with the ud english ewt dataset 12 @xcite . we use the pud treebanks 13 as test data. the hyperparameters used in this experiment are shown in appendix b. evaluation setup. we evaluate in a cross-lingual transfer setup how well the bilingual embeddings trained with different context windows transfer lexical knowledge across languages. here, we focus on the settings where both the source and target context sizes are varied. for each task, we train models with our pretrained english embeddings. we do not update the parameters of the embedding during training. then, we evaluate the model with the test data in other languages available in the dataset. at test time, we feed the model with the word embeddings of the test language aligned to the training english embeddings. we train nine models in total for each setting with different random seeds and english embeddings, and we present their average scores and standard deviations. ## conclusion and future work despite their obvious connection, the relation between the choice of context window and the structural similarity of two embedding spaces has not been fully investigated in prior work. in this study, we have offered the first thorough empirical results on the relation between the context window size and bilingual embeddings, and shed new light on the property of bilingual embeddings. in summary, we have shown that: we hope that our study will provide insights into ways to improve cross-lingual embeddings by not only mapping methods but also the properties of monolingual embedding spaces.