Data Augmentation with Unsupervised Machine Translation Improves the Structural Similarity of Cross-lingual Word Embeddings (2021)

Abstract: Unsupervised cross-lingual word embedding(CLWE) methods learn a linear transformation matrix that maps two monolingual embedding spaces that are separately trained with monolingual corpora. This method relies on the assumption that the two embedding spaces are structurally similar, which does not necessarily hold true in general. In this paper, we argue that using a pseudo-parallel corpus generated by an unsupervised machine translation model facilitates the structural similarity of the two embedding spaces and improves the quality of CLWEs in the unsupervised mapping method. We show that our approach outperforms other alternative approaches given the same amount of data, and, through detailed analysis, we show that data augmentation with the pseudo data from unsupervised machine translation is especially effective for mapping-based CLWEs because (1) the pseudo data makes the source and target corpora (partially) parallel; (2) the pseudo data contains information on the original language that helps to learn similar embedding spaces between the source and target languages.

## introduction cross-lingual word embedding (clwe) methods aim to learn a shared meaning space between two languages (the source and target languages), which is potentially useful for cross-lingual transfer learning or machine translation @xcite @xcite . although early methods for learning clwes often utilize multilingual resources such as parallel corpora @xcite and word dictionaries @xcite , recent studies have focused on fully unsupervised methods that do not require any cross-lingual supervision @xcite @xcite . most unsupervised methods fall into the category of mapping-based methods, which generally consist of the following procedures: train monolingual word embeddings independently in two languages; then, find a linear mapping that aligns the two embedding spaces. the mappingbased method is based on a strong assumption that the two independently trained embedding spaces have similar structures that can be aligned by a linear transformation, which is unlikely to hold true when the two corpora are from different domains or the two languages are typologically very different @xcite . to address this problem, several studies have focused on improving the structural similarity of monolingual spaces before learning mapping @xcite , but few studies have focused on how to leverage the text data itself. in this paper, we show that the pseudo sentences generated from an unsupervised machine translation (umt) system @xcite facilitates the structural similarity without any additional cross-lingual resources. in the proposed method, the training data of the source and/or target language are augmented with the pseudo sentences (figure 1 ). we argue that this method facilitates the structural similarity between the source and target embeddings for the following two reasons. firstly, the source and target embeddings are usually trained on monolingual corpora. the difference in the content of the two corpora may accentuate the structural difference between the two resulting embedding spaces, and thus we can mitigate that effect by making the source and target corpora parallel by automatically generated pseudo data. secondly, in the mapping-based method, the source and target embeddings are trained independently without taking into account the other language. thus, the embedding structures may not be optimal for clwes. we argue that pseudo sentences generated by a umt figure 1: our framework for training clwes using unsupervised machine translation (umt). we first train umt models using monolingual corpora for each language. we then translate all the training corpora and concatenate the outputs with the original corpora, and train monolingual word embeddings independently. finally, we map these word embeddings on a shared embedding. in the experiments using the wikipedia dump in english, french, german, and japanese, we observe substantial improvements by our method in the task of bilingual lexicon induction and downstream tasks without hurting the quality as monolingual embeddings. moreover, we carefully analyze why our method improves the performance, and the result confirms that making the source and target corpora parallel does contribute to performance improvement, and also suggests that the generated translation data contain information about the original language. joint-training approaches jointly train a shared embedding space given multilingual corpora with cross-lingual supervision such as parallel corpora @xcite , documentaligned corpora @xcite , or monolingual corpora along with a word dictionary @xcite . on the other hand, mapping-based approaches utilize monolingual embeddings that are already obtained from monolingual corpora. they assume structural similarity between monolingual embeddings of different languages and attempt to obtain a shared embedding space by finding a transformation matrix w that maps source word embeddings to the target embedding space @xcite . the transformation matrix w is usually obtained by minimizing the sum of squared euclidian distances between the mapped source embeddings and target embeddings: although finding the transformation matrix w is straightforward when a word dictionary is available, a recent trend is to reduce the amount of crosslingual supervision or to find w in a completely unsupervised manner @xcite . the general framework of unsupervised mapping methods is based on heuristic initialization of a seed dictionary d and iterative refinement of the transformation matrix w and the dictionary d, as described in algorithm 1. in our experiment, we use the unsupervised mappingbased method proposed by @xcite . their method is characterized by the seed dictionary initialized with nearest neighbors based on similarity distributions of words in each language. these mapping-based methods, however, are based on the strong assumption that the two independently trained embedding spaces have similar structures that can be aligned by a linear transformation. although several studies have tackled improving the structural similarity of monolingual spaces before learning mapping @xcite , not much attention has been paid to how to leverage the text data itself. input: the source embeddings x, the target embeddings y output: the transformation matrix w heuristically induce an initial seed word dictionary d while not convergence do compute w given the word dictionary d from the equation (1) update the word dictionary d by retrieving cross-lingual nearest neighbors in a shared embedding space obtained by w end return w algorithm 1: the general workflow of unsupervised mapping methods in this paper, we argue that we can facilitate structural correspondence of two embedding spaces by augmenting the source or/and target corpora with the output from an unsupervised machine translation system @xcite . ## experimental design in this section, we describe how we obtain mapping-based clwes using a pseudo parallel corpus generated from umt. we first train umt models using the source/target training corpora, and then translate them to the machine-translated corpora. having done that, we simply concatenate the machine-translated corpus with the original training corpus, and learn monolingual word embeddings independently for each language. finally, we map these embeddings to a shared clwe space. ## evaluation of cross-lingual mapping in this section, we conduct a series of experiments to evaluate our method. we first evaluate the performance of cross-lingual mapping in our method ( § 4.1) and investigate the effect of umt quality ( § 4.2). then, we analyze why our method improves the bilingual lexicon induction (bli) performance. through carefully controlled experiments, we argue that it is not simply because of data augmentation but because: (1) the generated data makes the source and target corpora (partially) parallel ( § 4.3); ## downstream tasks although clwes were evaluated almost exclusively on the bli task in the past, @xcite recently showed that clwes that perform well on bli do not always perform well in other cross-lingual tasks. therefore, we evaluate our embeddings on the four downstream tasks: topic classification (tc), sentiment analysis (sa), dependency parsing (dp), and natural language inference en-fr en-de en-ja ## analysis application to umt umt is one of the important applications of clwes. appropriate initialization with clwes is crucial to the success of umt @xcite . to investigate how clwes obtained from our method affect the performance of umts, we compare the bleu scores of umts initialized with clwes with and without a pseudo corpus at each iterative step. as shown in to investigate this point, we compare the lexical densities of the training corpus and the pseudocorpus used in the above experiments ( § 4, 5) using type-token ratio ( standardized to some extent as reported in @xcite . as a result, specific words might be easily mapped in clwes using a pseudo corpus foot_12 , and then the translation model makes it easier to translate phrases in more specific patterns. hence, the model cannot generate diverse data during back-translation, and the accuracy is not improved due to easy learning. ## conclusion and future work in this paper, we show that training cross-lingual word embeddings with pseudo data augmentation improves performance in bli and downstream tasks. we analyze the reason for this improvement and found that the pseudo corpus reflects the co-occurrence statistics and content of the other language and that the property makes the structure of the embedding suitable for cross-lingual word mapping. recently, @xcite have shown that fully unsupervised clwe methods fails in many language pairs and argue that researchers should not focus too much on the fully unsupervised settings. still, our findings that improve structural similarity of word embeddings in the fully unsupervised setting could be useful in semi-supervised settings, and thus we would like to investigate this direction in the future.