A Structure-Aware Generative Adversarial Network for Bilingual Lexicon Induction (2023)

Abstract: Bilingual lexicon induction (BLI) is the task of inducing word translations with a learned mapping function that aligns monolingual word embedding spaces in two different languages. However, most previous methods treat word embeddings as isolated entities and fail to jointly consider both the intra-space and inter-space topological relations between words. This limitation makes it challenging to align words from embedding spaces with distinct topological structures, especially when the assumption of isomorphism may not hold. To this end, we propose a novel approach called the Structure-Aware Generative Adversarial Network (SA-GAN) model to explicitly capture multiple topological structure information to achieve accurate BLI. Our model first incorporates two lightweight graph convolutional networks (GCNs) to leverage intra-space topological correlations between words for generating source and target embeddings. We then employ a GAN model to explore inter-space topological structures by learning a global mapping function that initially maps the source embeddings to the target embedding space. To further align the coarse-grained structures, we develop a pair-wised local mapping (PLM) strategy that enables word-specific transformations in an unsupervised manner. Extensive experiments conducted on public datasets, including languages with both distant and close etymological relationships, demonstrate the effectiveness of our proposed SA-GAN model.

## introduction bilingual lexicon induction (bli) has emerged as a crucial task in natural language processing (nlp), focusing on the discovery of corresponding words between two languages using monolingual corpora. due to its ability to facilitate the transfer of semantic knowledge between languages, bli has been successfully applied in various nlp applications, including machine translation @xcite , cross-lingual sentiment analysis @xcite ) and text classification @xcite . most bli methods aim to learn a mapping function that aligns word embeddings of two languages into a shared embedding space, which allows leveraging independently trained monolingual embeddings and then utilizing the learned mapping to generate bilingual lexicons @xcite . thereinto, @xcite first observed that a linear orthogonal mapping proved to be empirically effective in transforming the source embedding space to the target language's space. this mapping was achieved by minimizing the squared euclidean distance between the translation pairs in a given parallel vocabulary. they attribute the success of their method to the isomorphic assumption that the two embedding spaces exhibit similar geometric structures as they found that the linear projection outperformed its non-linear counterpart with multilayer neural networks. building upon this work, various bli methods have been proposed to improve the inductive performance by enforcing an orthogonality constraint @xcite , normalizing the embeddings @xcite , relaxing the isomorphic assumption @xcite , leveraging the clique-level information @xcite , refining with coherent point drift algorithm @xcite , distinguishing the relative orders @xcite , etc. from them, it can be noticed that reliable mapping functions can be learned even with weak supervision. furthermore, recent advancements have introduced several unsupervised models through adversarial training to learn mapping functions without the need for parallel data @xcite @xcite , offering a data-driven, scal-able, and language-independent approach to induce cross-lingual representations from low-resource languages. however, existing adversarial methods focus on word-level alignment and treat the words in the embedding space as isolated entities, ignoring the underlying topological structures among words. therefore, the relationship between words is not preserved and the topological structure of the embedding spaces is not well exploited or involved during training, leading to poor performance compared with other non-adversarial methods @xcite . in addition, conventional bli methods typically assume that the embedding spaces of different languages are nearly isomorphic, and they learn a global linear mapping function shared by all words based on this assumption. however, recent studies @xcite have found that the isomorphic assumption may not hold strictly due to deviations in the distributions of word embeddings for different languages. consequently, the performance of bli methods might be degraded, especially for the language pairs far from isometry. in such a case, a globally-aligned mapping function may not be an optimal solution. there have been some approaches that attempted to alleviate this problem by learning personalized mapping functions for different words or employing supervised non-linear mapping in latent space @xcite @xcite . however, the supervised signals are all indispensable to their proposals and cannot be applied to the unsupervised learning setting without any labeled data. to address these challenges, we propose a novel unsupervised model called structure-aware generative adversarial network (sa-gan) to explicitly capture multiple topological structure information for accurate bli. specifically, given a source language and a target language, sa-gan first views the embedding space of each language as a graph and utilizes two lightweight graph convolutional networks (gcns) to encode two embeddings for exploring the intra-space topological structures. with the extracted structural information, we formulate the learning of a mapping function in a fashion that admits an adversarial game. sa-gan employs a gan model to learn a linear mapping matrix, allowing for the global mapping of the extracted source embeddings into the target embedding space. unlike previous adversarial methods that usually enforce an orthogonality constraint on the mapping function, sa-gan removes this constraint during adversarial training since the isomorphic assumption may not hold true practically. the learned mapping matrix facilitates the construction of a seed dictionary. to further refine the coarsegrained structures and enhance the seed dictionary, sa-gan introduces a pairwise local mapping algorithm (plm). this algorithm can learn wordspecific transformations for different words based on their nearest neighbors within the seed dictionary. by doing so, our method reduces reliance on isometry and achieves improved bli performance in a fully unsupervised manner. to verify the effectiveness of sa-gan, we conduct extensive experiments with sixteen different language pairs, comprising both etymologically distant and close languages to thoroughly test our model performance with varying degrees of isomorphism between monolingual spaces. experimental results show that our model can achieve comparable performance to state-of-the-art unsupervised methods in most cases and even surpass previous supervised ones. our main contributions can be summarized as follows: ## conclusion in this paper, we proposed a novel unsupervised framework sa-gan for bilingual lexicon induction. different from previous works that generally treat words in the embedding space as isolated entities, sa-gan considers each embedding space as a graph and utilizes a gcn module to learn the topological information between words. additionally, sa-gan employs a gan to learn a linear mapping matrix without imposing an orthogonality constraint, thereby transforming both languages into the same embedding space. to further improve the performance, especially for the language pairs where the isomorphic assumption may not hold exactly, we propose a pairwise local mapping algorithm to learn word-specific transformations instead of only applying a shared global mapping to all words. extensive experiments conducted on the muse dataset demonstrate the superior performance of our model. sa-gan outperforms existing unsupervised alternatives and even surpasses state-of-the-art supervised methods, especially for etymologically distant language pairs.