DM - BLI : Dynamic Multiple Subspaces Alignment for Unsupervised Bilingual Lexicon Induction (2024)

Abstract: Unsupervised bilingual lexicon induction (BLI) task aims to find word translations between languages and has achieved great success in similar language pairs. However, related works mostly rely on a single linear mapping for language alignment and fail on distant or low-resource language pairs, achieving less than half the performance observed in rich-resource language pairs. In this paper, we introduce DM-BLI, a Dynamic Multiple subspaces alignment framework for unsupervised BLI. DM-BLI improves language alignment by utilizing multiple subspace alignments instead of a single mapping. We begin via unsupervised clustering to discover these subspaces in source embedding space. Then we identify and align corresponding subspaces in the target space using a rough global alignment. DM-BLI further employs intra-cluster and inter-cluster contrastive learning to refine precise alignment for each subspace pair. Experiments conducted on standard BLI datasets for 12 language pairs (6 rich-resource and 6 low-resource) demonstrate substantial gains achieved by our framework. We release our code at https://github.com/huling-2/DM-BLI.git.

## introduction unsupervised bilingual lexicon induction (bli) has shown to be a key multilingual nlp task to align cross-lingual word embeddings (clwes) @xcite and bridge lexical gap between low and rich-resource languages @xcite . existing bli approaches can be roughly divided into two categories: mapping-based methods @xcite @xcite and generation-based methods @xcite @xcite . mapping-based methods aim to * *corresponding author figure 1 . t-sne visualization of the clustered monolingual word embedding in a distant language pair of english (left) and japanese (right). different colors represent different subspaces. with a global orthogonal mapping from english to japanese, bli accuracies for subspaces 0-5 are 54.3%, 48.7%, 40.1%, 19.4%, 18.9% and 6.9%, respectively. the existing fully unsupervised mapping-based approaches still need to carefully address two issues. first, these approaches rely on a strong assumption that monolingual word embedding spaces are isomorphic and the mapping matrices should be under orthogonal constraint, but this assumption does not hold true for all languages @xcite , especially for distant language pairs @xcite . therefore, weak orthogonal constraints have been proposed to tackle this issue @xcite . second, a global mapping matrix does not consistently perform optimally across all subspaces @xcite . as shown in figure 1 , subspaces exhibit inconsistent structural similarity. with a global orthogonal mapping, bli accuracy varies among different subspaces: the highest accuracy is 54.3% in subspace 0 and the lowest accuracy is 6.89% in subspace 5. to alleviate the issue, recent research proposed a multiadversarial learning method @xcite and a graph-based paradigm @xcite to learn or refine a specific mapping for each subspace. however, in these approaches, multiple subspaces assigned by initial mappings are static. once initial solutions of these mappings are not good enough, they may get stuck in poor local optima. different from previous methods, we propose a dynamic multiple subspaces cross-lingual alignment framework for fully unsupervised bilingual lexicon induction, named dm-bli. it leverages intra-cluster and inter-cluster contrastive learning to achieve precise alignment at subspace level for both source and target languages, along with dynamically updating the subspace assignment of each word. dm-bli starts by clustering the embeddings of source language to establish multiple valid subspaces. then, we induce an initial solution to discover corresponding multiple subspaces in the target language. finally, we iteratively refine a pair of specific mappings for each subspace pair until convergence is reached. in summary, we make the following contributions: ## experiment setup we evaluate our framework in both supervised and unsupervised bli tasks on 12 language pairs, which contain 6 rich-resource language pairs: spanish (es), german (de), russian (ru), arabic (ar), japanese (ja) and chinese (zh), all cross-lingual to english (en) and six low-resource language pairs: finnish (fi), hindi (hi), turkish (tr), indonesian (id), bulgarian (bg) and catalan (ca), all cross-lingual to english (en). following previous research @xcite , we select relatively low-resource language pairs as low-resource setting rather than truly lowresource languages. ## conclusion in this paper, we propose a dynamic multiple subspaces alignment framework for unsupervised bli, called dm-bli. our method utilizes multiple subspaces alignment instead of a single mapping alignment to achieve more accurate alignment on the subspace level. the experiments show that our method can significantly improve the bilingual word induction performance compared with strong baselines even including gpt-3.5, especially for distant and low-resource language pairs. at the same time, the unsatisfied performances of bloom 7b and llama 13b on all language pairs also suggest that it is difficult to extract lexical information from large language models and the generalization of llms to low-resource languages remains an open challenge. in the future, we will consider combining our method with multilingual llms to take advantage of these two paradigms.