GARI : Graph Attention for Relative Isomorphism of A rabic Word Embeddings (2023)

Abstract: Bilingual Lexical Induction (BLI) is a core challenge in NLP, it relies on the relative isomorphism of individual embedding spaces. Existing attempts aimed at controlling the relative isomorphism of different embedding spaces fail to incorporate the impact of semantically related words in the model training objective. To address this, we propose GARI that combines the distributional training objectives with multiple isomorphism losses guided by the graph attention network. GARI considers the impact of semantical variations of words in order to define the relative isomorphism of the embedding spaces. Experimental evaluation using the Arabic language data set shows that GARI outperforms the existing research by improving the average P@1 by a relative score of up to 40.95% and 76.80% for in-domain and domain mismatch settings respectively.

## introduction bilingual lexical induction (bli) is a key task in natural language processing. it aims at the automated construction of translation dictionaries from monolingual embedding spaces. bli plays a significant role in multiple different natural language processing applications. for instance, the automated construction of lexical dictionaries plays a key role in the development of linguistic applications for low-resource languages, especially in cases where hand-crafted dictionaries are non-existent. automated construction of high-quality dictionaries also helps in augmenting the end performance of multiple down-streaming tasks, including but not limited to: machine translation @xcite , infor-mation retrieval @xcite , cross-lingual transfers @xcite . earlier methods aimed at the construction of cross-lingual embeddings use linear and/or nonlinear mapping functions in order to map the monolingual embeddings in a shared space. some examples in this regard include retrieval criteria for bilingual mapping by @xcite and bli in non-isomorphic spaces by @xcite . these methods rely on the approximate isomorphism assumption, i.e., they assume that underlying monolingual embedding spaces are geometrically similar, which severely limits their use to closely related data sets originating from similar domains and/or languages exhibiting similar characteristics. the limitations of the mapping-based methods, especially their inability to handle data sets originating from different domains and languages exhibiting different characteristics has been identified by @xcite @xcite . some other noteworthy aspects identified in the literature that limit the end performance of the bli systems, include: (a) algorithmic mismatch for independently trained monolingual embeddings, (b) different parameterization, (c) variable data sizes, (d) linguistic difference, etc., @xcite . in the recent past, there has been a shift in the training paradigm for the bli models, i.e., instead of relying on pre-trained embeddings trained independently of each other, they use explicit isomorphism metrics along with the distributional training objective @xcite . however, a key limitation of these models is their inability to incorporate the impact of semantically related tokens (including their lexical variations) in controlling the relative isomorphism of different spaces. this is illustrated in figure 1 , where the left half of the figure shows a set of semantically related english words, e.g., {strong, rugged, and robust}. these words though lexically different share the same semantics. correspondingly, their translations in the arabic language: { } are also semantically related. we hypothesize that each language encompasses a list of such semantically related words that may be used interchangeably within a fixed context, and in order to control the relative isomorphism of corresponding embedding spaces the end model should be robust to incorporate these semantic variations in the model training objective. to address these challenges, in this paper, we propose graph attention for relative isomorphism (gari). gari combines the distributional training objective with the isomorphism loss in a way that it incorporates the impact of semantically related words using graph attention, required to perform the end-task in a performance-enhanced way. we outline the key contributions of this work as follows: ## related work there is an immense literature on bli and controlling the relative isomorphism of the embedding spaces. in order to save space, we primarily limit the related work of this paper to one that is more relevant to our problem settings. we classify the related work into the following categories: (i) mapping pre-trained embeddings, (ii) combined training. mapping pre-trained embeddings. these methods rely on the use of linear and/or non-linear mappings to map the mono-lingual embeddings to a shared space. earlier works in this regard include principled bilingual dictionaries by @xcite that aim to learn bilingual mappings while preserving invariance for the monolingual analogy tasks. @xcite introduced a self-learning approach to relax the requirements for bilingual training seeds and/or parallel corpora. @xcite formulate the alignment as an optimal transport problem and employ gromov-wasserstein distance to compute the similarity of word pairs across different languages. @xcite propose additional transformation on top of the alignment step to force the synonyms towards a middle point for a better cross-lingual integration of the vector spaces. @xcite introduced language-specific rotations followed by a language-independent similarity in a common space. similar to the word embedding methods, the application of the mapping-based methods to the contextualized embeddings include context-aware mapping by @xcite and alignment of contextualized embeddings by @xcite . combined training. on contrary to the mapping-based methods that rely on pre-trained embeddings, these methods use parallel data as input in order to jointly minimize the mono-lingual as well as cross-lingual training objectives. @xcite introduced methods for cross-lingual word embeddings for multiple languages in a unified vector space aimed to combine the strengths of different languages. @xcite addressed the limitations of joint training methods by combining them with mapping-based schemes for model training. for more details on the joint training methods refer to the survey paper by @xcite . @xcite introduced isovec which uses multiple different isomorphism metrics with skip-gram as the distributional training objective to control the isomorphism. nevertheless, we observe that existing methods for controlling the relative isomorphism ignore the impact of words that are semantically related to a given word, severely limits the ability of these methods to control the relative isomorphism of the embedding spaces. ## background in this section, we first introduce the mathematical notation being used throughout the paper and formulate our problem definition. later, we provide a quick background of the vecmap @xcite , a toolkit for mapping across different embedding spaces. ## proposed approach in this paper, we address a core challenge in controlling the geometric isomorphism for source word embeddings relative to the target word embeddings, i.e., incorporate the impact of semantically coherent words in order to perform the end task in a performance augmented fashion. for this, we propose graph attention for relative isomorphism (gari), shown in figure 2 . details about the individual components of gari are provided in the following subsections. ## discussion in this section, we perform a detailed analysis of gari under different settings. for this, we perform analyses encompassing: (i) domain mismatch settings, (ii) correlation with isometric metrics, and (iii) error analysis. ## conclusions and future research in this work, we propose graph attention for relative isomorphism (gari). gari incorporates the impact of semantically related words in order to control the relative isomorphism of geometric spaces in a performance-enhanced way. experimental evaluation using the arabic data set shows that gari outperforms the existing state-of-theart research by 40.95% and 76.80% for in-domain and domain mismatch settings. in the future, we will extend this research to deep contextualized embeddings and non-euclidean geometries.